-
Notifications
You must be signed in to change notification settings - Fork 46
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Enabling disconnected cluster installation (#205)
Part of #143 This PR sets the core requirements for disconnected cluster installation. This includes required variables, roles for mirroring (legacy for the platform and `oc-mirror` for everything else), applying `oc-mirror` manifests post cluster up, and corresponding changes to the rest of the playbook and docs. ```bash # update the vars files ❯ ansible-playbook playbooks/0_setup.yaml ❯ ansible-playbook playbooks/3_setup_kvm_host.yaml # If using in in-place registry (for testing) set this up on the plar here and bind it to the interface where the cluster will be setup. Make sure the lpar can resolve to this and then the cluster forwarder IP is set to the gateway IP of the interface Eg 192.168.122.1 and then patch ins reg and pull secret values accordingly. A sample script for this is provided below ❯ ansible-playbook playbooks/4_create_bastion.yaml ❯ ansible-playbook playbooks/disconnected_mirror_artifacts.yaml ❯ ansible-playbook playbooks/5_setup_bastion.yaml ❯ ansible-playbook playbooks/6_create_nodes.yaml ❯ ansible-playbook playbooks/7_ocp_verification.yaml ❯ ansible-playbook playbooks/disconnected_apply_operator_manifests.yaml ``` Sample script for in-place registry on the LPAR itself for NAT-based cluster setup. To be run only after the libvirt interface is created/available on lpar: https://github.com/mohammedzee1000/IBM-Ansible-OpenShift-Provisioning-TestNotes/blob/main/scripts/testing_utils/create_reg.sh --------- Signed-off-by: Mohammed Zeeshan Ahmed <[email protected]> Signed-off-by: Mohammed Zeeshan Ahmed <[email protected]> Co-authored-by: Mohammed Zeeshan Ahmed <[email protected]> Co-authored-by: Jacob Emery <[email protected]>
- Loading branch information
3 people
authored
Oct 30, 2023
1 parent
3168120
commit 1e81c50
Showing
22 changed files
with
444 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| .DS_Store | ||
| .iso | ||
| .vscode | ||
| site | ||
| site | ||
| .oc-mirror-results |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # Run the Playbooks | ||
| ## Overview | ||
| For installing disconnected clusters, you will mostly be following rhe same process as a standard connected cluster. | ||
|
|
||
| The main additional steps we would be doing is mirroring the OCP images to another registry which is accessible to | ||
| the cluster and post the cluster coming up, we will be applying operator hub manifests such as image content source | ||
| policy and catalog source, generated by `oc-mirror`, to the cluster. | ||
|
|
||
| Disconnected playbook are mentioned below. Please refer the **4 Run the Playbooks** documentation for details of rest of the playbooks: | ||
|
|
||
| * disconnected_mirror_artifacts.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/disconnected_mirror_artifacts.yaml)) - Run before **6_create_nodes.yaml** | ||
| * disconnected_apply_operator_manifests.yaml ([code](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/disconnected_apply_operator_manifests.yaml)) - Run after **7_ocp_verification.yaml** | ||
| ## Pre-requisites | ||
| * A running registry where the OCP and operator hub images will be mirrored. If the CA of this registry is not automatically trusted, then keep the | ||
| CA cert content handy to update in inventory file. The CA cert is the file with which, do dont need to skip tls to access the registry. | ||
| * Make sure you have required pull secrets handy. You will need 2 pull secrets, one to apply on the cluster and another which will be used for | ||
| mirroring. The mirroring pull secret MUST have push access to the mirror registry as well as must give you access to Red Hat registries. | ||
| A good way to create this would be take the Red Hat pull secret from **Get Info page** and do a podman login with creds having write access. | ||
|
|
||
| ``` | ||
| cp -avrf /path/to/redhat-pull-secrets.json ./mirror-secret.json | ||
| podman login -u admin -p admin <mirror_registry> --tls-verify=false --authfile=./mirror-secret.json | ||
| cat ./mirror-secret.json | jq -r tostring | ||
| <copy this output> | ||
| ``` | ||
| * A mirror host. This can be any host that can access the internet (mainly the registry being mirrored from) as well as the registry being mirrored to. | ||
| This registries being mirrored from would typically be the Red Hat registries (registry.redhat.io, quay.io etc) | ||
| * The file server, configured mentioned below. | ||
| * Appropriately updated variables in your `all.yaml`. Refer the variables documentation. | ||
| ### File Server | ||
| This configuration will take place on the file server mentioned under **File Server** section in overall pre-requisites documentaion. The additional | ||
| configurations are mentioned over here. | ||
| * Make sure to have a directory housing the clients | ||
| * For FTP: | ||
| ``` | ||
| sudo mkdir /home/<username>/clients | ||
| ``` | ||
| * or HTTP: | ||
| ``` | ||
| sudo mkdir /var/www/html/clients | ||
| ``` | ||
| Make sure this directory contains a pre-downloaded `oc-mirror` binary in `tar.gz` format. Currently the supported binary is available for `x86_64` on Red Hat Customer portal openshift [downloads](https://console.redhat.com/openshift/downloads) page. It can also be found on mirror.openshift.com from `4.14` onwards for other architectures. | ||
| ### NOTE | ||
| * At this stage, only oc-mirror binary is fetched from File Server, so it is expected that the lpar for disconnected cluster can at least reach `mirror.openshift.com` to download the | ||
| other artifacts for cluster installation. | ||
| * The platorm related image content source policy will be baked into the install config as part of **5 Setup Bastion Playbook**. | ||
| * Right now on legacy platform mirroring is supported in this playook during the creation of the cluster. | ||
| * Manifests generated by `oc-mirror` will be applied to the cluster cluster up. So if you add platform details in image set, it | ||
| will be applied on cluster only after the cluster is up. | ||
| ## Disconnected Mirror Artifacts Playbook | ||
| ### Overview | ||
| Mirror the ocp platform and other necessary images to the mirror registry. Please run this playbook before you run **6 Create Nodes Playbook** and after | ||
| **0 Setup Playbook**. | ||
| ### Outcomes | ||
| * Download `oc` and `oc-mirror` to the mirror host. | ||
| * Template the mirror pull secret to the mirror host. | ||
| * Add the ca cert to the mirror host anchors if ca is not trusted. | ||
| * Mirror the platform images using `oc adm release mirror` if legacy mirroring is enabled. | ||
| * Template the image set to mirror host and then mirror it using `oc-mirror` plogin. | ||
| * Copy the results on the `oc-mirror` to ansible controller to apply to cluster in future steps. | ||
| ### Notes | ||
| * Currently, platform can **only** be mirrored the legacy way. While the image set can contain platform mirroring configs, it will **not** be applied to cluster during creation. | ||
| * This playbook can be run at any stage after the **0 Setup** playbook. Make sure to run this before the cluster starts pulling at the images from the registry | ||
| which typically happens where the **Create Nodes Playbook** is run. | ||
| # Disconnected apply oc mirror manifests to cluster Playbook | ||
| ### Overview | ||
| Post cluster creation, `oc-mirror` manifests are applied to the cluster. Please run this playbook after **7 OCP Verification Playbook**. | ||
| ### Outcomes | ||
| * Copy the `oc-mirror` results manifests to the bastion. | ||
| * Apply the copied manifests to the cluster. | ||
| * Disable default content sources. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -242,7 +242,36 @@ | |
| **hypershift.agents_parms.vcpus** | vCPUs for agents | 4 | ||
| **hypershift.agents_parms.nameserver** | Nameserver to be used for agents | 192.168.10.1 | ||
|
|
||
| ## 17 - (Optional) Create compute node in a day-2 operation | ||
| ## 17 - (Optional) Disconnected cluster setup | ||
| **Variable Name** | **Description** | **Example** | ||
| :--- | :--- | :--- | ||
| **disconnected.enabled** | True or False, to enable disconnected mode | False | ||
| **disconnected.registry.url** | String containing url of disconnected registry with or without port and without protocol | registry.tt.testing:5000 | ||
| **disconnected.registry.pull_secret** | String containing pull secret of the disconnected registry to be applied on the *cluster*. <br /> Make sure to enclose pull_secret in 'single quotes' and it has appropriate pull access. | '{"auths":{"registry.tt.<br />.testing:5000":{"auth":"b3Blb<br />...<br />4yQQ==","email":"test.<br />[email protected]"}}}' | ||
| **disconnected.registry.mirror_pull_ecret** | String containing pull secret to use for mirroring. Contains Red Hat secret and registry pull <br /> secret. Make sure to enclose pull_secret in 'single quotes' and must be able to push to mirror<br /> registry. | '{"auths":{"cloud.openshift<br />.com":{"auth":"b3Blb<br />...<br />4yQQ==","email":"redhat.<br />[email protected]", "registry.tt..testing:5000":<br />...<br />[email protected]"}}}' | ||
| **disconnected.registry.ca_trusted** | True or False to indicate that mirror registry CA is implicitly trusted or needs to be made<br /> trusted on mirror host and cluster. | False | ||
| **disconnected.registry.ca_cert** | Multiline string containing the mirror registry CA bundle | -----BEGIN CERTIFICATE-----<br />MIIDqDCCApCgAwIBAgIULL+d1HTYsiP+8jeWnqBis3N4BskwDQYJKoZIhvcNAQEF<br />...<br />-----END CERTIFICATE----- | ||
| **disconnected.mirroring.host.name** | String containing the hostname of the host, which will be used for mirroring | mirror-host-1 | ||
| **disconnected.mirroring.host.ip** | String containing ip of the host, which will be used for mirroring | 192.168.10.99 | ||
| **disconnected.mirroring.host.user** | String containing the username of the host, which will be used for mirroring | mirroruser | ||
| **disconnected.mirroring.host.pass** | String containing the password of the host, which will be used for mirroring | mirrorpassword | ||
| **disconnected.mirroring.file_server.clients_dir** | Directory path relative to the HTTP/FTP accessible directory on **env.file_server**<br /> where client binary tarballs are kept | clients | ||
| **disconnected.mirroring.file_server.oc_mirror_tgz** | Name of oc-mirror tarball on **env.file_server** in **disconnected.mirroring.file_server.clients_dir** | oc-mirror.tar.gz | ||
| **disconnected.mirroring.legacy.platform** | True or False if the platform should be mirrored using `oc adm release mirror`. | True | ||
| **disconnected.mirroring.legacy.ocp_quay_release_image_tag** | The tag of the release image *quay.io/openshift-release-dev/ocp-release* to mirror and use | 4.13.1-s390x | ||
| **disconnected.mirroring.legacy.ocp_org** | The org part of the repo on the mirror registry where the release image will be pushed | ocp4 | ||
| **disconnected.mirroring.legacy.ocp_repo** | The repo part of the repo on the mirror registry where the release image will be pushed | openshift4 | ||
| **disconnected.mirroring.legacy.ocp_tag** | The tag part of the repo on the mirror registry where the release image will be pushed.<br /> Full image would be as below.:<br /><br /> disconnected.registry.url/disconnected.mirroring.legacy.ocp_org/disconnected...ocp_repo<br />:disconnected..ocp_tag | v4.13.1 | ||
| **disconnected.mirroring.oc_mirror.oc_mirror_args.continue_on_error** | True or False to give `--continue-on-error` flag to `oc-mirror` | False | ||
| **disconnected.mirroring.oc_mirror.oc_mirror_args.source_skip_tls** | True or False to give `--source-skip-tls` flag to `oc-mirror` | False | ||
| **disconnected.mirroring.oc_mirror.image_set** | YAML fields containing a standard `oc-mirror` [image set](https://docs.openshift.com/container-platform/latest/installing/disconnected_install/installing-mirroring-disconnected.html#oc-mirror-creating-image-set-config_installing-mirroring-disconnected) with some minor changes to schema. <br /> Differences are documented as needed. Used to generate final image set. | see template | ||
| **disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.enabled** | True or False to use registry storage backend for pushing mirrored content directly to the registry. <br /> Currently only this backend is supported.| True | ||
| **disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.imageURL.org** | The org part of registry imageURL from standard image set. | mirror | ||
| **disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.imageURL.repo** | The repo part of registry imageURL from standard image set. <br /> Final imageURL will be as below:<br /> <br /> disconnected.registry.url/disconnected.mirroring.oc_mirror.image_set.storageConfig<br />.registry.imageURL.org/disconnected...imageURL.repo | oc-mirror-metadata | ||
| **disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.skipTLS** | True of False same purpose served as in standard image set i.e. skip the tls for the registry<br /> during mirroring.| false | ||
| **disconnected.mirrroing.oc_mirror.image_set.mirror** | YAML containing a list of what needs to be mirrored. See the oc mirror image set documentation. <br /> *WARNING*: Platform mirroring in this way is not supported. Use legacy way of platform mirroring | see oc-mirror [image set](https://docs.openshift.com/container-platform/latest/installing/disconnected_install/installing-mirroring-disconnected.html#oc-mirror-creating-image-set-config_installing-mirroring-disconnected) documentation | ||
|
|
||
| ## 18 - (Optional) Create compute node in a day-2 operation | ||
|
|
||
| **Variable Name** | **Description** | **Example** | ||
| :--- | :--- | :--- | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| --- | ||
| - name: Disconnected apply operator manifests | ||
| hosts: bastion | ||
| become: true | ||
| environment: | ||
| KUBECONFIG: "{{ '/home/' if (env.bastion.access.user != 'root') else '/'}}{{ env.bastion.access.user }}/.kube/config" | ||
| gather_facts: true | ||
| vars_files: | ||
| - "{{ inventory_dir }}/group_vars/all.yaml" | ||
| tasks: | ||
| - name: apply operator manifests when cluster is disconnected | ||
| ansible.builtin.include_role: | ||
| name: '{{ item }}' | ||
| loop: | ||
| - disconnected_check_vars | ||
| - disconnected_apply_operator_manifests_to_cluster | ||
| when: disconnected.enabled |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| --- | ||
| - name: Mirror ocp artifacts | ||
| hosts: mirrorhost | ||
| gather_facts: false | ||
| vars_files: | ||
| - "{{ inventory_dir }}/group_vars/all.yaml" | ||
| tasks: | ||
| - name: mirror ocp artifacts when disconnected install | ||
| ansible.builtin.include_role: | ||
| name: '{{ item }}' | ||
| loop: | ||
| - disconnected_check_vars | ||
| - disconnected_mirror_images | ||
| when: disconnected.enabled |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.