From 7cb508088fbd0fbc7555d4f6f63843e678006197 Mon Sep 17 00:00:00 2001 From: Vasilis Remmas Date: Mon, 11 Mar 2024 12:02:07 +0100 Subject: [PATCH] Add the remaining Helm chart options Signed-off-by: Vasilis Remmas --- docs/common/vars.rst | 8 +- docs/customizations/helm.rst | 493 +++++++++++++++++++++++++++++++++++ 2 files changed, 500 insertions(+), 1 deletion(-) diff --git a/docs/common/vars.rst b/docs/common/vars.rst index 8273d89..1af85d4 100644 --- a/docs/common/vars.rst +++ b/docs/common/vars.rst @@ -1,4 +1,10 @@ .. |ocp-channel| replace:: v24.1.0 .. |mofed-version| replace:: 24.01-0.3.3.1.3 .. |sriov-device-plugin-version| replace:: v3.5.1 -.. |k8s-rdma-shared-dev-plugin-version| replace:: v1.3.2 \ No newline at end of file +.. |k8s-rdma-shared-dev-plugin-version| replace:: v1.3.2 +.. |ib-kubernetes-version| replace:: v1.0.2 +.. |cni-plugins-version| replace:: v1.2.0 +.. |multus-version| replace:: v3.9.3 +.. |ipoib-cni-version| replace:: v1.1.0 +.. |whereabouts-version| replace:: v0.6.1 +.. |nvidia-ipam-version| replace:: v0.0.3 diff --git a/docs/customizations/helm.rst b/docs/customizations/helm.rst index 9715307..cf6574d 100644 --- a/docs/customizations/helm.rst +++ b/docs/customizations/helm.rst @@ -16,6 +16,7 @@ .. headings # #, * *, =, -, ^, " +.. include:: ../common/vars.rst ******************************** Helm Chart Customization Options @@ -109,54 +110,546 @@ The NFD labels required by the Network Operator and GPU Operator: MLNX_OFED Driver ================ +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - ofedDriver.deploy + - Bool + - true + - Deploy Node Feature Discovery + * - ofedDriver.deploy + - Bool + - false + - Deploy the MLNX_OFED driver container + * - ofedDriver.repository + - String + - nvcr.io/nvidia/mellanox + - MLNX_OFED driver image repository + * - ofedDriver.image + - String + - mofed + - MLNX_OFED driver image name + * - ofedDriver.version + - String + - |mofed-version| + - MLNX_OFED driver version + * - ofedDriver.env + - List + - [] + - An optional list of environment variables passed to the Mellanox OFED driver image + * - ofedDriver.terminationGracePeriodSeconds + - Int + - 300 + - MLNX_OFED termination grace period in seconds + * - ofedDriver.repoConfig.name + - String + - "" + - Private mirror repository configuration configMap name + * - ofedDriver.certConfig.name + - String + - "" + - Custom TLS key/certificate configuration configMap name + * - ofedDriver.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the MLNX_OFED driver images + * - ofedDriver.startupProbe.initialDelaySeconds + - Int + - 10 + - MLNX_OFED startup probe initial delay + * - ofedDriver.startupProbe.periodSeconds + - Int + - 20 + - MLNX_OFED startup probe interval + * - ofedDriver.livenessProbe.initialDelaySeconds + - Int + - 30 + - MLNX_OFED liveness probe initial delay + * - ofedDriver.livenessProbe.periodSeconds + - Int + - 30 + - MLNX_OFED liveness probe interval + * - ofedDriver.readinessProbe.initialDelaySeconds + - Int + - 10 + - MLNX_OFED readiness probe initial delay + * - ofedDriver.readinessProbe.periodSeconds + - Int + - 30 + - MLNX_OFED readiness probe interval + * - ofedDriver.upgradePolicy.autoUpgrade + - Bool + - false + - Global switch for the automatic upgrade feature. If set to false, all other options are ignored. + * - ofedDriver.upgradePolicy.maxParallelUpgrades + - Int + - 1 + - The amount of nodes that can be upgraded in parallel. 0 means no limit. All nodes will be upgraded in parallel. + * - ofedDriver.upgradePolicy.drain.enable + - Bool + - true + - Options for node drain (`kubectl drain`) before the driver reload, if auto upgrade is enabled. + * - ofedDriver.upgradePolicy.drain.force + - Bool + - false + - Use force drain of pods + * - ofedDriver.upgradePolicy.drain.podSelector + - String + - "" + - Pod selector to specify which pods will be drained from the node. An empty selector means all pods. + * - ofedDriver.upgradePolicy.drain.timeoutSeconds + - Int + - 300 + - Number of seconds to wait for pod eviction + * - ofedDriver.upgradePolicy.drain.deleteEmptyDir + - Bool + - false + - Delete pods local storage + ====================================== MLNX_OFED Driver Environment Variables ====================================== +The following are special environment variables supported by the MLNX_OFED container to configure its behavior: + +.. list-table:: + :header-rows: 1 + + * - Name + - Default + - Description + * - CREATE_IFNAMES_UDEV + - | * "true” for Ubuntu 20.04, RHEL v8.x and OCP <= v4.13. + | * "false" for newer OS. + - Create an udev rule to preserve "old-style" path based netdev names e.g enp3s0f0 + * - UNLOAD_STORAGE_MODULES + - "false" + - | Unload host storage modules prior to loading MLNX_OFED modules: + | * ib_isert + | * nvme_rdma + | * nvmet_rdma + | * rpcrdma + | * xprtrdma + | * ib_srpt + * - ENABLE_NFSRDMA + - "false" + - Enable loading of NFS related storage modules from a MLNX_OFED container + * - RESTORE_DRIVER_ON_POD_TERMINATION + - "true" + - Restore host drivers when a container + +In addition, it is possible to specify any environment variables to be exposed to the MLNX_OFED container, such as the standard "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY". + +.. warning:: + CREATE_IFNAMES_UDEV is set automatically by the Network Operator, depending on the Operating System of the worker nodes in the cluster (the cluster is assumed to be homogenous). + +To set these variables, change them into Helm values. For example: + +.. code-block:: yaml + + ofedDriver: + env: + - name: RESTORE_DRIVER_ON_POD_TERMINATION + value: "true" + - name: UNLOAD_STORAGE_MODULES + value: "true" + - name: CREATE_IFNAMES_UDEV + value: "true" + +The variables can also be configured directly via the NicClusterPolicy CRD. + ========================= RDMA Shared Device Plugin ========================= +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - rdmaSharedDevicePlugin.deploy + - Bool + - true + - Deploy RDMA shared device plugin + * - rdmaSharedDevicePlugin.repository + - String + - nvcr.io/nvidia/cloud-native + - RDMA shared device plugin image repository + * - rdmaSharedDevicePlugin.image + - String + - k8s-rdma-shared-dev-plugin + - RDMA shared device plugin image name + * - rdmaSharedDevicePlugin.version + - String + - |k8s-rdma-shared-dev-plugin-version| + - RDMA shared device plugin version + * - rdmaSharedDevicePlugin.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the RDMA Shared device plugin image + * - rdmaSharedDevicePlugin.resources + - List + - See below + - RDMA shared device plugin resources + * - rdmaSharedDevicePlugin.useCdi + - Bool + - False + - | Enable Container Device Interface (CDI) mode. + | **NOTE**: NVIDIA Network Operator does not configure container runtime to enable CDI + ========================================== RDMA Device Plugin Resource Configurations ========================================== +These configurations consist of a list of RDMA resources, each with a name and a selector of RDMA capable network devices to be associated with the resource. Refer to RDMA Shared Device Plugin Selectors for supported selectors. + +.. code-block:: yaml + + resources: + - name: rdma_shared_device_a + vendors: [15b3] + deviceIDs: [1017] + ifNames: [enp5s0f0] + - name: rdma_shared_device_b + vendors: [15b3] + deviceIDs: [1017] + ifNames: [enp4s0f0, enp4s0f1] + ============================ SR-IOV Network Device Plugin ============================ +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - sriovDevicePlugin.deploy + - Bool + - false + - Deploy SR-IOV Network device plugin + * - sriovDevicePlugin.repository + - String + - ghcr.io/k8snetworkplumbingwg + - SR-IOV Network device plugin image repository + * - sriovDevicePlugin.image + - String + - sriov-network-device-plugin + - SR-IOV Network device plugin image name + * - sriovDevicePlugin.version + - String + - |sriov-device-plugin-version| + - SR-IOV Network device plugin version + * - sriovDevicePlugin.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the SR-IOV Network device plugin image + * - sriovDevicePlugin.resources + - List + - See below + - SR-IOV Network device plugin resources + * - sriovDevicePlugin.useCdi + - Bool + - false + - | Enable Container Device Interface (CDI) mode. + | **NOTE**: NVIDIA Network Operator does not configure container runtime to enable CD. + =================================================== SR-IOV Network Device Plugin Resource Configuration =================================================== +Consists of a list of RDMA resources, each with a name and a selector of RDMA capable network devices to be associated with the resource. Refer to `SR-IOV Network Device Plugin Selectors `_ for supported selectors. + +.. code-block:: yaml + + resources: + - name: hostdev + vendors: [15b3] + - name: ethernet_rdma + vendors: [15b3] + linkTypes: [ether] + - name: sriov_rdma + vendors: [15b3] + devices: [1018] + drivers: [mlx5_ib] + ============= IB Kubernetes ============= +ib-kubernetes provides a daemon that works in conjunction with the `SR-IOV Network Device Plugin `_. It acts on Kubernetes pod object changes (Create/Update/Delete), reading the pod's network annotation, fetching its corresponding network CRD and reading the PKey. This is done in order to add the newly generated GUID or the predefined GUID in the GUID field of the CRD cni-args to that PKey for pods with ``mellanox.infiniband.app`` annotation. + +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - ibKubernetes.deploy + - bool + - false + - Deploy IB Kubernetes + * - ibKubernetes.repository + - string + - ghcr.io/mellanox + - IB Kubernetes image repository + * - ibKubernetes.image + - string + - ib-kubernetes + - IB Kubernetes image name + * - ibKubernetes.version + - string + - |ib-kubernetes-version| + - IB Kubernetes version + * - ibKubernetes.imagePullSecrets + - list + - [] + - An optional list of references to secrets used for pulling any of the IB Kubernetes images + * - ibKubernetes.periodicUpdateSeconds + - int + - 5 + - Interval of periodic update in seconds + * - ibKubernetes.pKeyGUIDPoolRangeStart + - string + - 02:00:00:00:00:00:00:00 + - Minimal available GUID value to be allocated for the pod + * - ibKubernetes.pKeyGUIDPoolRangeEnd + - string + - 02:FF:FF:FF:FF:FF:FF:FF + - Maximal available GUID value to be allocated for the pod + * - ibKubernetes.ufmSecret + - string + - See below + - Name of the Secret with the NVIDIA UFM access credentials, deployed in advance + ========== UFM Secret ========== +IB Kubernetes must access `NVIDIA UFM `_ in order to manage pods' GUIDs. To provide its credentials, the secret of the following format should be deployed in advance: + +.. code-block:: yaml + + apiVersion: v1 + kind: Secret + metadata: + name: ib-kubernetes-ufm-secret + namespace: nvidia-network-operator + stringData: + UFM_USERNAME: "admin" + UFM_PASSWORD: "123456" + UFM_ADDRESS: "ufm-hostname" + UFM_HTTP_SCHEMA: "" + UFM_PORT: "" + data: + UFM_CERTIFICATE: "" + +.. warning:: + InfiniBand Fabric manages a single pool of GUIDs. In order to use IB Kubernetes in different clusters, different GUID ranges must be specified to avoid collisions. + ================= Secondary Network ================= +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - secondaryNetwork.deploy + - Bool + - true + - Deploy Secondary Network + +Specifies components to deploy in order to facilitate a secondary network in Kubernetes. It consists of the following optionally deployed components: + +* `Multus-CNI `_: Delegate CNI plugin to support secondary networks in Kubernetes +* CNI plugins: Currently only `containernetworking-plugins `_ is supported +* IPAM CNI: Currently only `Whereabout IPAM CNI `_ is supported as a part of the secondaryNetwork section. NVIDIA-IPAM is configured separately. +* `IPoIB CNI `_: Allows the user to create IPoIB child link and move it to the pod + ========== CNI Plugin ========== +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - secondaryNetwork.cniPlugins.deploy + - Bool + - true + - Deploy CNI Plugins Secondary Network + * - secondaryNetwork.cniPlugins.image + - String + - plugins + - CNI Plugins image name + * - secondaryNetwork.cniPlugins.repository + - String + - ghcr.io/k8snetworkplumbingwg + - CNI Plugins image repository + * - secondaryNetwork.cniPlugins.version + - String + - |cni-plugins-version|-amd64 + - CNI Plugins image version + * - secondaryNetwork.cniPlugins.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the CNI Plugins images + ========== Multus CNI ========== +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - secondaryNetwork.multus.deploy + - Bool + - true + - Deploy Multus Secondary Network + * - secondaryNetwork.multus.image + - String + - multus-cni + - Multus image name + * - secondaryNetwork.multus.repository + - String + - ghcr.io/k8snetworkplumbingwg + - Multus image repository + * - secondaryNetwork.multus.version + - String + - |multus-version| + - Multus image version + * - secondaryNetwork.multus.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the Multus images + * - secondaryNetwork.multus.config + - String + - `` + - Multus CNI config. If empty, the config will be automatically generated from the CNI configuration file of the master plugin (the first file in lexicographical order in the cni-confg-dir). + ========= IPoIB CNI ========= +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - secondaryNetwork.ipoib.deploy + - Bool + - false + - Deploy IPoIB CNI + * - secondaryNetwork.ipoib.image + - String + - ipoib-cni + - IPoIB CNI image name + * - secondaryNetwork.ipoib.repository + - String + - "" + - IPoIB CNI image repository + * - secondaryNetwork.ipoib.version + - String + - |ipoib-cni-version| + - IPoIB CNI image version + * - secondaryNetwork.ipoib.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the IPoIB CNI images + =============== IPAM CNI Plugin =============== +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - secondaryNetwork.ipamPlugin.deploy + - Bool + - true + - Deploy IPAM CNI Plugin Secondary Network + * - secondaryNetwork.ipamPlugin.image + - String + - whereabouts + - IPAM CNI Plugin image name + * - secondaryNetwork.ipamPlugin.repository + - String + - ghcr.io/k8snetworkplumbingwg + - IPAM CNI Plugin image repository + * - secondaryNetwork.ipamPlugin.version + - String + - |whereabouts|-amd64 + - IPAM CNI Plugin image version + * - secondaryNetwork.ipamPlugin.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the IPAM CNI Plugin images + ================== NVIDIA IPAM Plugin ================== + +NVIDIA IPAM Plugin is recommended to be used on large-scale deployments of the NVIDIA Network Operator. + +.. list-table:: + :header-rows: 1 + + * - Name + - Type + - Default + - Description + * - nvIpam.deploy + - Bool + - false + - Deploy NVIDIA IPAM Plugin + * - nvIpam.image + - String + - nvidia-k8s-ipam + - NVIDIA IPAM Plugin image name + * - nvIpam.repository + - String + - ghcr.io/mellanox + - NVIDIA IPAM Plugin image repository + * - nvIpam.version + - String + - |nvidia-ipam-version| + - NVIDIA IPAM Plugin image version + * - nvIpam.imagePullSecrets + - List + - [] + - An optional list of references to secrets to use for pulling any of the Plugin images + + +.. warning:: + Since several parameters should be provided when creating custom resources during operator deployment, it is recommended to use a configuration file. While it is possible to override the parameters via CLI, we recommend to avoid the use of CLI arguments in favor of a configuration file. + + +.. code-block:: bash + + $ helm install -f ./values.yaml -n nvidia-network-operator --create-namespace --wait nvidia/network-operator network-operator