Daemonset installation missing configs

[ndarilek]

Hey folks, new to Multus so maybe I'm doing something dumb. I downloaded the manifest from https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml as of this morning, changed the logLevel to debug, and applied it. Unfortunately, I don't seem to be getting all the configs on my host that the plugin expects to be there. Occasionally I get logs about a CNI configuration missing but I can't find any now to show.

Various docs say I should find configs in /etc/cni/net.d. My net.d is empty. /opt/cni/bin/multus-shim exists though, so it seems the daemonset can write to my root. It's just failing to populate /etc/cni/net.d for some reason. Nothing in the container startup logs indicates that this isn't or shouldn't work.

I'm a bit confused because I see where multus-shim is installed in the manifest. I.e.:

        - name: install-multus-binary
          image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick
          command:
            - "cp"
            - "/usr/src/multus-cni/bin/multus-shim"
            - "/host/opt/cni/bin/multus-shim"

I don't see anything like that for /etc/cni/net.d though, which multiple sets of docs seem to indicate should happen. Is that done somewhere else?

My setup is Fedora Core OS and k3s which uses Flannel. I'm trying to bring up a Kubevirt VM on a bridged interface. I've run through multiple rounds of running k3s-uninstall.sh, removing /etc/cni and /opt/cni, then reinstalling some variation of k3s/multus/kubevirt again. The thin client install correctly installs the multus binary and leaves a kubectl config somewhere in /etc/cni. So the installation seems perfectly capable of writing to the host, it just doesn't install the main plugin config. I've also tried with the latest release version and that didn't work either. And clearly /etc/cni and /opt/cni are being recreated, so I don't think this is selinux or similar interfering.

I feel like I'm missing something obvious here. Thanks for any help.

[ndarilek]

Interestingly, this appeared in the logs a few minutes after the container started:

2023-09-09T14:19:27Z [error] failed to find the primary CNI plugin: failed to find the cluster master CNI plugin: could not find a plugin configuration in /host/etc/cni/net.d
2023-09-09T14:19:27Z [error] failed to create the configuration manager for the primary CNI plugin: failed to find the cluster master CNI plugin: could not find a plugin configuration in /host/etc/cni/net.d

So it clearly doesn't have its config but I'm not sure why not.

Thanks.

[ndarilek]

OK, so I figured out that I needed to update some Rancher-specific paths in the manifest. I feel like the new way of configuring things, with paths and host volumes in multiple places, and /host and /hostroot, is more difficult to reason about than just mounting a hostroot and editing an entrypoint as I think things used to work.

Anyway, I found a bunch of CNI plugins under a container/installation-specific k3s directory, and updated my config as follows:

  daemon-config.json: |
    {
        "chrootDir": "/hostroot",
        "confDir": "/host/etc/cni/net.d",
        "logLevel": "debug",
        "socketDir": "/host/run/multus/",
        "cniVersion": "0.3.1",
        "cniConfigDir": "/host/etc/cni/net.d",
        "binDir": "/hostroot/var/lib/rancher/k3s/data/8a84c92c5c245778005e7fa2a03c4d84118fde366594fe826bcd7351d8b88d21/bin",
        "multusConfigFile": "auto",
        "multusAutoconfigDir": "/host/etc/cni/net.d"
    }

that binDir now points at a binary directory containing multus-shim, flannel, bandwidth, etc. Yet if I view the logs for my multus pod I see lots of:

2023-09-11T21:37:43Z [debug] conflistDel: &{f917db06d5aeff65416b443bbe6e63bf4efb6dd2e0dd0925a0b4971bb3a1e554 /var/run/netns/cni-7c2ab36b-a560-a3cb-3164-efaaa67ea70e eth0 [[IgnoreUnknown true] [K8S_POD_NAMESPACE kubevirt] [K8S_POD_NAME virt-operator-6c649b9567-dx6xc] [K8S_POD_INFRA_CONTAINER_ID f917db06d5aeff65416b443bbe6e63bf4efb6dd2e0dd0925a0b4971bb3a1e554] [K8S_POD_UID 0f0aa05f-cb28-4624-b31a-ea040fac546b]] map[] }, {
"name":"cbr0",
"cniVersion":"1.0.0",
"plugins":[
{
"type":"flannel",
"delegate":{
"hairpinMode":true,
"forceAddress":true,
"isDefaultGateway":true
}
},
{
"type":"portmap",
"capabilities":{
"portMappings":true
}
},
{
"type":"bandwidth",
"capabilities":{
"bandwidth":true
}
}
]
}

2023-09-11T21:37:43Z [error] conflistDel: error in getting result from DelNetworkList: plugin type="bandwidth" failed (delete): failed to find plugin "bandwidth" in path [/hostroot/var/lib/rancher/k3s/data/8a84c92c5c245778005e7fa2a03c4d84118fde366594fe826bcd7351d8b88d21/bin]
2023-09-11T21:37:43Z [error] DelegateDel: error invoking ConflistDel - "cbr0": conflistDel: error in getting result from DelNetworkList: plugin type="bandwidth" failed (delete): failed to find plugin "bandwidth" in path [/hostroot/var/lib/rancher/k3s/data/8a84c92c5c245778005e7fa2a03c4d84118fde366594fe826bcd7351d8b88d21/bin]
2023-09-11T21:37:43Z [verbose] DEL finished CNI request ContainerID:"f917db06d5aeff65416b443bbe6e63bf4efb6dd2e0dd0925a0b4971bb3a1e554" Netns:"/var/run/netns/cni-7c2ab36b-a560-a3cb-3164-efaaa67ea70e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kubevirt;K8S_POD_NAME=virt-operator-6c649b9567-dx6xc;K8S_POD_INFRA_CONTAINER_ID=f917db06d5aeff65416b443bbe6e63bf4efb6dd2e0dd0925a0b4971bb3a1e554;K8S_POD_UID=0f0aa05f-cb28-4624-b31a-ea040fac546b" Path:"", result: "", err: DelegateDel: error invoking ConflistDel - "cbr0": conflistDel: error in getting result from DelNetworkList: plugin type="bandwidth" failed (delete): failed to find plugin "bandwidth" in path [/hostroot/var/lib/rancher/k3s/data/8a84c92c5c245778005e7fa2a03c4d84118fde366594fe826bcd7351d8b88d21/bin]

There is definitely a bandwidth executable in that directory when I exec into the pod:

[root@lab bin]# kubectl exec -it -n kube-system kube-multus-ds-nvmnx -- /bin/sh                                         
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init)                                     
# /hostroot/var/lib/rancher/k3s/data/8a84c92c5c245778005e7fa2a03c4d84118fde366594fe826bcd7351d8b88d21/bin/bandwidth     
CNI bandwidth plugin v1.3.0-k3s1                                                                                        
CNI protocol versions supported: 0.3.0, 0.3.1, 0.4.0, 1.0.0                                                             

Any other ideas?

[ndarilek]

Actually, since the plugin is a symlink, maybe it isn't working because of this?

multus-cni/vendor/github.com/containernetworking/cni/pkg/invoke/find.go

Line 41 in 7ea924b

if fi, err := os.Stat(fullpath); err == nil && fi.Mode().IsRegular() {

Is IsRegular failing there because the plugin is a symlink?

Assuming I'm correct, is there some reason for not using a valid symlink as a plugin?

[trevex]

Did you ever find a solution? I set up Multus on K3s and it is not working as it can't find the relevant files.

I am patching the daemonset with the following:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-multus-ds
  namespace: kube-system
spec:
  template:
    spec:
      volumes:
        - name: cni
          hostPath:
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d
        - name: cnibin
          hostPath:
            path: /var/lib/rancher/k3s/data/current/bin

Everything seems to work fine until I start a pod with a network attachement:

 plugin type="flannel" failed (add): netplugin failed with no error message: fork/exec /opt/cni/bin/flannel: no such file or directory

Basically same as your setup the plugins are all there and shown when:

k -n kube-system exec -it kube-multus-ds-tx225 -- sh
# ls /opt/cni/bin

[trevex]

From https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/thick-plugin.md#chroot-configuration I gathered, that cni bin needs to be within chroot dir, so I updated my config as follows:

kind: ConfigMap
apiVersion: v1
metadata:
  name: multus-daemon-config
  namespace: kube-system
data:
  daemon-config.json: |
    {
        "chrootDir": "/hostroot",
        "confDir": "/host/etc/cni/net.d",
        "logLevel": "verbose",
        "socketDir": "/host/run/multus/",
        "cniVersion": "0.3.1",
        "cniConfigDir": "/host/etc/cni/net.d",
        "multusConfigFile": "auto",
        "multusAutoconfigDir": "/host/etc/cni/net.d",
        "binDir": "/var/lib/rancher/k3s/data/current/bin"
    }

Leaving me with the same errors as you describe, e.g.:

plugin type="bandwidth" failed (delete): failed to find plugin "bandwidth" in path [/var/lib/rancher/k3s/data/current/bin]

[trevex]

Stumbled upon k3s-io/k3s#9356

The easiest workaround seems to not use inbuilt flannel judging from the referenced slack messages.