Workload Identity and error "nodePublishSecretRef secret is not set"

[xolott]

I deployed the secrets store csi driver into a freshly created cluster following these resources.

• https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-identity-access
• https://www.linkedin.com/pulse/use-workload-identity-preview-secrets-store-csi-arana-escobedo

I created a workload identity with this script:

# The cluster was created with workload-identity and oidc-issuer enabled

az extension add --name aks-preview
az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableWorkloadIdentityPreview')].{Name:name,State:properties.state}"
az provider register --namespace Microsoft.ContainerService


export RESOURCE_GROUP=rg
export CLUSTER_NAME=aknameexport UAMI=aks-key-vault-secret
export SERVICE_ACCOUNT_NAME="azure-sa"  
export SERVICE_ACCOUNT_NAMESPACE="default" 
export KEYVAULT_NAME=kv-name
export FEDERATED_IDENTITY_NAME="azure-csi-federated-identity"

az identity create --name $UAMI --resource-group $RESOURCE_GROUP
export USER_ASSIGNED_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP --name $UAMI --query 'clientId' -o tsv)
export IDENTITY_TENANT=$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query identity.tenantId -o tsv)

az keyvault set-policy -n $KEYVAULT_NAME --key-permissions get --spn $USER_ASSIGNED_CLIENT_ID
az keyvault set-policy -n $KEYVAULT_NAME --secret-permissions get --spn $USER_ASSIGNED_CLIENT_ID
az keyvault set-policy -n $KEYVAULT_NAME --certificate-permissions get --spn $USER_ASSIGNED_CLIENT_ID

export AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"
echo $AKS_OIDC_ISSUER

az identity federated-credential create --name $FEDERATED_IDENTITY_NAME --identity-name $UAMI --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

Now, using USER_ASSIGNED_CLIENT_ID I created the SA and the SecretProviderClass

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: xxx # USER_ASSIGNED_CLIENT_ID
  labels:
    azure.workload.identity/use: "true"
  name: azure-sa
  namespace: default

---

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-tls
  namespace: default
spec:
  provider: azure
  secretObjects: # secretObjects defines the desired state of synced K8s secret objects
    - secretName: ingress-tls-csi
      type: kubernetes.io/tls
      data:
        - objectName: secret-name
          key: tls.key
        - objectName: secret-name
          key: tls.crt
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: yyy # USER_ASSIGNED_CLIENT_ID
    keyvaultName: kv-name # KEYVAULT_NAME
    cloudName: ""
    objects: |
      array:
        - |
          objectName: secret-name
          objectType: secret
          objectVersion: ""
    tenantId: xxx # IDENTITY_TENANT

Then I deployed a pod just to test if everything is okay:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-pod
  namespace: default
  labels:
    app: my-pod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-pod
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      serviceAccountName: azure--sa
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-tls"
      containers:
        - name: my-pod
          image: my-image
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true

The deployment fails with this error:

MountVolume.SetUp failed for volume "secrets-store-inline" : 
rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/my-pod859b455c58-b6mjl, 
err: rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, 
error: failed to get credentials, nodePublishSecretRef secret is not set

The provider log show:

E0502 02:25:23.005523       1 server.go:58] "failed to process mount request" err="failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set"
E0502 02:27:25.025973       1 server.go:58] "failed to process mount request" err="failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set"
E0502 02:29:27.068907       1 server.go:58] "failed to process mount request" err="failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set"
E0502 02:31:29.098177       1 server.go:58] "failed to process mount request" err="failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set"

The driver log show:

I0502 02:31:29.708897       1 nodeserver.go:144] "node publish volume" target="/var/lib/kubelet/pods/some-id/volumes/kubernetes.io~csi/secrets-store-inline/mount" volumeId="csi-fbd10e52e363dae7a8e6d6614dce051bacb9d6f01e3f48a69a66a1e1417d6f71" attributes=map[csi.storage.k8s.io/ephemeral:true csi.storage.k8s.io/pod.name:keela-main-859b455c58-n56q7 csi.storage.k8s.io/pod.namespace:keela csi.storage.k8s.io/pod.uid:0c6b7ec8-ea6c-4bc4-bde8-06f4ac327baf csi.storage.k8s.io/serviceAccount.name:azure-sa csi.storage.k8s.io/serviceAccount.tokens:{"api://AzureADTokenExchange":{"token":"some-token","expirationTimestamp":"2023-05-02T03:21:13Z"}} secretProviderClass:azure-tls] mount flags=[]
I0502 02:31:29.710943       1 nodeserver.go:359] "Using gRPC client" provider="azure" pod="my-pod-859b455c58-n56q7"
E0502 02:31:29.711651       1 nodeserver.go:242] "failed to mount secrets store object content" err="rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set" pod="default/m-pod-859b455c58-n56q7"
I0502 02:31:29.711728       1 nodeserver.go:88] "unmounting target path as node publish volume failed" targetPath="/var/lib/kubelet/pods/0c6b7ec8-ea6c-4bc4-bde8-06f4ac327baf/volumes/kubernetes.io~csi/secrets-store-inline/mount" pod="default/my-pod-859b455c58-n56q7"

nodePublishSecretRef is needed when using a Service Principal, so why it is failing with that error? There are no references to nodePublishSecretRef in the documentation or blog posts that use Workload Identity

Some context:

• Cluster deployed without a network plugin. Cilium deployed instead but no rules were created, so network policies shouldn't be blocking connections here

[asparagusbeef]

Did you ever fix that? I'm having the same thing.

[ghost]

I saw this same error for a deployment today that is using workload-id. This error appeared after I switched the clientID: I had to helm delete my_app and then reinstall it and it worked fine after that.

[manikprabhu1208]

Even I am facing this issue. Did it work for you after helm uninstall and you again installed it?

[mkemmerz]

In my case I had a typo in the SecretProviderClass' arguments. I was using "spec.parameters.clientId" instead of "spec.parameters.clientID". Fixing the typo also resolved the issue. It seems to be case sensitive, which is a bit unfortunate.

[mwmkws]

I've been struggling with the same issue for a few days.
I haven't done it yet, but I share what I found while searching to fix the problem.

Go to the original blog

(an excerpt of important parts)
May be encountered if the helm csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver is stilled installed in your cluster. To resolved the issue you need to delete the helm installation and update the "azure-keyvault-secrets-provider" add-on:

az aks addon update -g ResourceGroupName -n ClusterName -a azure-keyvault-secrets-provider