4.22.0-okd-scos.0-.5: payload driver-toolkit image built against older kernel (6.12.0-212.el10) than machine-os-content ships

[aqeelat]

Describe the bug

4.22.0-okd-scos.0 through .5 ship a driver-toolkit image built against kernel 6.12.0-212.el10 while their machine-os-content boots -224/-230/-233. Six consecutive releases reused one stale DTK digest across four OS composes; .6/.7 carry rebuilt DTKs (verified working on .6). The DTK's own startup check reports it:

WARNING: broken Driver Toolkit image detected:
- Node kernel:    6.12.0-224.el10.x86_64
- Kernel package: 6.12.0-212.el10.x86_64
INFO: informing nvidia-driver-ctr to fallback on entitled-build.

Full per-release digest/compose/kernel table (via oc adm release info --image-for=driver-toolkit) available in discussion. Impact: out-of-tree kernel-module consumers (e.g. NVIDIA gpu-operator DTK mode) can't build on .0–.5; on OKD the entitled fallback also fails. Asks:

1. Was the stale DTK intentional or build-lag?
2. Can CI assert payload-internal consistency — DTK kernel NVR == machine-os kernel NVR — before promotion?
3. Confirm DTK-per-compose rebuild is standing practice from .6 on.

Version

4.22.0-okd-scos.0 through .6 (verified .0, .1, .6 live); IPI on GCP

Reproducibility

100% on .0–.5 for any Driver Toolkit consumer (observed on .0/.1; .6+ rebuilt and working)

Log Bundle

Inline logs included above (DTK container output); must-gather can be provided on request

[jatinsu]

I just checked in 4.22.0-okd-scos.8, it seems to be the correct DTK image. Any chance you could spin up a .8 cluster and check as well? I wouldn't recommend upgrading to .8 to check as this issue is still occurring, so you would have to spin up a new cluster.

[aqeelat]

Thanks @jatinsu — confirming from our side: .6's rebuilt DTK works live; we run 4.22.0-okd-scos.6 and the DTK-based driver build gets its headers from the payload image. (Only the driver container's kernel meta-package step still needs an external repo — an NVIDIA image-generation issue tracked in gpu-driver-container#985/#984 — and the fallback path is further complicated by gpu-operator#2811, where repoConfig isn't mounted into the DTK container.)

Unfortunately, we don't have the capacity to spin a fresh .8 cluster right now — and noted your #2348 warning, so we're deliberately staying on .6 while it has no outgoing graph edges. We'll validate .8's DTK when we upgrade after #2348 resolves and report back here.

Not sure if this is useful or not, but here is the full per-release table:

Per-release driver-toolkit digest, machine-os compose, and node kernel (4.22.0-okd-scos.0–.8)

Release	machine-os (ostree)	Node kernel	driver-toolkit image digest
.0	10.0.20260428-0	6.12.0-224.el10 (live)	sha256:911d450506de9d691c9ec79556acb5329458e7eb655ea556a1285e8c436129ce (contains 6.12.0-212.el10)
.1	10.0.20260521-0	6.12.0-230.el10 (live)	same as .0
.2	10.0.20260526-0	—	same as .0
.3	10.0.20260526-0	—	same as .0
.4	10.0.20260614-0	—	same as .0
.5	10.0.20260614-0	—	same as .0
.6	10.0.20260614-0	6.12.0-233.el10 (live)	sha256:9e78b79220dcc456cf9ae3b5ba0c88f8e6576cd46b1312503a34a0b916e8444b
.7	10.0.20260702-0	—	sha256:edfa31b43cfa273dec33e548851fbe1c508da8ddcd3de785fef541613e9e2b15
.8	10.0.20260806-0	—	sha256:becb6ead8bf16baaf89a17c9b36fc26665ffcbf89fb82e6ac9df4e86b30bcb8c

Method and caveats: digests and composes come from oc adm release info against each release image (payload inspection). Kernels marked "live" were verified on our running cluster; we did not verify the others — we never ran those releases. The .0–.5 image's inner kernel (6.12.0-212.el10) is as reported by the DTK container's own mismatch check on our .0/.1 clusters: WARNING: broken Driver Toolkit image detected: Node kernel: … / Kernel package: ….

Relevant issues

• [Bug]: DTK mode fails on amd64 - shared-dir prep unconditionally copies /usr/bin/unzboot which is only installed on aarch64 NVIDIA/gpu-driver-container#984 — DTK mode fails on amd64: unconditional unzboot copy (fixed on main by Guard unzboot copy by arch in ocp_dtk_entrypoint, amd64 doesn't have the binary NVIDIA/gpu-driver-container#986; no rebuilt images published yet)
• [Bug]: Published 580.126.20-rhel10.0 image runs real dnf in DTK mode - predates the dnf stub present on main NVIDIA/gpu-driver-container#985 — published 580.126.20-rhel10.0 runs real dnf in DTK mode (predates the dnf→true stub)
• [Bug]: spec.driver.repoConfig is not mounted into openshift-driver-toolkit-ctr (DTK-mode builds can't use it) NVIDIA/gpu-operator#2811 — repoConfig not mounted into openshift-driver-toolkit-ctr, which the skew-fallback path needs