Repository navigation
Use published SecureBuild images across latest add-ons - #6204
mikhailswift wants to merge 13 commits into
Conversation
|
Doomer review
Automatic approval: Enabled |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-ekco-0.28.17-k8s-docker-2026-10-14T19:22:11Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-cert-manager-1.21.2-k8s-docker-2026-10-14T19:23:09Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-contour-1.33.7-k8s-docker-2026-10-14T19:23:20Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-metrics-server-0.9.0-k8s-docker-2026-10-14T19:24:03Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-sonobuoy-0.57.5-k8s-docker-2026-10-14T19:25:49Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-prometheus-0.94.1-91.8.0-k8s-docker-2026-10-14T19:26:12Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-containerd-2.3.6-k8s-ctrd-2026-10-14T19:29:43Z |
![doomer-ai[bot]](https://avatars.githubusercontent.com/in/5032069?s=60&v=4){kind=small}
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7eb0749-rook-1.19.7-k8s-docker-2026-10-14T19:34:01Z |
|
Doomer isn't automatically approving this pull request. Why: The Doomer review did not complete successfully. Reviewer failure details: - <code>registry.doomer.ai/library/review/engineering@sha256:2a50f498bff3fc16561ed0bec4fc7233633acd11900e80cf990a3207f0f41b9d [batch-001]</code>: <code>ModelReviewError: model_request_recovery_exhausted: Model request recovery exhausted after 3 attempt(s) (attempt_limit).</code>; Partial review: 1 review jobs failed; unsupported candidates, if any, were withheld. Completed reviewers' findings are retained. No clean-review opinion. Findings from completed reviewers do not establish that the whole review passed. Comment Alternatively, a reviewer can review and approve this pull request on GitHub. |
b0a2bf7 to
c418917
Compare
|
Doomer isn't automatically approving this pull request. Why: The Doomer review did not complete successfully. Reviewer failure details: - <code>registry.doomer.ai/library/review/code@sha256:910c1934ccd3e519656119b30b4eaa9cabbf1815020ec6f9bb55b4860eb069a7 [full-change]</code>: <code>host execution failed (child exit -1): context deadline exceeded</code>; Partial review: 1 review jobs failed; unsupported candidates, if any, were withheld. Completed reviewers' findings are retained. No clean-review opinion. Findings from completed reviewers do not establish that the whole review passed. Comment Alternatively, a reviewer can review and approve this pull request on GitHub. |
|
/doomer run |
|
Doomer recognized |
|
Doomer isn't automatically approving this pull request. Why: The Doomer review did not complete successfully. Reviewer failure details: - <code>registry.doomer.ai/library/review/code@sha256:910c1934ccd3e519656119b30b4eaa9cabbf1815020ec6f9bb55b4860eb069a7 [full-change]</code>: <code>ModelReviewError: model_request_recovery_exhausted: Model request recovery exhausted after 1 attempt(s) (deadline).</code>; Partial review: 1 review jobs failed; unsupported candidates, if any, were withheld. Completed reviewers' findings are retained. No clean-review opinion. Findings from completed reviewers do not establish that the whole review passed. Comment Alternatively, a reviewer can review and approve this pull request on GitHub. |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-82a7372-metrics-server-0.9.0-k8s-docker-2026-10-15T02:40:15Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-82a7372-ekco-0.28.17-k8s-docker-2026-10-15T02:40:18Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-82a7372-contour-1.33.7-k8s-docker-2026-10-15T02:40:23Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-82a7372-cert-manager-1.21.2-k8s-docker-2026-10-15T02:40:36Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7f0c981-rook-1.19.7-k8s-docker-2026-10-15T22:19:53Z |
- Move old-k8s legs from 1.21.x to 1.25.x: contour CRDs use x-kubernetes-validations (CEL), which requires Kubernetes >= 1.25. - Use 'latest' kubernetes for the current-version legs and float the upgrade baseline on kurl.sh/latest. - Exclude old-kernel OSes (amzn-20, centos-79, ol-79, ol-8x, ubuntu-1804) from the 'k8s latest' leg; containerd 2.3.6 bundles runc 1.5.1 which cannot create containers on their kernels.
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-589ea77-metrics-server-0.9.0-k8s-docker-2026-10-16T00:50:51Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-589ea77-contour-1.33.7-k8s-docker-2026-10-16T00:50:50Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-589ea77-cert-manager-1.21.2-k8s-docker-2026-10-16T00:50:56Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-589ea77-ekco-0.28.17-k8s-docker-2026-10-16T00:50:57Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-589ea77-sonobuoy-0.57.5-k8s-docker-2026-10-16T00:53:17Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-589ea77-prometheus-0.94.1-91.8.0-k8s-docker-2026-10-16T00:54:42Z |
| done | ||
|
|
||
| while IFS= read -r image; do | ||
| if ! docker manifest inspect "$image" >/dev/null; then |
There was a problem hiding this comment.
Should fix before merging
Any docker manifest inspect failure gets reported as "publish this tag", so registry/transport errors get dressed up as a missing publish. Capture the exit status and stderr and categorize the failure instead.
There was a problem hiding this comment.
Should fix before merging
SecureBuild verification categorizes every docker manifest lookup failure as an unpublished tag.
| while IFS= read -r -d '' file; do | ||
| files+=("$file") | ||
| done < <(find "$dir" -type f \( -name Manifest -o -name '*.yaml' -o -name '*.sh' \) -print0) | ||
| bash ../../../bin/verify-securebuild-images.sh "$dir/Manifest" "$replacements" "${files[@]}" || return 1 |
There was a problem hiding this comment.
Should fix before merging
The new network-dependent verify step runs after remove_previous_versions, so a failed verify leaves every existing ekco version deleted and generation aborted. Reorder the script so the verify runs first, or only clean old versions once the new one has been verified.
| mkdir -p "${src}/assets" | ||
| curl -L -o "${src}/assets/sonobuoy.tar.gz" "https://github.com/vmware-tanzu/sonobuoy/releases/download/v${SONOBUOY_VERSION}/sonobuoy_${SONOBUOY_VERSION}_linux_amd64.tar.gz" | ||
| local download= | ||
| download="$(mktemp "${src}/assets/sonobuoy.tar.gz.XXXXXX")" |
There was a problem hiding this comment.
If you have the time
If mktemp fails for sonobuoy_binary, the script aborts with no message while every other failure path wraps errors in logFail. Route this one through logFail too so a failed install isn't silently unattributed.
| manifest="$1" | ||
| replacements="$2" | ||
| shift 2 | ||
| images="$(sed -E "$replacements" "$manifest" | awk '$1 == "image" && $3 ~ /^docker[.]io\/kurlsh\// {print $3}' | sort -u)" |
There was a problem hiding this comment.
If you have the time
The SecureBuild closure check at the top of this script doesn't catch upstream images that were silently left unrewritten. If an upstream ref skips the rewrite, the closure still validates. Worth asserting every upstream image was actually rewritten, or at least flagging any that weren't.
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-589ea77-rook-1.19.7-k8s-docker-2026-10-16T00:59:59Z |
|
Testgrid run Failure mode
Root cause
Entrypoint, user, and the Ask Can the SecureBuild rook-ceph image be rebuilt with package parity — at minimum (For the record: the 4 CentOS 9 stream failures in this run are |
|
The missing-storage-tool image defect is repaired and the rebuilt Added the util-linux subpackages ( Pulled and executed both published architectures. All nine reported missing binaries are present and executable. Verified manifests:
These are isolated runtime smoke checks; OSD provisioning and cluster install/upgrade still need the existing Testgrid scenarios. Rebuild the Rook add-on archive with the new image before rerunning—previously built online/air-gap artifacts and queued runs can still contain the old image despite the unchanged tag. No new Testgrid run was submitted as part of this image repair. |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-f1099ba-rook-1.19.7-k8s-docker-2026-10-16T06:05:31Z |
|
Follow-up: rebuilt image fixed the discover crash, but the rook-ceph image is still broken — Re-ran rook-only tests against the rebuilt image (run Reproduced on a CMX VM (Ubuntu 24.04, exact staging installer from the failed run). Operator log loops forever on: Rook's object-store controller execs Isolation (same live cluster, healthy OSD, 10 pools, RADOS I/O verified working from the operator pod):
So the SecureBuild Ask: rebuild |
|
@doomer-ai are you listening? |
| # Keep older releases on upstream images; use SecureBuild from this release onward. | ||
| function use_securebuild_images() { | ||
| local dir="../$CERT_MANAGER_VERSION" | ||
| [ "$(printf '%s\n' "1.21.2" "$CERT_MANAGER_VERSION" | sort -V | head -n 1)" = "1.21.2" ] || return 0 |
There was a problem hiding this comment.
@doomer-ai does this properly handle new versions of this addon going forward
There was a problem hiding this comment.
Yes, mechanically. The gate on line 8 is a lower bound (SecureBuild applies when CERT_MANAGER_VERSION sorts at or above 1.21.2), not a pinned version, so any newer version passes. The sed rules match quay.io/jetstack/cert-manager-* tags regardless of version, and generate() runs use_securebuild_images on every generation. The caveat: each new version still requires the corresponding docker.io/kurlsh images to be published — verify-securebuild-images.sh checks the Manifest rewrite and runs docker manifest inspect, so it fails loudly ("Publish this tag before generating") rather than silently shipping broken images. If upstream ever moves off quay.io/jetstack or renames the controller/cainjector/webhook/acmesolver images, the rewrite won't match and the same verification will error out instead of producing stale upstream references.
Sources: addons/cert-manager/template/generate.sh, addons/cert-manager/template/generate.sh, bin/verify-securebuild-images.sh
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-contour-1.33.7-k8s-docker-2026-10-16T21:00:39Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-ekco-0.28.17-k8s-docker-2026-10-16T21:00:46Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-metrics-server-0.9.0-k8s-docker-2026-10-16T21:01:12Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-sonobuoy-0.57.5-k8s-docker-2026-10-16T21:04:24Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-cert-manager-1.21.2-k8s-docker-2026-10-16T21:06:38Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-containerd-2.3.6-k8s-ctrd-2026-10-16T21:07:26Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-rook-1.19.7-k8s-docker-2026-10-16T21:08:25Z |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-7e18f2f-prometheus-0.94.1-91.8.0-k8s-docker-2026-10-16T21:10:15Z |
| if [ "$(printf '%s\n' "1.36.5" "$version" | sort -V | head -n 1)" = "1.36.5" ]; then | ||
| local upstream_pause= pause_image= | ||
| upstream_pause="$(awk '$1 == "image" && $2 == "pause" {print $3; exit}' "$manifest")" | ||
| case "$upstream_pause" in |
There was a problem hiding this comment.
Don’t merge until this is fixed
Silently ignoring an unrecognized pause ref means a typo'd or renamed upstream tag skips the SecureBuild gate without a trace. Fail loudly instead: match against an explicit allowlist and abort generation on anything else.
| local binary="/usr/local/lib/sonobuoy/sonobuoy" | ||
| if [ -f "${src}/sonobuoy.sh" ] && [ -x "$binary" ] && \ | ||
| [ "$("$binary" version 2>/dev/null | awk '/^Sonobuoy Version:/ {print $NF}')" = "v${SONOBUOY_VERSION}" ]; then | ||
| install -m 0755 "${src}/sonobuoy.sh" /usr/local/bin/sonobuoy |
There was a problem hiding this comment.
Should fix before merging
Both fast-path reinstall calls exit silently on failure — no logFail, no return-1 — so a broken install looks like success. Wrap them the same way the tar steps are wrapped so failures actually surface.
| curl -L -o "${src}/assets/sonobuoy.tar.gz" "https://github.com/vmware-tanzu/sonobuoy/releases/download/v${SONOBUOY_VERSION}/sonobuoy_${SONOBUOY_VERSION}_linux_amd64.tar.gz" | ||
| local download= | ||
| download="$(mktemp "${src}/assets/sonobuoy.tar.gz.XXXXXX")" | ||
| if ! curl -fL -o "$download" "https://github.com/vmware-tanzu/sonobuoy/releases/download/v${SONOBUOY_VERSION}/sonobuoy_${SONOBUOY_VERSION}_linux_amd64.tar.gz"; then |
There was a problem hiding this comment.
Should fix before merging
mktemp failure here gets logged as a download failure, so a staging-dir problem masquerades as a network issue. Check mktemp separately, e.g. if ! download="$(mktemp "${src}/assets/sonobuoy.tar.gz.XXXXXX")"; then logFail "Failed to create a staging file next to ${src}/assets/sonobuoy.tar.gz."; return 1; fi.
|
Doomer isn't automatically approving this pull request. Why: The Doomer review did not complete successfully. Reviewer failure details: - <code>registry.doomer.ai/library/review/engineering@sha256:2a50f498bff3fc16561ed0bec4fc7233633acd11900e80cf990a3207f0f41b9d [batch-001]</code>: <code>ModelReviewError: model_request_recovery_exhausted: Model request recovery exhausted after 3 attempt(s) (attempt_limit).</code>; Partial review: 1 review jobs failed; unsupported candidates, if any, were withheld. Completed reviewers' findings are retained. No clean-review opinion. Findings from completed reviewers do not establish that the whole review passed. Comment Alternatively, a reviewer can review and approve this pull request on GitHub. |
|
Updated image: Immutable multi-architecture reference: Follow-up to #6204 (comment): the rebuilt I reproduced the old image's The final published stable image was pulled by immutable digest and checked independently:
Published digests for
These checks verify the reported client hang is resolved in the rebuilt image; they are not a full Rook rollout or S3 end-to-end test. Live cluster metadata checks ran on amd64; both architectures received the runtime checks and scans. Existing Rook add-on archives still embed the old image and must be rebuilt before rerunning the existing Testgrid scenarios. Republishing the Docker tag alone does not update those archives. |
|
Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-6204-07a5359-rook-1.19.7-k8s-docker-2026-10-17T06:37:55Z |
This branch has not been deployed
3 participants
What this PR does / why we need it:
Switch the latest add-on versions to the SecureBuild images now published publicly to
kurlsh. Keep the runtime manifests, dynamically selected images, offline image bundles, and generation templates aligned in one PR.This integrates 31 published tags across 30 image repositories. Application versions remain the same; image tags drop upstream packaging suffixes where the SecureBuild tags use the plain application version.
Which issue(s) this PR fixes:
None.
Special notes for your reviewer:
kurlshwhile preserving each generated component tag. Packaging suffixes such as-distrolessand HAProxy’s Alpine suffix are removed to match SecureBuild tag naming. All eight generators verify every selecteddocker.io/kurlshtag withdocker manifest inspectbefore rewriting files; unavailable tags or registry lookup failures stop generation. Future version updates must publish the matching SecureBuild tags before generation. The rewrite rules live directly in each generator; there are no separate.sedfiles.docker.io/prefixes from 1.33.7 onward and normalizes fully qualified and bare upstream references. Older versions retain the historical Docker EE prefix handling.runandgen; explicit CLI image options can override those defaults. Historical releases keep their original CLI installation and defaults. Generation validates both image entries and fails explicitly on missing, malformed, duplicate, or unexpected references. Conformance images remain unchanged.replicated/ekco:v0.28.17image already uses SecureBuild and stays in place. HAProxy retains the existing configuration mount path. Its SecureBuild image now starts with-W -db, matching the upstream container's foreground behavior even when EKCO's generated config containsdaemon. The corrected multi-architecture image was republished atdocker.io/kurlsh/haproxy:3.4.6(index digestsha256:31236dc94c90a52b0ce99d122e8ad338d706abc4f6ae9beb8e1588fa87cf8e40).kurlshstarting at 1.36.5 and retains the pause version returned by kubeadm for future releases. Generation verifies the published pause tag withdocker manifest inspectand fails before rewriting the Manifest if it cannot be verified; it does not silently fall back to an upstream image. Containerd already reads the Kubernetes package Manifest when selecting its sandbox image. Existing per-version control-plane image overrides are unchanged.Steps to reproduce
Local validation completed:
git diff --checkpasses, andscripts/Manifestmatches its test fixture./sonobuoycompatibility symlink required by its generated pods; anonymous amd64/arm64 pulls verify that path.Build and shell CI passed on
cfb56dceb. Full Testgrid install/upgrade verification remains incomplete: the earlier EKCO matrix has cancelled cases and runtime/runner failures, and the other matrices were skipped. The rebuilt HAProxy default startup and port 6444 listener passed on both amd64 and arm64; SecureBuild reports no vulnerabilities for either architecture. This PR groups the migration for integration testing; no new test cases or coverage expansion are added.Does this PR introduce a user-facing change?
Does this PR require documentation?
NONE