Summarize each tests run in one report and keep only release artifacts

[avsej]

Problem

• Each test job publishes its own JUnit summary. The results of one run are spread over one job page per test configuration, and build failures appear only in the job list.
• Nothing says whether a failed job hit a test failure, a broken build, a timeout or an infrastructure problem. A cancelled run gets no summary.
• Every run keeps scripts-* and tests-* artifacts that only its own jobs read.
• Every run keeps each per-Ruby binary gem, although its fat gem holds the same binary.
• Failure artifacts list test/**/*.{log,xml}. upload-artifact does not expand braces, so the JUnit reports and test logs are never in them.
• A hung test runs until the job timeout. The job is then cancelled, and no logs are uploaded.
• The test log prints a test only when it ends, so a hung test never appears in it.
• The server and CNG jobs run the tests with the checkout's lib/ on the load path. They test the Ruby sources, not the packaged gem.
• Run scripts paste job outputs with ${{ }}. The gem versions come from lib/couchbase/version.rb, which a pull request controls.
• Checkouts keep the job token in .git/config for the rest of the job.
• Actions and the Alpine image are referenced by tags, which their owners can move. cbdinocluster is downloaded without a checksum.
• Nothing audits the workflows for unsafe patterns, reports a pinned action that falls behind its latest release, or reports a build image that falls out of security support.
• The musl gem is built on Alpine 3.18, which reached end of life on 2025-05-09.
• RuboCop offenses appear only in the job log.
• No CI job publishes test coverage.

Change

• Jobs that need scripts or tests check out the commit. Test jobs remove lib/ so every suite loads the installed gem.

• Test jobs upload their JUnit reports as junit-<job name> artifacts.

• The test log prints each test's name and UTC start time when it starts, and its result when it ends. A hung test is the last, unfinished line.

• Each Test step has its own timeout: 8 minutes in mock jobs, 20 minutes in cluster jobs. A hung test fails its step, so the logs are still uploaded.

• The test jobs and the summary job have a job timeout above the sum of their step timeouts, so a step times out before its job does.

• A summary job runs after all others, including on a cancelled run. bin/ci-summary.rb writes one step summary for the run:

  • each failed job with its cause, the step it stopped at and the error annotation;
  • a table of every test configuration, including jobs that did not run, left no report or a stale one, or reported no tests;
  • each failed test with the configurations it failed in and its output;
  • the slowest tests;
  • every job's result, with links;
  • the artifacts the run keeps.

• The summary annotates the first ten failed tests at the line of the test file where they failed. These annotations replace the mikepenz/action-junit-report steps.

• A tests-cleanup workflow, triggered by workflow_run, runs bin/ci-cleanup.rb. It deletes each per-Ruby gem once its fat gem exists, and the JUnit reports when the run succeeded. Reports stay after a failure, so "Re-run failed jobs" can still summarize the jobs it skips.

  • For workflow_run, GitHub takes the workflow and its checkout from the default branch. Pull request code therefore never runs with its actions: write token. It downloads no artifact from the run it cleans.
  • The script uses only the Ruby standard library and the runner's gh, so no gem is installed next to the token.
  • It refuses a run of another workflow or a run that has not completed. The repository and every id are validated before they reach an API path.

• The summary job, which runs pull request code, has read permissions only.

• Job outputs reach run scripts through environment variables rather than ${{ }}, and no checkout keeps the token in .git/config.

• Causes, from the first failed or cancelled step and the job's annotations:

  Cause	Decided by
  Test failures	Test failed and the report has failed tests
  Test run aborted	Test failed with no failed tests, or with no current report
  Gem install	Install failed
  Build	Precompile, Build …, Repackage or documentation failed
  Infrastructure	any other step, or a job that never started
  Cancelled	cancelled by a person, superseded by a newer run, or cancelled with no stated reason
  Timed out	the job or a step ran out of time
  Runner lost	the runner stopped responding

  A failed step keeps its cause when a later step times out or the run is cancelled.

• The report survives a failed checkout, artifact download, Ruby setup or API outage. Unavailable sections are replaced by a note, and without a report the job results from needs are written. The summary job fails whenever its report is incomplete.

• RuboCop also writes its offenses with the github formatter, so they appear as annotations.

• Every action and reusable workflow from another repository is pinned to a commit, commented with the most specific tag or the branch it was resolved from. The Alpine image is pinned to its digest. The nightly FIT runs are the exception. fit-tests.yml follows fit-cli's ci release tag, its latest stable release. It is also the channel the nightly installs. bin/check-action-pins.rb runs in linters:

  • It fails on a uses: not pinned that way, or on a container image without a digest.
  • It fails on a tag comment that does not match its commit, or on a pin it could not check.
  • It warns about a newer release tag or a moved branch, once per action and version.

• Pinned actions are updated to their latest releases, including major bumps of actions/setup-go (v7, in start-cng) and actions/checkout (v7, in validate-fit-performer).

• create-cluster verifies the checksum of the cbdinocluster binary it downloads, installs it into /usr/local/bin, and writes neither GITHUB_PATH nor GITHUB_ENV.

• The linters workflow runs zizmor and fails on findings of medium severity or above. The Bundler cache of the source job is an accepted finding: GitHub scopes caches by branch, so a pull request cannot write a cache that main or release* reads.

• The musl gem is built on the oldest Alpine release with security support, now 3.21. An older musl than the build one may lack symbols the extension uses.

• A coverage job on ubuntu-26.04 builds the extension from the checkout and runs the tests with lib/ on the load path, against the mock and against a cluster. SimpleCov merges both runs into one Cobertura report of lib/ and couchbase-opentelemetry/lib/, uploaded as the coverage artifact. The C++ extension is not measured.

• bin/check-alpine-image.rb runs in linters. It warns when the pinned release is past end of life or is not the oldest supported one. It also warns when its tag points at another image.

Known gaps

• Pinning covers this repository's uses: only. Actions that pinned actions fetch at run time are not covered: sdk-docker-build-action uses actions/checkout@v6.
• The nightly FIT runs follow fit-cli's ci release, in a job that can assume a cloud role.
• fit-cli fetches its installer from its own main branch.
• tests-cleanup runs only from the default branch, so this pull request cannot exercise it. After merge it can be run by hand on a run id through workflow_dispatch.
• The Alpine musl gem is built but not tested.

[Copilot]

🔵 Needs a closer look

It changes privileged cleanup, artifact lifecycle, test execution, and supply-chain controls across the full CI matrix.

0 open findings

🧠 Review effort: Balanced