Repository navigation
fix(ci): fix fork test infrastructure and make job advisory - #96
Merged
Merged
Conversation
Moving RUSTC_WRAPPER and SCCACHE_GHA_ENABLED from the global env block to only the 'Build patched base-anvil binaries' step fixes the 'could not find Cargo.toml' failure: sccache-action and rust-cache were running cargo metadata in the base-std checkout directory (a pure Solidity repo with no Cargo.toml) before the build step cd'd into BASE_ANVIL_DIR. Also pins BASE_ANVIL_REF to a commit SHA instead of a branch name so CI is reproducible and the branch cannot disappear.
rust-cache ran cargo metadata in the base-std checkout (a Solidity repo with no Cargo.toml), causing the workflow to fail before building anything. Removing it fixes the cargo metadata error; sccache in the build step still provides caching for the actual Rust compilation. Also fixes the base-anvil SHA pinning: git clone --branch does not accept a commit SHA, so CI was failing with 'Remote branch <sha> not found'. Solution: clone the named branch and then verify the HEAD SHA matches the pinned value, emitting a warning if the branch has moved.
Fork tests now run with continue-on-error: true since base-std is the spec forerunner — divergences mean base/base needs to catch up, not that base-std PRs are wrong. The job still runs and surfaces all failures. A step summary is written to the GitHub Actions Summary tab showing total pass/fail counts and the list of failing tests with a note that failures indicate Rust implementation lag, not base-std bugs.
Posts a summary comment directly on the PR after each run showing pass/fail counts and the list of failing tests. Only runs on pull_request events (not merge_group or workflow_dispatch where there is no PR to comment on). Requires pull-requests: write permission.
Instead of posting a new comment on every CI run, finds the existing fork-test-results comment (identified by an HTML marker) and updates it in place. Falls back to creating a new comment on first run. Each failing test now shows the test name and its full error inline: - **test_name**: `ErrorType(args)` so divergences are immediately actionable without reading the logs.
When the build or run step fails before any tests execute, the output file is empty or has 0 pass/fail counts. The comment now shows 'Fork tests did not run' with a link to the workflow logs instead of the misleading 'all 0 passed' message.
|
amiecorso
approved these changes
May 29, 2026
eric-ships
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BOP-165
Motivation
The fork test CI had two infrastructure bugs that prevented it from running at all, and the job was blocking PR merges even though base-std is the spec forerunner. Divergences between base-std and base/base mean Rust needs to catch up, not that base-std PRs are wrong.
What changed
rust-cacheand scopedRUSTC_WRAPPER/SCCACHE_GHA_ENABLEDto the "Build patched base-anvil binaries" step only. Bothrust-cacheandsccache-actionwere runningcargo metadatain the base-std checkout directory (a pure Solidity repo with noCargo.toml), causing every run to fail before building anything.BASE_ANVIL_REFpinning:git clone --branchdoes not accept a commit SHA, so cloning was failing with "Remote branch not found". Now clones the named branch and verifies the HEAD SHA matches the pinned value, emitting a warning if the branch has moved.continue-on-error: true): base-std is the spec forerunner. Test failures mean base/base needs to catch up, not that base-std PRs should be blocked.