Repository navigation
Validate e2e consensus traces against the Lean model in CI - #8732
Draft
cjen1-msft wants to merge 11 commits into
Draft
cjen1-msft wants to merge 11 commits into
cjen1-msft wants to merge 11 commits into
Conversation
cjen1-msft
force-pushed
the
cjen1-msft-laughing-memory
branch
from
November 10, 2026 17:16
74c28e9 to
973e0e8
Compare
cjen1-msft
changed the base branch from
main
to
cjen1-msft-symmetrical-bassoon
November 16, 2026 15:47
cjen1-msft
force-pushed
the
cjen1-msft-laughing-memory
branch
from
November 16, 2026 15:47
973e0e8 to
4255789
Compare
Each stream now holds one node's events, so a rule can no longer consume another node's event and `event` no longer needs the group's head. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add --capture-cft-trace to the e2e arguments. When set, each Network lazily starts a recording Fluentd collector when it first configures a node, appends a ccf.cft-only sink to a per-node copy of the observability configuration, and on stop_all_nodes (in a finally) writes <label>_trace/<n>/cft.ndjson with a manifest. The capture is complete only if every configured node, including retired ones, exited cleanly and reported no dropped events for that sink, and every connection reached EOF; otherwise it is inconclusive, which never fails the test. Nodes from released versions are rejected. Enable it for e2e_forwarding_smoke so CI exercises the capture path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exit only after an idle select that started after stop, so a connection still queued in the listen backlog is accepted and read rather than lost from a capture reported as complete. Spell out the capture path and manifest in the argument help. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add <label>_trace/<n>/cft.ndjson and manifest.json to every workflow step that uploads the e2e workspace logs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recorder keeps one stop state (the close deadline) and joins its bounded thread directly. The sink helper updates the node's arguments in place, the capture sink is identified as the node's last sink, and a capture with fewer connections than configured nodes is inconclusive, so a sink that never reached a node cannot be reported complete. Workflows upload each capture folder with one glob. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Counting connections was only a lower bound: one node's extra connection could hide a node whose configuration lost the capture sink. Compare the recorded process IDs with the configured nodes instead, and retry rather than drop a connection on a spurious wakeup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove an earlier run's captures the first time a test process uses a trace folder, so a local re-run does not leave stale captures beside new ones. Listen on ::1 for IPv6-only networks, whose node configs must not contain IPv4 addresses. Mark a capture inconclusive when a node recovered a service: replay bootstraps from a fresh genesis and does not model recovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
add_e2e_test passes --capture-cft-trace unless the test sets NO_CFT_TRACE or is a benchmark, which expects its own collector to be the only sink. lts_compatibility opts out because released nodes do not support the capture sink. The PID file check accepts the exporter's shutdown line after the line it looks for. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A new job downloads the Virtual jobs' logs and replays every complete capture with consensus-replay, failing if one is rejected or if there is none to replay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cjen1-msft
force-pushed
the
cjen1-msft-laughing-memory
branch
from
November 16, 2026 16:08
6d42d75 to
68ef415
Compare
cjen1-msft
force-pushed
the
cjen1-msft-symmetrical-bassoon
branch
from
November 16, 2026 16:37
917310c to
4ac693d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #8735.
The Lean model of the CCF consensus protocol is only checked against
cft_driverscenarios, which run every node in one process on a scripted schedule. This PR checks it against real networks. Every e2e test now records its nodes'ccf.cfttrace events, and a new CI job replays each complete capture against the model, failing if the implementation takes a step the model does not allow. The existing e2e tests, including reconfiguration, elections and partitions, become a conformance check between the model and the implementation on every PR.A capture is replayed only if it is known to be complete. Otherwise it is recorded as inconclusive, with reasons, and skipped: for example when a node was killed, or when the network recovered a service, which replay does not model because it bootstraps from a fresh genesis.
How it works and how to use it
add_e2e_testpasses--capture-cft-traceto every test.lts_compatibilityopts out withNO_CFT_TRACEbecause released nodes do not support the capture sink, and benchmarks (LABEL perf) are excluded because they expect their own collector to be the nodes' only sink.Networkstarts one local recording Fluentd collector when it first configures a node, and gives each node an extra sink that accepts onlyccf.cft, keeping any sinks the test configured.stop_all_nodeswrites<workspace>/<label>_trace/<n>/.cft.ndjsonhas one{process_id, h_ts, msg}record per line from every node.manifest.jsonhascomplete,reasons,nodesandrecords_per_process. Networks that share a label get separate<n>folders, and a test process removes an earlier run's captures the first time it uses a trace folder.0 dropped eventsfor the capture sink and closed its connection with no partial message; the capture has records from as many processes as nodes; and no node recovered a service.stop_all_nodes, writes no capture.logs-*artifacts. TheReplay e2e consensus tracesjob buildsconsensus-replayand runstests/cft_trace_validation.pyover them. It fails if a complete capture is rejected, or if there is none to replay.python3 tests/cft_trace_validation.py <consensus-replay> build/workspace.stop_all_nodesdiff is re-indentation for atry/finally; review it with whitespace hidden.Alternatives