Skip to content

Validate e2e consensus traces against the Lean model in CI - #8732

Draft
cjen1-msft wants to merge 11 commits into
cjen1-msft-symmetrical-bassoonfrom
cjen1-msft-laughing-memory
Draft

cjen1-msft wants to merge 11 commits into
cjen1-msft-symmetrical-bassoonfrom
cjen1-msft-laughing-memory

Conversation

@cjen1-msft

@cjen1-msft cjen1-msft commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #8735.

The Lean model of the CCF consensus protocol is only checked against cft_driver scenarios, which run every node in one process on a scripted schedule. This PR checks it against real networks. Every e2e test now records its nodes' ccf.cft trace events, and a new CI job replays each complete capture against the model, failing if the implementation takes a step the model does not allow. The existing e2e tests, including reconfiguration, elections and partitions, become a conformance check between the model and the implementation on every PR.

A capture is replayed only if it is known to be complete. Otherwise it is recorded as inconclusive, with reasons, and skipped: for example when a node was killed, or when the network recovered a service, which replay does not model because it bootstraps from a fresh genesis.

How it works and how to use it
  • add_e2e_test passes --capture-cft-trace to every test. lts_compatibility opts out with NO_CFT_TRACE because released nodes do not support the capture sink, and benchmarks (LABEL perf) are excluded because they expect their own collector to be the nodes' only sink.
  • Each Network starts one local recording Fluentd collector when it first configures a node, and gives each node an extra sink that accepts only ccf.cft, keeping any sinks the test configured.
  • stop_all_nodes writes <workspace>/<label>_trace/<n>/. cft.ndjson has one {process_id, h_ts, msg} record per line from every node. manifest.json has complete, reasons, nodes and records_per_process. Networks that share a label get separate <n> folders, and a test process removes an earlier run's captures the first time it uses a trace folder.
  • A capture is complete only if every node the network configured, including retired ones, exited with status 0, logged 0 dropped events for the capture sink and closed its connection with no partial message; the capture has records from as many processes as nodes; and no node recovered a service.
  • A network whose nodes are stopped one by one, not through stop_all_nodes, writes no capture.
  • In CI, the Virtual jobs upload captures in their logs-* artifacts. The Replay e2e consensus traces job builds consensus-replay and runs tests/cft_trace_validation.py over them. It fails if a complete capture is rejected, or if there is none to replay.
  • Locally: python3 tests/cft_trace_validation.py <consensus-replay> build/workspace.
  • Most of the stop_all_nodes diff is re-indentation for a try/finally; review it with whitespace hidden.

Alternatives

  • Replay inside each test when its network stops, as the recovery trace replayer does: every e2e job would need a Lean toolchain, and replay time would count against test timeouts. Replaying the uploaded captures in one separate job leaves the test jobs as they are.
  • Fail a test whose capture is incomplete: tests that kill or restart nodes would fail for reasons unrelated to what they test.
  • Opt in per test: coverage would depend on remembering the argument for each new test, so capture is on by default with an explicit opt-out.

@cjen1-msft
cjen1-msft force-pushed the cjen1-msft-laughing-memory branch from 74c28e9 to 973e0e8 Compare November 10, 2026 17:16
@cjen1-msft
cjen1-msft changed the base branch from main to cjen1-msft-symmetrical-bassoon November 16, 2026 15:47
@cjen1-msft
cjen1-msft force-pushed the cjen1-msft-laughing-memory branch from 973e0e8 to 4255789 Compare November 16, 2026 15:47
cjen1-msft and others added 11 commits November 16, 2026 07:54
Each stream now holds one node's events, so a rule can no longer consume
another node's event and `event` no longer needs the group's head.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add --capture-cft-trace to the e2e arguments. When set, each Network
lazily starts a recording Fluentd collector when it first configures a
node, appends a ccf.cft-only sink to a per-node copy of the
observability configuration, and on stop_all_nodes (in a finally)
writes <label>_trace/<n>/cft.ndjson with a manifest. The capture is
complete only if every configured node, including retired ones, exited
cleanly and reported no dropped events for that sink, and every
connection reached EOF; otherwise it is inconclusive, which never fails
the test. Nodes from released versions are rejected.

Enable it for e2e_forwarding_smoke so CI exercises the capture path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exit only after an idle select that started after stop, so a
connection still queued in the listen backlog is accepted and read
rather than lost from a capture reported as complete. Spell out the
capture path and manifest in the argument help.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add <label>_trace/<n>/cft.ndjson and manifest.json to every workflow
step that uploads the e2e workspace logs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recorder keeps one stop state (the close deadline) and joins its
bounded thread directly. The sink helper updates the node's arguments
in place, the capture sink is identified as the node's last sink, and
a capture with fewer connections than configured nodes is inconclusive,
so a sink that never reached a node cannot be reported complete.
Workflows upload each capture folder with one glob.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Counting connections was only a lower bound: one node's extra
connection could hide a node whose configuration lost the capture sink.
Compare the recorded process IDs with the configured nodes instead, and
retry rather than drop a connection on a spurious wakeup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove an earlier run's captures the first time a test process uses a
trace folder, so a local re-run does not leave stale captures beside
new ones. Listen on ::1 for IPv6-only networks, whose node configs must
not contain IPv4 addresses. Mark a capture inconclusive when a node
recovered a service: replay bootstraps from a fresh genesis and does
not model recovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
add_e2e_test passes --capture-cft-trace unless the test sets
NO_CFT_TRACE or is a benchmark, which expects its own collector to be
the only sink. lts_compatibility opts out because released nodes do not
support the capture sink. The PID file check accepts the exporter's
shutdown line after the line it looks for.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A new job downloads the Virtual jobs' logs and replays every complete
capture with consensus-replay, failing if one is rejected or if there
is none to replay.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cjen1-msft
cjen1-msft force-pushed the cjen1-msft-laughing-memory branch from 6d42d75 to 68ef415 Compare November 16, 2026 16:08
@cjen1-msft cjen1-msft changed the title Add per-network ccf.cft trace capture to e2e tests Validate e2e consensus traces against the Lean model in CI Nov 16, 2026
@cjen1-msft
cjen1-msft force-pushed the cjen1-msft-symmetrical-bassoon branch from 917310c to 4ac693d Compare November 16, 2026 16:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant