Repository navigation
test(oracle-gaps): two checks at once are recorded by role, not by process order (#82) - #126
Merged
Merged
Conversation
…er one reuse Two checks at once reach one daemon, which runs one check at a time in the order their requests arrive. That order is a race between two processes and no part of the contract, so the driver recorded the pair by which process it started first, and the determinism unit of "an edit sequence through the daemon" saw the test-output note on either check (#82, #92). The driver now records the pair by what each check did: the check that computed more runs comes first. The later check of one state owes the reuse law that a recheck owes (unchanged reuse): it computes no run that was not cut off, where no test reads the clock. A new reach state, "of two checks at once, one computed and the other reused", is required with law 9's states. Co-authored-by: Isaac <no-reply@databricks.com>
…nce-race Co-authored-by: Isaac <no-reply@databricks.com>
The second witness of #82's race: with the pair recorded by role, a GREEN run of the determinism unit still differed, in the done line. Each of two checks at once gets its own output directory (out-daemon, out-daemon-2), and the done line names it, so the trace still told which process the daemon ran first. The stream records now name the run's own directory <out>, as its notes already did; a line that names another run's directory still shows it. Co-authored-by: Isaac <no-reply@databricks.com>
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
commit: |
…nce-race Co-authored-by: Isaac <no-reply@databricks.com>
…e the done line Review of #126: - F2: the reach state for two checks at once now needs a held overlap. A constructed-only field, holdOverlap (fc.constant(false) in the drawn record, recorded in the fingerprint, set by the edit sequence through the daemon), arms the test-only hold ORACLE_GAPS_TEST_HOLD_OVERLAP. The state is entered only when the daemon reports that it held its first check until it had accepted the second request. - F3: computedRuns reads the CLI's count of runs that ended without a cut (evidence.ran), not computed - cut, which counts cut events. - F4: a run's own output directory is named <out> only as a whole path. - F5: a completed check through the daemon names its own report in its done line. - F6: ties between the two checks are ordered by their records. This commit is oracle-only: against the CLI without the hold and the count, the constructed history cannot enter its state (RED). Co-authored-by: Isaac <no-reply@databricks.com>
…t of runs that ended without a cut - ORACLE_GAPS_TEST_HOLD_OVERLAP (a path): a daemon holds the first check it runs until it has accepted a second check request, then writes that count to the path. Once per daemon, released by the count, never by time. It changes no outcome (G43, R57). Documented in the README's test hooks. - evidence.ran in results.json: the units a check computed that ended without a cut. `cut` counts cut events, so computed - cut is not a count of runs (review of #126, F3). Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…nce-race Co-authored-by: Isaac <no-reply@databricks.com>
…n report Review of #126, D3: a calibration whose corruption makes a completed check through the daemon name another check's report on both streams, so that only the new done-line law rejects it. Bound to the constructed history "an unchanged check through the daemon under a config's pre plugin". Co-authored-by: Isaac <no-reply@databricks.com>
…ive the overlap hold a loud backstop Review of #126: - D1: evidence.ran is written to results.json only under the retention test hook, as #61 did for its batch counts. A normal run's results.json is main's. The changeset is dropped. - D2: a batch unit that spun past its call budget is timed out, so it no longer counts as ran, as unit() counts it. - D4: the overlap hold moves to overlap-hold.mjs, which the daemon loads only when ORACLE_GAPS_TEST_HOLD_OVERLAP is set. Its backstop fails loudly: with no second check request within 120 s, the held check fails with an error that names the hook, and the daemon closes, releasing its socket and lock. Co-authored-by: Isaac <no-reply@databricks.com>
…nce-race Resolves #57's merge: its drawn fields come first and holdOverlap stays last (also in the fingerprint, and in constructedNeutral), and law 9's check record keeps #57's log, outDir, reported and reportState. A run's own output directory is now named <out> throughout readRunIn, as #57 names a derived one, so outDir and the note-log line are as independent of which process the daemon ran first as the streams are. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 17, 2026
KyleAMathews
added a commit
that referenced
this pull request
Nov 17, 2026
The daemon's overlap hold sits before the check, which keeps its beats; the incremental observation extends #126's checkRecord with the held, rewritten and slow-start flags; holdOverlap joins the hook fields that the take-back and the canary hold neutral. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 17, 2026
#126 records two checks at once by role. Conflicts: check.mjs keeps #99's deletion of confirmAlone (main's evidence.ran stays in unit and unitsBatch, and its comment drops the confirmation it no longer has); the history grammar keeps main's holdOverlap before #99's hangMs, and the fingerprint lists record main's constant line in that place. Co-authored-by: Isaac <no-reply@databricks.com>
8 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #92: this resolves its edit-sequence instance (the second failure in #82, "an edit sequence through the daemon"). #92 stays open for its other instance, #57's single-CLI history
a daemon check with the default output directory, which this race cannot cause (comment). #82 stays open for its hang history (observation 18, "reach"). #98 stays open until #90's exclusion list drops the unit.What was wrong
The history begins with two checks at once (
concurrentStart). The daemon runs one check at a time, from the FIFO queue indaemon.mjs(queue = queue.then(...), and its header: "One check runs at a time"). Whichever check it runs first computes every unit, and the other reuses them. The driver recorded the pair in the order it started the two processes. Thenote test outputline belongs to the check that ran the tests, so the trace showed which process reached the daemon first, and the determinism unit failed whenever two runs took different orders.Option chosen: B, the model accepts both orders, with the overlap held
The order of two concurrent checks is a race between two processes. The contract does not promise it: the README promises reuse while inputs are unchanged and says nothing about which of two checks runs first. Option A would pin an order that the law leaves free. Option B pins only what the law needs:
byRole). Each check is still judged in full, so nothing is relaxed.unchanged reuse. It computes no run that ended without a cut, exactly as arecheckdoes. "One computed, the other reused" was not judged before this PR.ORACLE_GAPS_TEST_HOLD_OVERLAP(G43; documented in the README's test hooks), makes a daemon hold the first check it runs until it has accepted a second check request. Then it writes that count to a mark. It holds once per daemon, is released by the count and never by time, and changes no outcome. Its code is inoverlap-hold.mjs, which the daemon loads only when the variable is set, as fix(oracle-gaps): retire a batch's copies from Vitest's state #61's retention hook is. Its backstop fails loudly: with no second request within 120 s, the held check fails with an error that names the hook, and the daemon closes, releasing its socket and lock. It is switched on only by a constructed-only field,holdOverlap:fc.constant(false)in the drawn record, appended last, recorded inhistory-fingerprint.json, and returned tofalseby the take-back. Only the edit sequence through the daemon sets it. The new required state is entered only when the mark shows the held overlap. Two sequential checks cannot enter it. The hold pins the overlap, not the order: either check may still be the one the daemon runs first.computed − cutcounted cut events, so it could go below zero and hide a run that was computed again. A cut copy confirmed alone adds a cut and no computed run (check.mjs). This needed a production counter:evidence.ran, the units a check computed that ended without a cut. A copy that spun past its call budget is timed out and does not count, on the batch path as inunit(). Following fix(oracle-gaps): retire a batch's copies from Vitest's state #61's precedent,ranis written toresults.jsononly under the retention test hook, which every oracle run sets. A normal run'sresults.jsonis main's, so there is no changeset. On a fixture, the evidence block without the hook is{"reused":0,"computed":5,"cut":0,"idle":0,"spin":0,"rerun":0,"unchanged":0,"budget":0}on both the head and main at 153539b. With the hook, the head adds"ran":5.<out>throughout its record, and only as a whole path. That covers its streams, itsoutDir, and its stream-log line (feat(oracle-gaps): write the stream to <out>/stream.log and name it in the first line #57), as feat(oracle-gaps): write the stream to <out>/stream.log and name it in the first line #57 names a derived one. The escaped spelling of a path that holds a control character is matched too. A sibling whose name begins with it (out-daemon-2-3besideout-daemon-2) keeps its own name.… report <out>/summary.md). A calibration,a check through the daemon names another check's report, rewrites a completed daemon check's done line on both streams to name another check's report. The law rejects it.Proposed register entry (C; the coordinator records it)
v21.2 IDs this touches: I8 (incremental equals cold), I18 (determinism compares results.json and the fixed lines), R52 (progress lines are not outcome), R57 and G43 (test-only hooks; holds released by the driver or a count, never by time: C-30, now M), R9 / P9 (oracle-first), R42 / M-20 (main's determinism failures are filed), P4 (random seeds judge the new law), P10 (the done line, the new law of point 6), G42 / G34 (the daemon as a cache over the log).
Evidence
The Arca pool was unreachable for the final reruns: every Arclet's ssh connection timed out. So the reruns at head 59f3da6 (main at 153539b, #57, merged) ran on this machine, through
/tmp/e2e-lock.sh(2 slots,--maxWorkers=2). Earlier rounds ran on Arca, as named.of two checks held in the daemon at once, one computed and the other reusedis entered 1 time in each run offixed seed 20261105: an edit sequence through the daemon(local, 3 of 3), and ofrandom seed: an edit sequence through the daemon(local, 2 of 2). The framework fails a constructed history that does not enter a state it names.fixed seed 20261105: an edit sequence through the daemonfails with "did not enter of two checks held in the daemon at once, one computed and the other reused" (Arcaf82-e3-red-1).note test outputline (Arcaf82-d2-red-*). At 9aed105: 4 of 10. A forced-order scratch driver fails 2 of 2 on main.determinism: constructed history: an edit sequence through the daemon: 20 of 20 pass, about 110 s each.a check through the daemon names another check's report: passes.law: unchanged reuse, "the check through the daemon at start (a second check at once), of a state that nothing changed, computed 8 runs that were not cut off, owed none". The hold makes the second request arrive while the first check is in flight by a count, so the kill doesn't depend on timing.overlap-hold.mjsdirectly. It holds until the second request is accepted, then writes2. A secondhold()is a no-op. With timers fired at once, the backstop rejects with the named error and writes no mark. No e2e history reaches the backstop: it would need a second check that never arrives.two checks started the daemon at onceis not gated on overlap.ranandcomputed − cuthas no e2e witness.g1-ff5 triage: filed as #124
A production counterexample observed through a complete artifact (C-31). It is not a driver timing assumption: at
startthe history runs one CLI, a plainrunCliwith no hook or hold. The daemon'sresults.jsonblockedsut.ts:M1on a failingtests/a.test.tsbaseline that the cold run passed. The cause is not determined. #124 now has the full counterexample history (M "Replay by history").CI
Run 44084587185 at 59f3da6 is green overall, after one job was rerun.
every outcome classdid not entera hang cut off for idleness) and 5/10 (every outcome class, the abort stop, the win32 abort stop) fail the same way on main 153539b (run 44074811051). They are non-blocking (C (c), Windows static and e2e gap oracles: remaining failures by class (non-blocking on #58) #72), and tracked in P12: Windows e2e shard 2/10 (every outcome class alone) takes 10 min #118 and fix(oracle-gaps): supervise the check on win32; log each CLI run in the e2e driver #79.At d65293e, run 43919288577 was green overall.
every outcome classdid not entera hang cut off for idleness).Weight
Code in the test-time package only, so zero shipped bytes (C "Zero shipped bytes"):
daemon.mjs+16 lines: loading and calling the hold.overlap-hold.mjs(new, 60 lines), loaded only under its hook.check.mjs+4 hook-gated counters and a comment.Tests and docs:
gap-report-oracle.property.test.mjs+196/−37.This pull request and its description were written by Isaac.