Skip to content

test(oracle-gaps): two checks at once are recorded by role, not by process order (#82) - #126

Merged
KyleAMathews merged 11 commits into
mainfrom
fix/daemon-edit-sequence-race
Nov 17, 2026
Merged

KyleAMathews merged 11 commits into
mainfrom
fix/daemon-edit-sequence-race

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Refs #92: this resolves its edit-sequence instance (the second failure in #82, "an edit sequence through the daemon"). #92 stays open for its other instance, #57's single-CLI history a daemon check with the default output directory, which this race cannot cause (comment). #82 stays open for its hang history (observation 18, "reach"). #98 stays open until #90's exclusion list drops the unit.

What was wrong

The history begins with two checks at once (concurrentStart). The daemon runs one check at a time, from the FIFO queue in daemon.mjs (queue = queue.then(...), and its header: "One check runs at a time"). Whichever check it runs first computes every unit, and the other reuses them. The driver recorded the pair in the order it started the two processes. The note test output line belongs to the check that ran the tests, so the trace showed which process reached the daemon first, and the determinism unit failed whenever two runs took different orders.

Option chosen: B, the model accepts both orders, with the overlap held

The order of two concurrent checks is a race between two processes. The contract does not promise it: the README promises reuse while inputs are unchanged and says nothing about which of two checks runs first. Option A would pin an order that the law leaves free. Option B pins only what the law needs:

  1. The pair is recorded by role, in a total order. The check that computed more runs comes first, as the daemon's earlier check. Ties are ordered by each check's serialized record (byRole). Each check is still judged in full, so nothing is relaxed.
  2. The later check owes unchanged reuse. It computes no run that ended without a cut, exactly as a recheck does. "One computed, the other reused" was not judged before this PR.
  3. The overlap is held, not timed (R57). A new test-only hold, ORACLE_GAPS_TEST_HOLD_OVERLAP (G43; documented in the README's test hooks), makes a daemon hold the first check it runs until it has accepted a second check request. Then it writes that count to a mark. It holds once per daemon, is released by the count and never by time, and changes no outcome. Its code is in overlap-hold.mjs, which the daemon loads only when the variable is set, as fix(oracle-gaps): retire a batch's copies from Vitest's state #61's retention hook is. Its backstop fails loudly: with no second request within 120 s, the held check fails with an error that names the hook, and the daemon closes, releasing its socket and lock. It is switched on only by a constructed-only field, holdOverlap: fc.constant(false) in the drawn record, appended last, recorded in history-fingerprint.json, and returned to false by the take-back. Only the edit sequence through the daemon sets it. The new required state is entered only when the mark shows the held overlap. Two sequential checks cannot enter it. The hold pins the overlap, not the order: either check may still be the one the daemon runs first.
  4. Runs are counted, not cut events. The driver's old measure computed − cut counted cut events, so it could go below zero and hide a run that was computed again. A cut copy confirmed alone adds a cut and no computed run (check.mjs). This needed a production counter: evidence.ran, the units a check computed that ended without a cut. A copy that spun past its call budget is timed out and does not count, on the batch path as in unit(). Following fix(oracle-gaps): retire a batch's copies from Vitest's state #61's precedent, ran is written to results.json only under the retention test hook, which every oracle run sets. A normal run's results.json is main's, so there is no changeset. On a fixture, the evidence block without the hook is {"reused":0,"computed":5,"cut":0,"idle":0,"spin":0,"rerun":0,"unchanged":0,"budget":0} on both the head and main at 153539b. With the hook, the head adds "ran":5.
  5. Each run's own output directory is named <out> throughout its record, and only as a whole path. That covers its streams, its outDir, and its stream-log line (feat(oracle-gaps): write the stream to <out>/stream.log and name it in the first line #57), as feat(oracle-gaps): write the stream to <out>/stream.log and name it in the first line #57 names a derived one. The escaped spelling of a path that holds a control character is matched too. A sibling whose name begins with it (out-daemon-2-3 beside out-daemon-2) keeps its own name.
  6. New law: a completed check through the daemon names its own report in its done line (… report <out>/summary.md). A calibration, a check through the daemon names another check's report, rewrites a completed daemon check's done line on both streams to name another check's report. The law rejects it.

Proposed register entry (C; the coordinator records it)

Two checks at once (#126): of two checks of one state through one daemon, the one the daemon runs second computes no run that ends without a cut (law unchanged reuse, judged by the CLI's evidence.ran). Authority: the README's reuse contract ("a run is reused while its inputs are unchanged"). The law relies on an implementation fact, not a promise: the daemon serves one check at a time from the FIFO queue in daemon.mjs, and the earlier check's units are stored and flushed before its reply. Whether "one check at a time" is contract is flagged for M. It inherits C-7's exemption: tests that read the clock (t.flaky) are not judged. C-7 is not among the 15 C entries ratified on 2026-11-16 (v21.2 §12), so the exemption is pending ratification. Domain: the constructed edit sequence through the daemon, with its overlap held, and every generated history that draws concurrentStart, crashAtOnce or cliAtOnce, so random seeds judge the law on every PR (P4). Which check the daemon runs first is outside the contract, so traces record the pair by role.

v21.2 IDs this touches: I8 (incremental equals cold), I18 (determinism compares results.json and the fixed lines), R52 (progress lines are not outcome), R57 and G43 (test-only hooks; holds released by the driver or a count, never by time: C-30, now M), R9 / P9 (oracle-first), R42 / M-20 (main's determinism failures are filed), P4 (random seeds judge the new law), P10 (the done line, the new law of point 6), G42 / G34 (the daemon as a cache over the log).

Evidence

The Arca pool was unreachable for the final reruns: every Arclet's ssh connection timed out. So the reruns at head 59f3da6 (main at 153539b, #57, merged) ran on this machine, through /tmp/e2e-lock.sh (2 slots, --maxWorkers=2). Earlier rounds ran on Arca, as named.

  • Law: of two checks at once on one state, the later computes no run that ends without a cut. The pair is recorded by role. A completed daemon check names its own report. The authority and scope are in the register entry above.
  • Reach: the required state of two checks held in the daemon at once, one computed and the other reused is entered 1 time in each run of fixed seed 20261105: an edit sequence through the daemon (local, 3 of 3), and of random seed: an edit sequence through the daemon (local, 2 of 2). The framework fails a constructed history that does not enter a state it names.
  • RED:
    • Oracle-only commit 24afe68 (R9): fixed seed 20261105: an edit sequence through the daemon fails with "did not enter of two checks held in the daemon at once, one computed and the other reused" (Arca f82-e3-red-1).
    • Determinism unit on main (fac63df): 8 of 10 fail at observation 4, each differing only in the note test output line (Arca f82-d2-red-*). At 9aed105: 4 of 10. A forced-order scratch driver fails 2 of 2 on main.
    • Done-line law: the calibration above passes, so the law rejects the corrupted trace. The framework fails a calibration whose corruption applies to nothing.
  • GREEN at 59f3da6 (local):
    • Determinism unit determinism: constructed history: an edit sequence through the daemon: 20 of 20 pass, about 110 s each.
    • Fixed-seed unit: 3 of 3 pass. Random-seed unit: 2 of 2 pass.
    • Calibration a check through the daemon names another check's report: passes.
    • At d65293e, on Arca: the determinism unit passed 20 of 20, and the forced-order scratch driver (hold off) passed 2 of 2.
  • Mutant ("both checks compute"): in the daemon, a check whose request arrives while another check's connection is in flight runs against an empty store. Its results equal cold.
    • At 59f3da6, under the held overlap, it is killed 3 of 3 (local): law: unchanged reuse, "the check through the daemon at start (a second check at once), of a state that nothing changed, computed 8 runs that were not cut off, owed none". The hold makes the second request arrive while the first check is in flight by a count, so the kill doesn't depend on timing.
    • At fac63df, without the law, it survives.
  • Hold backstop: a script drove overlap-hold.mjs directly. It holds until the second request is accepted, then writes 2. A second hold() is a no-op. With timers fired at once, the backstop rejects with the named error and writes no mark. No e2e history reaches the backstop: it would need a second check that never arrives.
  • Limits:
    • Only the constructed edit sequence holds its overlap. Generated histories with two checks at once still judge the reuse law, but their overlap is neither held nor measured, so they claim no reach.
    • The older state two checks started the daemon at once is not gated on overlap.
    • No history reaches a cut inside a pair of checks at once, so the difference between ran and computed − cut has no e2e witness.
    • The reuse law trusts the CLI's own count (review D5): the later check's captured test output, absent in this history, could back it, and is not yet observed.
    • Determinism units run locally only (L5; ci(oracles): run the e2e oracle's random and determinism units on every PR #90 not merged).
    • The final reruns ran on a laptop at load 3 to 6, not on the Arclet pool.

g1-ff5 triage: filed as #124

A production counterexample observed through a complete artifact (C-31). It is not a driver timing assumption: at start the history runs one CLI, a plain runCli with no hook or hold. The daemon's results.json blocked sut.ts:M1 on a failing tests/a.test.ts baseline that the cold run passed. The cause is not determined. #124 now has the full counterexample history (M "Replay by history").

CI

Run 44084587185 at 59f3da6 is green overall, after one job was rerun.

At d65293e, run 43919288577 was green overall.

Weight

Code in the test-time package only, so zero shipped bytes (C "Zero shipped bytes"):

  • daemon.mjs +16 lines: loading and calling the hold.
  • overlap-hold.mjs (new, 60 lines), loaded only under its hook.
  • check.mjs +4 hook-gated counters and a comment.

Tests and docs:

  • gap-report-oracle.property.test.mjs +196/−37.
  • The model's constant field.
  • The fingerprint.
  • The README hook text.

This pull request and its description were written by Isaac.

KyleAMathews and others added 3 commits November 16, 2026 13:29
…er one reuse

Two checks at once reach one daemon, which runs one check at a time in the
order their requests arrive. That order is a race between two processes and
no part of the contract, so the driver recorded the pair by which process it
started first, and the determinism unit of "an edit sequence through the
daemon" saw the test-output note on either check (#82, #92).

The driver now records the pair by what each check did: the check that
computed more runs comes first. The later check of one state owes the reuse
law that a recheck owes (unchanged reuse): it computes no run that was not
cut off, where no test reads the clock. A new reach state, "of two checks at
once, one computed and the other reused", is required with law 9's states.

Co-authored-by: Isaac <no-reply@databricks.com>
…nce-race

Co-authored-by: Isaac <no-reply@databricks.com>
The second witness of #82's race: with the pair recorded by role, a GREEN
run of the determinism unit still differed, in the done line. Each of two
checks at once gets its own output directory (out-daemon, out-daemon-2),
and the done line names it, so the trace still told which process the
daemon ran first. The stream records now name the run's own directory
<out>, as its notes already did; a line that names another run's directory
still shows it.

Co-authored-by: Isaac <no-reply@databricks.com>
@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 59f3da6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 00d3083e-9f37-4145-9c71-47daf4bdc61d
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pkg-pr-new

pkg-pr-new Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@126
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@126

commit: 59f3da6

KyleAMathews and others added 4 commits November 16, 2026 17:24
…nce-race

Co-authored-by: Isaac <no-reply@databricks.com>
…e the done line

Review of #126:
- F2: the reach state for two checks at once now needs a held overlap. A
  constructed-only field, holdOverlap (fc.constant(false) in the drawn
  record, recorded in the fingerprint, set by the edit sequence through
  the daemon), arms the test-only hold ORACLE_GAPS_TEST_HOLD_OVERLAP. The
  state is entered only when the daemon reports that it held its first
  check until it had accepted the second request.
- F3: computedRuns reads the CLI's count of runs that ended without a
  cut (evidence.ran), not computed - cut, which counts cut events.
- F4: a run's own output directory is named <out> only as a whole path.
- F5: a completed check through the daemon names its own report in its
  done line.
- F6: ties between the two checks are ordered by their records.

This commit is oracle-only: against the CLI without the hold and the count,
the constructed history cannot enter its state (RED).

Co-authored-by: Isaac <no-reply@databricks.com>
…t of runs that ended without a cut

- ORACLE_GAPS_TEST_HOLD_OVERLAP (a path): a daemon holds the first check it
  runs until it has accepted a second check request, then writes that count
  to the path. Once per daemon, released by the count, never by time. It
  changes no outcome (G43, R57). Documented in the README's test hooks.
- evidence.ran in results.json: the units a check computed that ended
  without a cut. `cut` counts cut events, so computed - cut is not a count
  of runs (review of #126, F3).

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews and others added 4 commits November 17, 2026 11:20
…nce-race

Co-authored-by: Isaac <no-reply@databricks.com>
…n report

Review of #126, D3: a calibration whose corruption makes a completed check
through the daemon name another check's report on both streams, so that
only the new done-line law rejects it. Bound to the constructed history "an
unchanged check through the daemon under a config's pre plugin".

Co-authored-by: Isaac <no-reply@databricks.com>
…ive the overlap hold a loud backstop

Review of #126:
- D1: evidence.ran is written to results.json only under the retention
  test hook, as #61 did for its batch counts. A normal run's results.json
  is main's. The changeset is dropped.
- D2: a batch unit that spun past its call budget is timed out, so it no
  longer counts as ran, as unit() counts it.
- D4: the overlap hold moves to overlap-hold.mjs, which the daemon loads
  only when ORACLE_GAPS_TEST_HOLD_OVERLAP is set. Its backstop fails
  loudly: with no second check request within 120 s, the held check fails
  with an error that names the hook, and the daemon closes, releasing its
  socket and lock.

Co-authored-by: Isaac <no-reply@databricks.com>
…nce-race

Resolves #57's merge: its drawn fields come first and holdOverlap stays last
(also in the fingerprint, and in constructedNeutral), and law 9's check
record keeps #57's log, outDir, reported and reportState. A run's own output
directory is now named <out> throughout readRunIn, as #57 names a derived
one, so outDir and the note-log line are as independent of which process
the daemon ran first as the streams are.

Co-authored-by: Isaac <no-reply@databricks.com>
@KyleAMathews
KyleAMathews merged commit 5523123 into main Nov 17, 2026
91 of 96 checks passed
@KyleAMathews
KyleAMathews deleted the fix/daemon-edit-sequence-race branch November 17, 2026 15:39
KyleAMathews added a commit that referenced this pull request Nov 17, 2026
The daemon's overlap hold sits before the check, which keeps its beats;
the incremental observation extends #126's checkRecord with the held,
rewritten and slow-start flags; holdOverlap joins the hook fields that the
take-back and the canary hold neutral.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 17, 2026
#126 records two checks at once by role. Conflicts: check.mjs keeps #99's
deletion of confirmAlone (main's evidence.ran stays in unit and unitsBatch,
and its comment drops the confirmation it no longer has); the history
grammar keeps main's holdOverlap before #99's hangMs, and the fingerprint
lists record main's constant line in that place.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant