Skip to content

fix(oracle-gaps): prefix an internal error's lines, and judge the stream law for them - #128

Merged
KyleAMathews merged 8 commits into
mainfrom
fix/internal-error-prefix
Nov 16, 2026
Merged

KyleAMathews merged 8 commits into
mainfrom
fix/internal-error-prefix

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #125.

Summary

The gap CLI's internal-error ending printed two lines with no oracle-gaps: prefix:

oracle-gaps internal error: <message>
This is a bug in oracle-gaps, not in your oracle. Its stack is in <log>.
oracle-gaps: done error, log <log>

The e2e oracle skipped the stream law for the one history that reaches an internal error: the constructed history "a batch whose run rejects" (#61). This PR fixes the oracle first, and then the CLI. The second round answers the review in pr128-review.md (F1 to F7) under the register's 2026-11-16 entries. It also merges origin/main (fe3a6b8, #121).

Oracle

Files: gap-report-oracle.property.test.mjs, gap-report-model-oracle.mjs, gap-report-static-oracle.property.test.mjs.

  • The stream law applies to every run. Law 15 now judges every run, including an internal error. The model gains the stream kind error.

  • One error line, tied to the log (general law, F6). A run whose done line is done error, log <log> (an endRun ending) owes exactly one error line, on stderr. That line must name the same log, and the log must be its output directory's error.log, spelled with escapes. The run owes no error line on stdout. Every other run owes none, including the supervisor's own exit 3.

  • What the rejected batch must still show (law 12). Exit 3, no stack frame on stderr, and an error.log. A new state, a run ended in an internal error, is required, and the rejected-batch history enters it.

  • Exempt lines are told by their own shapes (F1). Indentation alone exempts nothing. The reader (exemptBlocks) exempts:

    • a baseline failure's indented details;
    • a refusal block;
    • a Node warning, with its --trace-warnings frames and its hint;
    • V8's fatal block;
    • esbuild's ✘ [ERROR] and ▲ [WARNING] blocks;
    • Vite's [vite] line.

    A blank line before the end of the stream is a stray, unless it is inside one of these blocks or comes right before one.

  • Four new calibrations. Each runs the rejected batch, puts lines before its done line (foreign), and requires the law to name our own line alone:

    • a V8 fatal block, plus our own unprefixed line;
    • a traced warning's frames, plus our own unprefixed line;
    • an esbuild block (the real trace), plus our own unprefixed line;
    • our own indented line on its own.
  • Witnesses for the escaping (F4 and F5).

    • The rejected batch's fault message (the test-only retention-hook.mjs) contains a line break, followed by a forged done line.
    • On POSIX, the history's --out name contains a line break.
  • Limits and the static oracle (F2).

    • --help is exempt under the register's M entry "--help is exempt". The static oracle's Help law now cites that entry.
    • --stop-daemon is read the same way (C, pending ratification).
    • esbuild's block for a syntax error in an imported module is exempt under the M entry "Other tools' output is exempt". No pre-bundle is added.
  • Nits (F7). The law prose and the README now list every exempt form. The prose says the stdout-summary exemption is unused. The stale output: <dir> text in the model is corrected.

Production

Files: ending.mjs, check.mjs, report.mjs.

  • The internal error prints one line, then the done line:
    • oracle-gaps: error <message>: a bug in oracle-gaps, not in your oracle; its stack is in <log>
    • oracle-gaps: done error, log <log>
  • Control characters in the message and in every printed path print as \uXXXX escapes (escapeControls). These paths are:
    • in the error line and its done line;
    • in the refused done line;
    • in a completed run's done line, its results note, and its test-output note.
  • A message that already starts with oracle-gaps: no longer repeats the CLI's name.
  • The stack stays in error.log, and the exit status stays 3.

Evidence

  • Law:
    • v21.2 P10 (amended): every line carries an oracle-gaps: <kind> prefix, except the declared exempt lines. The exempt lines are the register's M entries of 2026-11-16: "P10 exemptions ratified", "--help is exempt" and "Other tools' output is exempt". Exactly one done line comes last on both streams, and it matches the exit code.
    • v21.2 rule IDs: I3 (the prefix and the exempt lines), I7 (one done line; error means exit 3), R10 (one diagnostic per failure), G25 and G26.
    • The line form follows the register's C entry "The internal-error line" (2026-11-16): exactly one oracle-gaps: error … line on stderr and none on stdout, then done error, log <path>, exit 3. Line breaks in the message and the path are escaped, and the stack is in error.log. The supervisor's own exit 3 owes no error line.
    • Law 12, "exits 3, says so, keeps its stack in error.log", is the maintainer decision of 2026-11-10.
  • Reach: state a run ended in an internal error. On the final head 3b11762 it was reached once by each of the fixed-seed and random-seed units of "a batch whose run rejects" (r125-green3-1). In the three Linux fixed-seed shards, 58 reach lines report 0 and 1 reports 1 (the rejected batch). No generated history reaches an internal error.
  • RED:
    • Round 1. Oracle 7f204be on production fac63df. Command: cd packages/oracles && pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "a batch whose run rejects" (r125-red-1). The fixed and random seed units fail with greppable stream: "not a line of the stream: This is a bug in oracle-gaps, not in your oracle. …"
    • Round 2. Oracle b11646e on the production code of merge f68edc5. Command: the same, with -t "a batch whose run rejects|our own" (r125-red2-1). 6 failed: the two seed units, plus the four calibrations, whose clean run fails. The difference is the run wrote 1 lines on stderr that are not lines of the stream: ["break/error.log"]: the unescaped log path split the line.
  • GREEN on 3b11762:
    • the round-2 command: 7 passed, including the four calibrations and determinism (r125-green3-1);
    • Linux fixed-seed shards 1/3, 2/3 and 3/3 (r125-g3-shard{1,2,3}, exit 0);
    • test:oracle-gaps:fast (33,893 tests) and test (115 tests) (r125-g3-fast, exit 0).
  • Mutant: each one runs on 3b11762 and is killed in the fixed and random seed units.
    • prefix drops oracle-gaps: error from the error line (r125-m3-prefix). The law names that line as a stray.
    • E drops the message's escape (r125-m3-esc): "stderr has 2 done lines". The forged done line from the message comes out as a second done line.
    • path drops the path's escape (r125-m3-path): stray break/error.log.
    • B, from round 1, is an indented and unprefixed second half of the error line (r125-mut-b1). It was killed, and it passed with the loose reader (r125-mut-b-loose1).
  • Limits:
    • Constructed reach only. The only internal error that any history reaches is the constructed rejected batch, as a cold run. An internal error through the daemon uses the same endRun, but no history reaches it; that path is checked by inspection.
    • Escapes. The path escape is witnessed on POSIX only, because a win32 name cannot hold a line break. Escaping in a completed run's lines is checked by inspection.
    • Exempt shapes. The V8, traced-warning and esbuild shapes are calibrated, but no generated history prints them. No history prints Vite's [vite] form, and no calibration covers it. A form of those tools that exemptBlocks does not name is read as a stray, so it fails loudly.
    • Not judged. What the stack in error.log says is not judged. The streams of --stop-daemon are not observed. v21.2 P10's note-log first-line clause is not part of this PR.

CI

The first round's run, at 7e9dd69, passed. Its only failures were Windows e2e shards 2/10 and 4/10. Those shards are non-blocking, and they fail the same way on main: the Windows idle gap, and the abort witness that waits for #79. Shard 2/10 took 11m59s. The review traces the overrun to the unit "every outcome class", which this PR does not touch. The same overrun is on #121's head (11m34s), and it is tracked in #118. The run for 3b11762 (43917930134) has the same result. Every Linux, framework, Vitest 5, static, unit and Windows unit job passes, and the longest Linux job took 7m28s. Windows e2e 2/10 and 4/10 fail with the same two pre-existing causes. Shard 2/10 took 10m30s, again over P12's 10-minute budget, and the overrun is tracked in #118.

This pull request and its description were written by Isaac.

KyleAMathews and others added 3 commits November 16, 2026 13:44
…ernal error

The end-to-end oracle skipped law 15 for the constructed rejected-batch
history, the one history that reaches an internal error. It now applies
the stream law to every run, observes both streams of the rejected run,
counts internal error lines (the new `error` kind: at most one, and
none unless the run exits 3), and judges law 12's ending: one error line
on stderr, no stack frame there, and an error.log. A new declared state,
"a run ended in an internal error", is entered by the rejected-batch
history.

Co-authored-by: Isaac <no-reply@databricks.com>
An internal error printed two lines with no oracle-gaps: prefix, which
P10 and v21.1 I3 forbid; the ratified exemptions name no such line. The
ending now prints one `oracle-gaps: error` line, with the message's
line breaks escaped, before its done line. The stack stays in error.log
and the exit code stays 3.

Fixes #125

Co-authored-by: Isaac <no-reply@databricks.com>
… stream law

The stream law's reader exempted any indented line, anywhere, and skipped
every empty line, though its own statement and the register's ratified
exemptions (2026-11-16) allow indented lines only under a baseline
failure. It now exempts an indented line only under a baseline fail line,
counts an empty line before the stream's end as a stray, and records in
the limits the two endings (--help, and esbuild's block for a syntax
error in a module the config imports) that print unexempt lines.

Co-authored-by: Isaac <no-reply@databricks.com>
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 237085c6-5110-48f7-8744-f0a85ea17832
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3b11762

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@tanstack/oracles Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Co-authored-by: Isaac <no-reply@databricks.com>
@pkg-pr-new

pkg-pr-new Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@128
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@128

commit: 3b11762

KyleAMathews and others added 4 commits November 16, 2026 16:35
…efix

Co-authored-by: Isaac <no-reply@databricks.com>
… internal error's line

Review of #128, F1, F2, F4, F6, F7, under the register's M entries of
2026-11-16 ("P10 exemptions ratified", "`--help` is exempt", "Other
tools' output is exempt") and its C entry "The internal-error line":

- The stream reader tells each exempt line by its own shape: a baseline
  failure's details, a refusal's block, Node's warning with its
  --trace-warnings frames, V8's fatal block, esbuild's message block, and
  Vite's own line. Four calibrations put a V8 block, a traced warning, and
  an esbuild block beside our own unprefixed line, and our own indented
  line alone, before the rejected batch's done line; the law must name our
  line alone.
- The general law owes a run that ends through endRun exactly one error
  line, on stderr, naming the done line's log, its output directory's
  error.log, and every other run none.
- The rejected batch's fault message holds a line break and a forged done
  line, and its output directory's name holds a line break on POSIX, so
  the message's and the path's escapes are witnessed.
- The Limits cite the rulings for --help, --stop-daemon, and esbuild's
  block, and the static oracle's Help law cites the --help ruling.

Co-authored-by: Isaac <no-reply@databricks.com>
An internal error's error and done lines, a refusal's done line, and a
completed run's results note, report path, and test-output note named
the output directory raw, so a line break in its name split the line or
forged another (review of #128, F5; the register's C entry "The
internal-error line"). They now print control characters as their
\\uXXXX escapes. A message that already starts with `oracle-gaps: ` no
longer names the CLI twice in the error line.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@KyleAMathews
KyleAMathews merged commit 2552d6a into main Nov 16, 2026
48 of 50 checks passed
@KyleAMathews
KyleAMathews deleted the fix/internal-error-prefix branch November 16, 2026 18:08
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
The driver's first run keeps the gate variable beside #128's rejected
batch output break and foreign-line calibration.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
#128's internal-error line and its stricter stream reader join #57's
default output directory (a run with no --out keeps no suffix, and
outBreak applies to the --out the driver names) and #57's stop prose
and limits.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
After #128 the rejected-batch history's output directory holds a line
break, and the stream law judges that run. Its observation now carries
the stream log, and the note it owes spells the path as every line of
the stream does: a control character as its \uXXXX escape (C 2026-11-16
"The internal-error line"). RED: the note line splits in two.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
…g path

A --out whose name holds a line break printed the note on two lines, on
both streams; every other line of the stream names a path with its
control characters escaped (#128). The note now does too.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 16, 2026
…stic-cuts

#121's and #128's constructed histories come before #99's. The rejected
batch's `outBreak` and #128's `foreign` ride on the history's shared run
options beside #99's hook env. capture-setup's `hit` keeps #121's probe-site
skip beside #99's cut rows.

Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews added a commit that referenced this pull request Nov 17, 2026
…n the first line (#57)

* test(oracle-gaps): law 15 owes a stream log that the first line names

The maintainer's decision of 2026-11-12: an agent that runs a check in
the background learns where to read its stream from the run itself. A run
that starts a check names `<out>/stream.log` in its first line on both
streams (`oracle-gaps: note log <path>`), and that file holds the
stderr's lines, in order, cold, through the daemon, and after a stop or
an abort. The driver records each run's first lines and whether its log
equals its stderr; two calibrations give a log that misses a line and a
naming line after another line.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): a refusal after a check started names its stream log first

The maintainer's decision of 2026-11-12 names the stream log first in
every run that starts a check, refusals in it included. The refusal law
(#46) owed stdout = the done line alone; it now owes [note log, done
line] when the check started, and [done line] for a refusal of the run's
arguments, which comes before any check (outsideSource, and the static
oracle's argument refusals). Law 15's log checks apply to runs that start
a check.

Co-authored-by: Isaac <no-reply@databricks.com>

* feat(oracle-gaps): write the stream to <out>/stream.log and name it first

A run that starts a check creates `<out>/stream.log`, writes
`oracle-gaps: note log <path>` as its first line on both streams, and
writes every stderr line to the log too: the cold check's lines, a daemon
reply's stderr, an internal error's or refusal's ending, the completed
done line, and the supervisor's own done line for a stopped or crashed
run.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): own the stream log in the supervisor; refuse a second run on a live --out

The supervisor opens <out>/stream.log and writes the note before it spawns
the child; it resolves --out once and passes it to the child and to the
daemon request; it tees the child's stderr, V8 lines included, to the log.
A run whose log cannot be opened is refused before any check (exit 2, one
done line), and a second run on a live output directory is refused without
writing into it (maintainer decision, 2026-11-13).

Oracle: out-mode (explicit or default) and log-blocked and overlap
histories, with the law that a completed run's results.json sits beside its
stream log and that a refused run writes nothing into a live directory.
Stop histories use the stop-only witness.

Co-authored-by: Isaac <no-reply@databricks.com>

* wip(oracle-gaps): checkpoint at the pre-restructure CLI with the oracle fixes

Oracle changes: out-mode, stop-only and stop-at-spawn dimensions appended to the
history grammar (earlier fields unchanged on the sampled histories); the five
new states as own-history states; overlap generated only for an explicit out, a
live first run, and a fault-free history; the overlap wait ends on the first
run's exit; constructed fault histories reset the end-to-end dimensions their
generated base may draw. CLI: pre-restructure index.mjs from 5a92a3e.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): claim the output directory atomically and release run.pid on exit

A run claims its output directory with run.pid: a temp file that already holds
the pid, linked to run.pid (fails if it exists), so two starts cannot both
claim. The claim is released on every exit but SIGKILL. A claim whose pid is
gone is taken over; a live one refuses the second run and names the pid.

Oracle: the spawn-delay law (outcome a or b, judged by the started marker,
added with the next commit), the overlap hold guard (a test-only hook), the
stop history that waits for the CLI to start, and the appended history fields.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): install the stop handlers before any project module loads

The CLI's first project statement is now the handler install, so a stop that
arrives during the loads is kept and acted on once the note is written. The
single run path (P4) is in the same file; the static-import variant with the
same handlers passed the default-out units, so the dynamic loads are not the
cause of any batch failure (review of #57).

A test-only marker, ORACLE_GAPS_TEST_STARTED, is written as the first project
statement so the oracle can tell a stop before the CLI started (no output is
allowed) from one during its loads (the full output is required).

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the spawn stop signals once the CLI has started, drawn 0-150 ms otherwise

The constructed stop history waits for the started marker (ORACLE_GAPS_TEST_STARTED)
and then signals at once, so the signal lands in the CLI's loads on every machine.
Generated histories keep the drawn 0-150 ms delay and signal (SIGINT or SIGTERM).
Verified: the real CLI passes; the late-handler mutant (handlers after the project
loads) is rejected with started-but-no-output.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): one decision for whether a refused run owes its note

A refusal of the arguments comes before any check, so its output is its done
line alone, through the daemon as in a cold run. The cold stream check, the
daemon stream check, and the daemon refusal's note now share owesNote(expected),
instead of the daemon path applying the note requirement to every refusal.
Verified: the outsideSource unit passes on the CLI; the mutant that requires a
note for argument refusals fails it.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): install the stop handlers before the started marker

The started marker (ORACLE_GAPS_TEST_STARTED) is written after the stop handlers
are installed, so a marker means the CLI can handle a stop. A test-only pause
after the marker (ORACLE_GAPS_TEST_PAUSE_AFTER_STARTED, ms) widens the window
for the spawn-stop witness: with the marker before the handlers, a stop during
the pause ended the CLI with a marker and no output (0 of 20 passes); with the
fixed order, 40 of 40 pass.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): take over a stale claim under a mutex, and name results.json's state

A stale run.pid is taken over only by the racer that creates run.pid.takeover (a
directory only one racer can make), and only after it re-reads the stale pid under
the mutex. Two concurrent starts against a stale claim now yield exactly one claim
(witness: 5 of 5 trials, one claimant each; the mutant without the mutex and the
re-read yields two in 5 of 5). A test-only pause (ORACLE_GAPS_TEST_PAUSE_AT, inert
unless set) widens the window. The results-beside-log message names the directory
and whether results.json is missing, unparseable or read (batch 3, issue #76).

Co-authored-by: Isaac <no-reply@databricks.com>

* ci(oracles): place the new Windows fixed-seed units in shards

The fixed-seed units added with the spawn stop, the stream log, and the overlap
(five units) were not in any Windows fixed-seed shard, so the pr-shards partition
check failed in the Test job. Placed in shards 2 and 3.

Co-authored-by: Isaac <no-reply@databricks.com>

* chore(oracle-gaps): drop the unsupervised POSIX seam

The unsupervised (win32) path is being replaced by a win32 supervisor (#74), so
the seam that forced it on POSIX is removed; its stop witness was not kept. The
unsupervised path's stop law (no done line on a stop) is a Limit of this PR.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): create the takeover mutex with its pid, and never take over an unreadable claim

The takeover mutex was made as a directory and its pid written after, so a racer
could read it empty, take it for dead, and remove a live mutex. The pid is now
written to a temp file and linked into place, so no reader sees an empty mutex.
An unreadable or empty mutex or claim counts as live and is never taken over.
Witness (two concurrent starts against a stale claim, a test pause between the
create and the pid write): the current code gives one claimed and one crashing
run in every trial; the fix gives one claimed and one refused in every trial.

Co-authored-by: Isaac <no-reply@databricks.com>

* docs(oracle-gaps): document every test-only ORACLE_GAPS_TEST hook

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): hold the exit at a named point, and delete the wall-clock hold

The overlap and the spawn stop 'after' case kept the first run alive with a
wall-clock hold (HOLD_EXIT_MS): under load the first run can end before the
second starts, so the overlap outcome depended on timing. The hold is now a named
point: after the done line the process keeps its exit (and its output claim)
until ORACLE_GAPS_TEST_HOLD_UNTIL names an existing file. The driver releases it
once the second run has been refused, or once the 'after' stop has been sent.
HOLD_EXIT_MS is deleted. The README documents the new hook.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): constructed histories for the stale-claim race and dead-pid recovery

Two constructed histories gate their reach states: two starts race for a stale claim
(exactly one claims, the other refuses), and a stale claim and mutex from a dead pid
are taken over (the run claims). The race is paused inside the takeover mutex's
creation (ORACLE_GAPS_TEST_PAUSE_AT). Placed in the e2e and Windows shards.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): claimScenario last in the draw order; claim states after the declaration

claimScenario is appended after spawnOnStart, so the generator draws every
earlier field as before (fingerprint canary: 0 of 16,500 pre-existing values differ
against 7db86c5 over 300 histories at seed 20261105). The claim scenario's reach states
are added after the states set is declared (the first VM run failed with a
ReferenceError there).

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): sequence the stale-claim race with a held run, not a timed pause

ORACLE_GAPS_TEST_PAUSE_MS is removed. A test hook now holds the CLI at one named point
(ORACLE_GAPS_TEST_PAUSE_AT): it writes ORACLE_GAPS_TEST_PAUSE_MARKER when it reaches the
point, then waits for ORACLE_GAPS_TEST_PAUSE_UNTIL. The race driver holds run A at the
takeover mutex's creation, starts run B once A is held, and releases A only after B has
claimed; A must refuse. Register: test hooks; no wall-clock ordering.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): generated histories never draw a claim scenario

The generator drew claimScenario race or recovery in about 2 of 12
histories, so fixed-seed histories built from generated records ran the
claim driver. The driver's out-claim writes, and a generated default
output directory, broke the shard units (a TypeError from mkdirSync on
an undefined out directory, and the declaration-fault units).

The field is now a constant in the drawn record, which draws nothing. The
histories take-back reads a constructed claim scenario as that constant,
because fast-check refuses any value the arbitrary cannot produce. The
claim driver runs only in the two constructed histories. The claim
driver's fixture is the one runCli already wrote, and its out-claim
directory is an owned write.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): a claim race that ends before the mutex is an outcome

The race driver waited for run A to reach the takeover mutex and threw
when A never did. A CLI that refuses a stale claim before the mutex (the
mutant in the claim evidence) then failed with a driver error, not with
the law. The driver now records a run that ends first as its outcome, and
waits for B's note or B's end, so the race is judged by the claim law:
a race gave [refused, refused], owed [claimed, refused].

The correct path is unchanged: A is held at the mutex, B claims, and A is
released only while B still holds the directory.

Co-authored-by: Isaac <no-reply@databricks.com>

* ci(oracles): place the claim histories in fixed-seed shard 1

The two constructed claim histories ran in the rest shard, which took
over 9 minutes on the pool. Shard 1 has the lightest measured time, and
the two histories add about 22 s to it. The shard partition check still
splits every test exactly.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): append the claim constructions, so existing seeds keep their histories

A constructed history samples its generated base with seed + index. The
two claim histories sat in the middle of the constructed list, so every
later history (the declaration faults among them) sampled a different
base than on the green commit, and brokenOwner, outsideSource and
setupThrows failed. The claim histories now come last, so each existing
history keeps its index and its base.

Also gives each claim history its own comment (the copied F4 comment
described the overlap).

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): the stream log's note names it in forward slashes on win32

The note line printed join(out, stream.log), which on win32 has
backslashes, so the greppable-stream law failed on Windows. The register's
Windows-paths entry (2026-11-13) requires forward slashes for report paths.

RED: the Windows fixed-seed end-to-end jobs on 5aaa3a1 (CI run 43646266358,
jobs 132604647764, 132604647986, 132604647967) failed "the run's first lines
are [oracle-gaps: note log <package>\out\stream.log, ...], not
... out/stream.log".

The note is posix(path) (paths.mjs), which is the identity on POSIX. The
expected note converts backslashes only on win32, as #79 does
(678b643, 9626f00).

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): name the forward-slash spelling of a scratch path on win32

After the note line used forward slashes, the Windows fixed-seed jobs
still failed the greppable-stream law: the note named the package as
D:/a/.../.scratch-xxx where the driver expected <package>, because the
driver named only the native spelling of each scratch path.

RED: the Windows fixed-seed end-to-end jobs on 2293582 (CI run 43651225289)
fail with "the run's first lines are [oracle-gaps: note log
D:/a/oracles/.../.scratch-xxx/out/stream.log, ...], not ... <package>/out/stream.log".

On win32 each scratch path is also named in forward slashes; POSIX has one
spelling and is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): stale claims refuse; no takeover

The maintainer's decision, "Stale claims refuse; no takeover": a run.pid
that names no running process, or that cannot be read, refuses the second
run, like a live claim. Exit 2, one done line, and a fix line that names
the claim. Nothing is taken over, so the double claim and the hangs of the
takeover mutex are gone by construction.

- Removed: the takeover mutex, its waits, the unreadable-claim loop, and
  the mutex-create pause point. The pause point takeover is gone too; the
  note point is the one new pause point.
- Waits in the CLI are bounded (testPause, holdUntil): a wait past 60 s
  fails the run loudly.
- The stream log's writes go through appendLog, which turns a failure into
  the refusal cannotWrite. The uncaught-exception handlers are registered
  before the supervisor runs, so a supervisor error ends the run through
  endWith. A log that cannot take the done line keeps both streams' done
  line and the decided exit code.
- The supervisor's variable is ORACLE_GAPS_INTERNAL_SUPERVISED. The child
  kills its group only when it leads that group (checked with ps).
- The test stop watcher is one function for both paths; the claim set is
  one variable.
- Claim histories: race (two starts), stale (one start), and empty (one
  start). Each refusal names its cause and the claim, and leaves run.pid
  and the directory as they were.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): define the claim state map; a claim a start removed is not kept

The claim driver referenced claimStateOf, which was not defined, so every
claim history threw. And a mutant that takes a claim removes run.pid when
it exits, so reading the claim back must tolerate its absence (the law
then judges the outcome). The driver also kills any start it spawned when
it throws.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): a stream log lost after its note, then stopped or completed

F1 (review of #57): a stream log that becomes unwritable after the note
crashed the supervisor with a stack, because its stream-log append was
unguarded and the uncaught handler was registered after the supervisor
returned.

- The supervisor records a lost stderr relay once and keeps relaying, so
  the run's own ending is the only done line on each stream.
- New constructed histories: the log is swapped for a directory while the
  supervisor is paused at the note (ORACLE_GAPS_TEST_PAUSE_AT=note), then
  the run is stopped or allowed to complete. The law: one done line on each
  stream, the same on both, the exit code it owes, and no stack.
- Stops sent by child.kill on win32 owe no done line (register, "Windows
  stops", C 2026-11-13); an abort is judged as before.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): review of the delta: determinism, unreadable claims, loud failures

- Determinism: the claim and lost-log observations carry the scratch paths
  normalized (<out>, <package>), as the rest of the oracle does.
- An unreadable run.pid (a directory, or no permission) refuses with its own
  cause and fix: "run.pid in <out> can't be read". A claim that vanishes
  between the link and the read is retried (three attempts at most).
- The win32 stop exemption excuses only the done line; the note, the log and
  the results checks stay.
- The ps group check fails the run when ps cannot read the group, so the
  watchdog is never silently unarmed.
- A hold that never releases exits nonzero with an internal-error line.
- Removed the dead logLost variable; corrected the stale recovery comments.
- New constructed history: a start refuses an unreadable claim (appended).

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): the mid-run check's stream check gets the run's log and results

The merge brought main's mid-run observation, which copied only the run's
exit, report and streams; the stream check (endsBreach) also reads the
run's log, output directory and results, so it threw a TypeError in the
fixed-seed shard 3 units through the daemon.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): a mid-run run that refuses owes no note, as the cold run does

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): the overlap observation names the holding pid only as <pid>

A second run's refusal names the first run's pid, which differs between two
runs of the same history, so the determinism unit "two runs on one output
directory" failed. The pid's place is observed, not its value.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): a daemon refusal keeps the output directory it names

The daemon sent a refusal's message, fix and stack but not the `out` that
args.mjs attaches, and the client rebuilt the refusal without it. Its report
then had no directory: write() returned false, so the done line lacked its
report, and the outsideSource unit failed.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): every refusal names its run's output directory, default or not

A run with no --out writes to the default directory of its package. A refusal
raised while the arguments resolve (an outside source, say) carried no
directory, so its report was never written, and law 12 failed (the outsideSource
unit). The refusal now carries the directory the run would have used.

RED: the outsideSource unit at 9d71756 (pool tags r57m-shard3, r57m-diag-outside).

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): a default-directory refusal is judged on its shape, not its directory

The cold refusal law required the report in the driver's --out, but a run with
the default directory has none the driver knows: the CLI chooses it. The law
now requires the directory only for an explicit --out.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the daemon's refusal is judged on its directory as the cold one is

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): a default-directory refusal names the directory the default rule predicts

The driver no longer skips the refusal's directory check for the default output
mode: the report must sit in the default directory's shape, and a logged run's
note must name the same directory. The explicit gate is gone. The default
directory is normalized in observations, so the random run name does not make
two runs differ.

RED: the outsideSource unit at 92d7ab0 (the default-mode gate narrowed it).

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): a default run's report may be named by the scratch placeholder <out>

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the static oracle normalizes a default output directory's random name

A refusal of the arguments now carries the directory its run would have used
(args.mjs), so its ending names a random default directory. The static
determinism units recorded that name, and two runs differed at the arguments
observation (CI, Gap report oracles 2/3).

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): delta check: claim cause and fix, mid-list inserts, README, stray file

- Claim write failures give the claim's own cause and fix (run.pid cannot be
  written), not the stream log's.
- The driver judges a claim refusal's fix line exactly: the claim's directive,
  if no run uses <out>, delete <out>/run.pid. The reviewer's mutant (f775dff)
  printed the stream-log fix and now fails.
- The eleven #57 constructed histories moved to the end of the list, so main's
  26 names are an exact prefix in order: every existing index and seed keeps
  its history.
- README: the diff3 marker and the stale HOLD_EXIT_MS paragraph are removed.
- Removed the stray packages/oracles/.dbg4.tmp.mjs; a dangling JSDoc is fixed.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the supervised child exits when its supervisor is killed; a hold the CLI cannot release is marked

- The supervisor records its child's pid beside the started marker, and a
  test that SIGKILLs the supervisor requires the child to be gone within 5 s
  (the child's watchdog, one tick). POSIX only: win32 has no supervisor until
  #79 (platform-scoped reach). The supervisor's own claim is left as SIGKILL
  leaves it, a stale claim, which the next run refuses.
- A hold that never releases writes a .unreleased marker beside the hold file;
  the CLI keeps the exit its done line stated, and the driver throws when the
  marker exists (a test-only hook: the failure belongs to the test).

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the death witness waits for a silent child, so only the watchdog can end it

The supervised child pauses before it writes anything (testPause child), with
its watchdog armed. The driver kills the supervisor once the child is paused.
A child that writes would die of a broken pipe, which would hide the watchdog.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): the death witness measures the child, and a held child keeps its watchdog running

The driver waited for the supervisor's `close` event, which fires only after
the child (which inherits the supervisor's stdout pipe) has ended, so the 5 s
check ran after the child's own 60 s hold had ended it. The witness passed
without the watchdog. The child's pause was also a synchronous wait, which
stopped the watchdog's timer, so the watchdog could never end a held child.

The driver now waits for `exit`, checks the recorded child pid within 5 s,
and kills that pid when the check fails. A supervisor that ends before its
child pauses fails the witness. The child's hold at `child` polls on timers.

Co-authored-by: Isaac <no-reply@databricks.com>

* docs(oracle-gaps): the supervisor death witness names its platform scope

win32 has no supervisor until #79, so the death state is not reached there and the law has no observation to judge. Comment only.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): progress durations are not compared; the supervisor death law is stopping

A progress line's wall-clock duration ("in 0.2s") is live progress, not
outcome, so the shared stream normalizer names it `<t>`. Every other field
stays. The supervisor death violation is tagged with the stopping law, not
output namespace.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): claim fix lines print a posix path; the supervisor-death state is platform-scoped

Windows CI showed the claim refusals' fix line printed a backslash path
(`delete <out>\run.pid`), where the owed line names the claim as
`<out>/run.pid`. The fix line now prints the claim in posix form.

The supervisor-death state exists only where a supervisor does, so it is
required and entered only off win32 (platform-scoped reach, register
"Platform-scoped reach (#73)"). On win32 the history runs without the death
witness and enters no state; that run is not judged by the death law.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): a daemon-served check writes to the directory its run named

A run without --out names its output directory per call (args.mjs), so the
client's stream log and the daemon's results.json were in two different
directories. Windows CI showed exit 0 with results.json missing beside the
stream log (run 43778158734, job 133009093614: OBSERVER-DIAG names the client's
and the daemon's directories, one second and a random suffix apart).

The request now carries the directory the run named, unless the arguments
already name one.

RED: Windows CI at 8120cb8 (run 43725828219), the daemon units on shards 1, 7,
8 and 9, and the diagnostic run 43778158734. GREEN: local POSIX default-output
daemon unit (1 passed); Windows CI pending on this head.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the POSIX oracle reaches the daemon request that names no directory

On POSIX the supervisor appends `--out` to the child, so every daemon request
carries the directory the run named and the default-directory state is never
produced there. On win32 there is no supervisor, the request has no `--out`,
and the daemon names another directory than the stream log's.

A test-only hook (ORACLE_GAPS_TEST_UNSUPERVISED) runs a POSIX check unsupervised,
the path win32 takes. The driver sets it on the daemon runs that name no
directory. RED: the fix reverted (the request without `--out`) on POSIX fails the
default-output daemon unit with results.json missing beside its stream log, the
same observation as Windows CI. GREEN: the fixed head passes.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the eight #57 drawn fields are constructed-only, and the unsupervised hook is a constructed field

F2 (generator rule, 2026-11-14): outMode, logBlocked, overlap, stopWitness,
spawnStop, spawnSignal, spawnDelay and spawnOnStart were drawn with values
that change what a history runs. They are now constants with neutral values,
set only by constructed histories, and the take-back maps a constructed value
to the neutral one through one table (constructedNeutral). concurrentStart,
tornRows, pauseEngages and warmReplaced are main's and are exempt.

F1: ORACLE_GAPS_TEST_UNSUPERVISED is set only for a history whose
unsupervised field is true. The drawn record holds false. The new constructed
history "a daemon check with the default output directory, unsupervised" is
appended last. The supervised default-directory history runs sequential
checks, so the supervisor-drops-`--out` mutant is killed by it.

RED (forwarding reverted, POSIX, the unsupervised unit): "through the daemon at
start: the run completed, but <out>/results.json is missing". Mutant (supervisor
drops --out for daemon runs without one, the supervised unit): same
observation. GREEN: the six default-directory units pass.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): place the unsupervised default-directory history in the partitions

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the cli observation compares gap lines as a sorted set, not in streamed order

Determinism flake (pool runs r57c-det-6, job of the unsupervised default-directory
history): two runs of one history printed the same 43 gap lines in a different
order, so the cli observation's lead differed at its first gap line. The order
of streamed gap lines is not part of the contract (C, 2026-11-13; live progress,
results.json and the gap-line set are the record). The lead keeps every other
line in order, and stdoutGaps, which holds each gap line once, is sorted.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): merge-gate fixes: limits, changeset, and nits

F3: the declared limits say the supervisor-death state is required only off
win32 (win32 has no supervisor until #79; register C entry (e) of #58), and
name the scope rather than an OS gap.
F4: the declared limits no longer say a default --out is judged only by the
static oracle, or that a SIGKILL of the CLI is judged by inspection: a SIGKILL
of a supervisor is witnessed on POSIX by its constructed history.
F6: the changeset lists the run.pid refusals (live, stale, empty, unreadable)
and the default-directory refusal.
F7: the daemon request always carries the directory the run named (no
`--out=dir` special case); the claim refusals print the claim directory in
posix form on win32; the stranded optionsOf JSDoc sits with its function.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the stderr gap lines are compared as a sorted set, like stdout's

The determinism flake's full difference (pre-merge pool run r57f-det-1, observation
cli, first difference at stderrGaps): two runs of one history streamed the same gap
lines on stderr in a different order. stderrGaps is the streamed gap lines, which
the register counts as live progress (C, 2026-11-13); results.json is the record.
The list is now sorted, as stdoutGaps already is, and every line is kept.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): draw #57's fields, judge win32 stops, and witness the start holds, ps and a dangling claim

The oracle side of the review of #57 under the maintainer's decision
"Randomness wins for new fields" (2026-11-16):

- outMode, logBlocked, stopWitness, claimScenario, spawnSignal and
  spawnDelay are drawn. Fields that switch a test-only hook (overlap,
  spawnStop, spawnOnStart, logLost, supervisorDeath, unsupervised) stay
  constants that constructed histories set.
- A drawn stop no longer takes `after`, which switches
  ORACLE_GAPS_TEST_HOLD_UNTIL; only a constructed history sets it, and
  the take-back drops it.
- The done line of every win32 stop is judged again: each goes through
  the test stop hook. A new constructed history stops an unsupervised
  run, so POSIX reaches stopNow too.
- A spawn stop is sent only once the CLI is held at its start, then
  after its drawn delay: after the started marker until the CLI keeps
  the stop, or before the CLI's first statement through the driver's own
  preload (outcome a, now judged on stopped runs). New constructed
  histories: a stop before the CLI starts and a SIGTERM stop as the CLI
  starts (POSIX only). "A stop as the CLI starts" is entered only when
  the CLI kept the stop before its child started.
- A drawn and constructed run with no command on its PATH (no ps), and a
  claim scenario whose run.pid is a dangling link.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the unsupervised witness stops after its note, and a drawn field never switches a hook

On POSIX an unsupervised run has no group to end, so a stop while the
witness's test spins leaves the test's own child (C 2026-11-13, Windows
(e)); the unsupervised witness now stops after its note instead, with
its stream log lost, the path every win32 stop takes. The history
grammar's canary also checks that no drawn history switches a test-only
hook, and that no drawn stop is the stop after the completed line.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the unsupervised witness is a stop held at the run's note

A stream log lost after the note makes the unsupervised run refuse
before its stop is taken, so that history never reached stopNow. The
witness now holds the run at its note (ORACLE_GAPS_TEST_PAUSE_AT), sends
the stop, and releases the hold: a constructed-only stop kind, `note`,
which the take-back drops as it drops `after`.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): a start hold released by the stop, no ps, and a dangling claim reads as unreadable

- The test-only hold after the started marker
  (ORACLE_GAPS_TEST_PAUSE_AFTER_STARTED) is released by the stop the CLI
  keeps (or, on win32, by the test stop file), never by time: a 200 ms
  hold let a late stop land after the supervisor had started its child
  (register C 2026-11-14, "Determinism under load covers witnesses
  too").
- The supervised child tells that it leads its own process group by a
  signal 0 to the group its pid names, not through `ps -o pgid= -p`,
  which a slim container lacks and BusyBox's ps does not accept: every
  supervised check exited 3 there.
- A run.pid that the claim's link finds and no read does (a link to no
  file) refuses as an unreadable claim, without writing its report into
  the directory; the "keeps changing" refusal wrote results.json and
  summary.md beside another run's claim.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): a blocked log refuses after the arguments and before the check

With logBlocked drawn again, a random seed (-118669542) drew a blocked log
beside a declaration fault: the driver claimed "refused <fault>", a
state the property does not declare, and the model owed the runtime
faults' refusal (brokenVitest, projects) before the log's and the log's
before an argument refusal (outsideSource). The CLI refuses its
arguments first, then a log it cannot write, then what the check finds
(P10 as amended in v21.2: an argument refusal owes the done line alone,
and once a check starts its first line is the note). The model's refusal
now follows those stages, and the driver claims a fault's refusal only
when the run reached its fault.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): split the warm and abort stops of every outcome class into two histories

On Windows the one history that held both stops took 557 s, and its job
10m16s, past P12's 10 minutes. Each stop now has its own constructed
history over the same base, appended last, and each sits in its own
Windows shard (4 and 5). The required states and the union of
"every outcome class" are unchanged; the output directory stays drawn.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the warm and abort stop histories run over the stop witness alone

Split over the first half's heavy base, each stop history cost that
base again: Linux fixed-seed 2/3 took 10m26s, and on Windows the abort
stop, which fails there on main's win32 witness (#79), shrank through
three heavy histories (592 s, job 12m41s). Each now runs over the stop
witness alone, the base of "a stop with the default output directory":
the same two states, cheaper histories (R7). Locally each passes in
under 15 s, and a mutant of the stopped line and one of the abort's
error line are each killed.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(oracle-gaps): the rejected batch's note names its log on one line

After #128 the rejected-batch history's output directory holds a line
break, and the stream law judges that run. Its observation now carries
the stream log, and the note it owes spells the path as every line of
the stream does: a control character as its \uXXXX escape (C 2026-11-16
"The internal-error line"). RED: the note line splits in two.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(oracle-gaps): the note line escapes a control character in its log path

A --out whose name holds a line break printed the note on two lines, on
both streams; every other line of the stream names a path with its
control characters escaped (#128). The note now does too.

Co-authored-by: Isaac <no-reply@databricks.com>

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Internal-error ending lines carry no oracle-gaps: prefix, and the e2e oracle skips the stream law for them

1 participant