Repository navigation
fix(oracle-gaps): prefix an internal error's lines, and judge the stream law for them - #128
Merged
Merged
Conversation
…ernal error The end-to-end oracle skipped law 15 for the constructed rejected-batch history, the one history that reaches an internal error. It now applies the stream law to every run, observes both streams of the rejected run, counts internal error lines (the new `error` kind: at most one, and none unless the run exits 3), and judges law 12's ending: one error line on stderr, no stack frame there, and an error.log. A new declared state, "a run ended in an internal error", is entered by the rejected-batch history. Co-authored-by: Isaac <no-reply@databricks.com>
An internal error printed two lines with no oracle-gaps: prefix, which P10 and v21.1 I3 forbid; the ratified exemptions name no such line. The ending now prints one `oracle-gaps: error` line, with the message's line breaks escaped, before its done line. The stack stays in error.log and the exit code stays 3. Fixes #125 Co-authored-by: Isaac <no-reply@databricks.com>
… stream law The stream law's reader exempted any indented line, anywhere, and skipped every empty line, though its own statement and the register's ratified exemptions (2026-11-16) allow indented lines only under a baseline failure. It now exempts an indented line only under a baseline fail line, counts an empty line before the stream's end as a stray, and records in the limits the two endings (--help, and esbuild's block for a syntax error in a module the config imports) that print unexempt lines. Co-authored-by: Isaac <no-reply@databricks.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
🦋 Changeset detectedLatest commit: 3b11762 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Co-authored-by: Isaac <no-reply@databricks.com>
commit: |
…efix Co-authored-by: Isaac <no-reply@databricks.com>
… internal error's line Review of #128, F1, F2, F4, F6, F7, under the register's M entries of 2026-11-16 ("P10 exemptions ratified", "`--help` is exempt", "Other tools' output is exempt") and its C entry "The internal-error line": - The stream reader tells each exempt line by its own shape: a baseline failure's details, a refusal's block, Node's warning with its --trace-warnings frames, V8's fatal block, esbuild's message block, and Vite's own line. Four calibrations put a V8 block, a traced warning, and an esbuild block beside our own unprefixed line, and our own indented line alone, before the rejected batch's done line; the law must name our line alone. - The general law owes a run that ends through endRun exactly one error line, on stderr, naming the done line's log, its output directory's error.log, and every other run none. - The rejected batch's fault message holds a line break and a forged done line, and its output directory's name holds a line break on POSIX, so the message's and the path's escapes are witnessed. - The Limits cite the rulings for --help, --stop-daemon, and esbuild's block, and the static oracle's Help law cites the --help ruling. Co-authored-by: Isaac <no-reply@databricks.com>
An internal error's error and done lines, a refusal's done line, and a completed run's results note, report path, and test-output note named the output directory raw, so a line break in its name split the line or forged another (review of #128, F5; the register's C entry "The internal-error line"). They now print control characters as their \\uXXXX escapes. A message that already starts with `oracle-gaps: ` no longer names the CLI twice in the error line. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
The driver's first run keeps the gate variable beside #128's rejected batch output break and foreign-line calibration. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
After #128 the rejected-batch history's output directory holds a line break, and the stream law judges that run. Its observation now carries the stream log, and the note it owes spells the path as every line of the stream does: a control character as its \uXXXX escape (C 2026-11-16 "The internal-error line"). RED: the note line splits in two. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
…g path A --out whose name holds a line break printed the note on two lines, on both streams; every other line of the stream names a path with its control characters escaped (#128). The note now does too. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 16, 2026
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
…stic-cuts #121's and #128's constructed histories come before #99's. The rejected batch's `outBreak` and #128's `foreign` ride on the history's shared run options beside #99's hook env. capture-setup's `hit` keeps #121's probe-site skip beside #99's cut rows. Co-authored-by: Isaac <no-reply@databricks.com>
8 of 9 tasks
KyleAMathews
added a commit
that referenced
this pull request
Nov 17, 2026
…n the first line (#57) * test(oracle-gaps): law 15 owes a stream log that the first line names The maintainer's decision of 2026-11-12: an agent that runs a check in the background learns where to read its stream from the run itself. A run that starts a check names `<out>/stream.log` in its first line on both streams (`oracle-gaps: note log <path>`), and that file holds the stderr's lines, in order, cold, through the daemon, and after a stop or an abort. The driver records each run's first lines and whether its log equals its stderr; two calibrations give a log that misses a line and a naming line after another line. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a refusal after a check started names its stream log first The maintainer's decision of 2026-11-12 names the stream log first in every run that starts a check, refusals in it included. The refusal law (#46) owed stdout = the done line alone; it now owes [note log, done line] when the check started, and [done line] for a refusal of the run's arguments, which comes before any check (outsideSource, and the static oracle's argument refusals). Law 15's log checks apply to runs that start a check. Co-authored-by: Isaac <no-reply@databricks.com> * feat(oracle-gaps): write the stream to <out>/stream.log and name it first A run that starts a check creates `<out>/stream.log`, writes `oracle-gaps: note log <path>` as its first line on both streams, and writes every stderr line to the log too: the cold check's lines, a daemon reply's stderr, an internal error's or refusal's ending, the completed done line, and the supervisor's own done line for a stopped or crashed run. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): own the stream log in the supervisor; refuse a second run on a live --out The supervisor opens <out>/stream.log and writes the note before it spawns the child; it resolves --out once and passes it to the child and to the daemon request; it tees the child's stderr, V8 lines included, to the log. A run whose log cannot be opened is refused before any check (exit 2, one done line), and a second run on a live output directory is refused without writing into it (maintainer decision, 2026-11-13). Oracle: out-mode (explicit or default) and log-blocked and overlap histories, with the law that a completed run's results.json sits beside its stream log and that a refused run writes nothing into a live directory. Stop histories use the stop-only witness. Co-authored-by: Isaac <no-reply@databricks.com> * wip(oracle-gaps): checkpoint at the pre-restructure CLI with the oracle fixes Oracle changes: out-mode, stop-only and stop-at-spawn dimensions appended to the history grammar (earlier fields unchanged on the sampled histories); the five new states as own-history states; overlap generated only for an explicit out, a live first run, and a fault-free history; the overlap wait ends on the first run's exit; constructed fault histories reset the end-to-end dimensions their generated base may draw. CLI: pre-restructure index.mjs from 5a92a3e. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): claim the output directory atomically and release run.pid on exit A run claims its output directory with run.pid: a temp file that already holds the pid, linked to run.pid (fails if it exists), so two starts cannot both claim. The claim is released on every exit but SIGKILL. A claim whose pid is gone is taken over; a live one refuses the second run and names the pid. Oracle: the spawn-delay law (outcome a or b, judged by the started marker, added with the next commit), the overlap hold guard (a test-only hook), the stop history that waits for the CLI to start, and the appended history fields. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): install the stop handlers before any project module loads The CLI's first project statement is now the handler install, so a stop that arrives during the loads is kept and acted on once the note is written. The single run path (P4) is in the same file; the static-import variant with the same handlers passed the default-out units, so the dynamic loads are not the cause of any batch failure (review of #57). A test-only marker, ORACLE_GAPS_TEST_STARTED, is written as the first project statement so the oracle can tell a stop before the CLI started (no output is allowed) from one during its loads (the full output is required). Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the spawn stop signals once the CLI has started, drawn 0-150 ms otherwise The constructed stop history waits for the started marker (ORACLE_GAPS_TEST_STARTED) and then signals at once, so the signal lands in the CLI's loads on every machine. Generated histories keep the drawn 0-150 ms delay and signal (SIGINT or SIGTERM). Verified: the real CLI passes; the late-handler mutant (handlers after the project loads) is rejected with started-but-no-output. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): one decision for whether a refused run owes its note A refusal of the arguments comes before any check, so its output is its done line alone, through the daemon as in a cold run. The cold stream check, the daemon stream check, and the daemon refusal's note now share owesNote(expected), instead of the daemon path applying the note requirement to every refusal. Verified: the outsideSource unit passes on the CLI; the mutant that requires a note for argument refusals fails it. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): install the stop handlers before the started marker The started marker (ORACLE_GAPS_TEST_STARTED) is written after the stop handlers are installed, so a marker means the CLI can handle a stop. A test-only pause after the marker (ORACLE_GAPS_TEST_PAUSE_AFTER_STARTED, ms) widens the window for the spawn-stop witness: with the marker before the handlers, a stop during the pause ended the CLI with a marker and no output (0 of 20 passes); with the fixed order, 40 of 40 pass. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): take over a stale claim under a mutex, and name results.json's state A stale run.pid is taken over only by the racer that creates run.pid.takeover (a directory only one racer can make), and only after it re-reads the stale pid under the mutex. Two concurrent starts against a stale claim now yield exactly one claim (witness: 5 of 5 trials, one claimant each; the mutant without the mutex and the re-read yields two in 5 of 5). A test-only pause (ORACLE_GAPS_TEST_PAUSE_AT, inert unless set) widens the window. The results-beside-log message names the directory and whether results.json is missing, unparseable or read (batch 3, issue #76). Co-authored-by: Isaac <no-reply@databricks.com> * ci(oracles): place the new Windows fixed-seed units in shards The fixed-seed units added with the spawn stop, the stream log, and the overlap (five units) were not in any Windows fixed-seed shard, so the pr-shards partition check failed in the Test job. Placed in shards 2 and 3. Co-authored-by: Isaac <no-reply@databricks.com> * chore(oracle-gaps): drop the unsupervised POSIX seam The unsupervised (win32) path is being replaced by a win32 supervisor (#74), so the seam that forced it on POSIX is removed; its stop witness was not kept. The unsupervised path's stop law (no done line on a stop) is a Limit of this PR. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): create the takeover mutex with its pid, and never take over an unreadable claim The takeover mutex was made as a directory and its pid written after, so a racer could read it empty, take it for dead, and remove a live mutex. The pid is now written to a temp file and linked into place, so no reader sees an empty mutex. An unreadable or empty mutex or claim counts as live and is never taken over. Witness (two concurrent starts against a stale claim, a test pause between the create and the pid write): the current code gives one claimed and one crashing run in every trial; the fix gives one claimed and one refused in every trial. Co-authored-by: Isaac <no-reply@databricks.com> * docs(oracle-gaps): document every test-only ORACLE_GAPS_TEST hook Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): hold the exit at a named point, and delete the wall-clock hold The overlap and the spawn stop 'after' case kept the first run alive with a wall-clock hold (HOLD_EXIT_MS): under load the first run can end before the second starts, so the overlap outcome depended on timing. The hold is now a named point: after the done line the process keeps its exit (and its output claim) until ORACLE_GAPS_TEST_HOLD_UNTIL names an existing file. The driver releases it once the second run has been refused, or once the 'after' stop has been sent. HOLD_EXIT_MS is deleted. The README documents the new hook. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): constructed histories for the stale-claim race and dead-pid recovery Two constructed histories gate their reach states: two starts race for a stale claim (exactly one claims, the other refuses), and a stale claim and mutex from a dead pid are taken over (the run claims). The race is paused inside the takeover mutex's creation (ORACLE_GAPS_TEST_PAUSE_AT). Placed in the e2e and Windows shards. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): claimScenario last in the draw order; claim states after the declaration claimScenario is appended after spawnOnStart, so the generator draws every earlier field as before (fingerprint canary: 0 of 16,500 pre-existing values differ against 7db86c5 over 300 histories at seed 20261105). The claim scenario's reach states are added after the states set is declared (the first VM run failed with a ReferenceError there). Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): sequence the stale-claim race with a held run, not a timed pause ORACLE_GAPS_TEST_PAUSE_MS is removed. A test hook now holds the CLI at one named point (ORACLE_GAPS_TEST_PAUSE_AT): it writes ORACLE_GAPS_TEST_PAUSE_MARKER when it reaches the point, then waits for ORACLE_GAPS_TEST_PAUSE_UNTIL. The race driver holds run A at the takeover mutex's creation, starts run B once A is held, and releases A only after B has claimed; A must refuse. Register: test hooks; no wall-clock ordering. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): generated histories never draw a claim scenario The generator drew claimScenario race or recovery in about 2 of 12 histories, so fixed-seed histories built from generated records ran the claim driver. The driver's out-claim writes, and a generated default output directory, broke the shard units (a TypeError from mkdirSync on an undefined out directory, and the declaration-fault units). The field is now a constant in the drawn record, which draws nothing. The histories take-back reads a constructed claim scenario as that constant, because fast-check refuses any value the arbitrary cannot produce. The claim driver runs only in the two constructed histories. The claim driver's fixture is the one runCli already wrote, and its out-claim directory is an owned write. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a claim race that ends before the mutex is an outcome The race driver waited for run A to reach the takeover mutex and threw when A never did. A CLI that refuses a stale claim before the mutex (the mutant in the claim evidence) then failed with a driver error, not with the law. The driver now records a run that ends first as its outcome, and waits for B's note or B's end, so the race is judged by the claim law: a race gave [refused, refused], owed [claimed, refused]. The correct path is unchanged: A is held at the mutex, B claims, and A is released only while B still holds the directory. Co-authored-by: Isaac <no-reply@databricks.com> * ci(oracles): place the claim histories in fixed-seed shard 1 The two constructed claim histories ran in the rest shard, which took over 9 minutes on the pool. Shard 1 has the lightest measured time, and the two histories add about 22 s to it. The shard partition check still splits every test exactly. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): append the claim constructions, so existing seeds keep their histories A constructed history samples its generated base with seed + index. The two claim histories sat in the middle of the constructed list, so every later history (the declaration faults among them) sampled a different base than on the green commit, and brokenOwner, outsideSource and setupThrows failed. The claim histories now come last, so each existing history keeps its index and its base. Also gives each claim history its own comment (the copied F4 comment described the overlap). Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): the stream log's note names it in forward slashes on win32 The note line printed join(out, stream.log), which on win32 has backslashes, so the greppable-stream law failed on Windows. The register's Windows-paths entry (2026-11-13) requires forward slashes for report paths. RED: the Windows fixed-seed end-to-end jobs on 5aaa3a1 (CI run 43646266358, jobs 132604647764, 132604647986, 132604647967) failed "the run's first lines are [oracle-gaps: note log <package>\out\stream.log, ...], not ... out/stream.log". The note is posix(path) (paths.mjs), which is the identity on POSIX. The expected note converts backslashes only on win32, as #79 does (678b643, 9626f00). Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): name the forward-slash spelling of a scratch path on win32 After the note line used forward slashes, the Windows fixed-seed jobs still failed the greppable-stream law: the note named the package as D:/a/.../.scratch-xxx where the driver expected <package>, because the driver named only the native spelling of each scratch path. RED: the Windows fixed-seed end-to-end jobs on 2293582 (CI run 43651225289) fail with "the run's first lines are [oracle-gaps: note log D:/a/oracles/.../.scratch-xxx/out/stream.log, ...], not ... <package>/out/stream.log". On win32 each scratch path is also named in forward slashes; POSIX has one spelling and is unchanged. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): stale claims refuse; no takeover The maintainer's decision, "Stale claims refuse; no takeover": a run.pid that names no running process, or that cannot be read, refuses the second run, like a live claim. Exit 2, one done line, and a fix line that names the claim. Nothing is taken over, so the double claim and the hangs of the takeover mutex are gone by construction. - Removed: the takeover mutex, its waits, the unreadable-claim loop, and the mutex-create pause point. The pause point takeover is gone too; the note point is the one new pause point. - Waits in the CLI are bounded (testPause, holdUntil): a wait past 60 s fails the run loudly. - The stream log's writes go through appendLog, which turns a failure into the refusal cannotWrite. The uncaught-exception handlers are registered before the supervisor runs, so a supervisor error ends the run through endWith. A log that cannot take the done line keeps both streams' done line and the decided exit code. - The supervisor's variable is ORACLE_GAPS_INTERNAL_SUPERVISED. The child kills its group only when it leads that group (checked with ps). - The test stop watcher is one function for both paths; the claim set is one variable. - Claim histories: race (two starts), stale (one start), and empty (one start). Each refusal names its cause and the claim, and leaves run.pid and the directory as they were. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): define the claim state map; a claim a start removed is not kept The claim driver referenced claimStateOf, which was not defined, so every claim history threw. And a mutant that takes a claim removes run.pid when it exits, so reading the claim back must tolerate its absence (the law then judges the outcome). The driver also kills any start it spawned when it throws. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a stream log lost after its note, then stopped or completed F1 (review of #57): a stream log that becomes unwritable after the note crashed the supervisor with a stack, because its stream-log append was unguarded and the uncaught handler was registered after the supervisor returned. - The supervisor records a lost stderr relay once and keeps relaying, so the run's own ending is the only done line on each stream. - New constructed histories: the log is swapped for a directory while the supervisor is paused at the note (ORACLE_GAPS_TEST_PAUSE_AT=note), then the run is stopped or allowed to complete. The law: one done line on each stream, the same on both, the exit code it owes, and no stack. - Stops sent by child.kill on win32 owe no done line (register, "Windows stops", C 2026-11-13); an abort is judged as before. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): review of the delta: determinism, unreadable claims, loud failures - Determinism: the claim and lost-log observations carry the scratch paths normalized (<out>, <package>), as the rest of the oracle does. - An unreadable run.pid (a directory, or no permission) refuses with its own cause and fix: "run.pid in <out> can't be read". A claim that vanishes between the link and the read is retried (three attempts at most). - The win32 stop exemption excuses only the done line; the note, the log and the results checks stay. - The ps group check fails the run when ps cannot read the group, so the watchdog is never silently unarmed. - A hold that never releases exits nonzero with an internal-error line. - Removed the dead logLost variable; corrected the stale recovery comments. - New constructed history: a start refuses an unreadable claim (appended). Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): the mid-run check's stream check gets the run's log and results The merge brought main's mid-run observation, which copied only the run's exit, report and streams; the stream check (endsBreach) also reads the run's log, output directory and results, so it threw a TypeError in the fixed-seed shard 3 units through the daemon. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): a mid-run run that refuses owes no note, as the cold run does Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): the overlap observation names the holding pid only as <pid> A second run's refusal names the first run's pid, which differs between two runs of the same history, so the determinism unit "two runs on one output directory" failed. The pid's place is observed, not its value. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): a daemon refusal keeps the output directory it names The daemon sent a refusal's message, fix and stack but not the `out` that args.mjs attaches, and the client rebuilt the refusal without it. Its report then had no directory: write() returned false, so the done line lacked its report, and the outsideSource unit failed. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): every refusal names its run's output directory, default or not A run with no --out writes to the default directory of its package. A refusal raised while the arguments resolve (an outside source, say) carried no directory, so its report was never written, and law 12 failed (the outsideSource unit). The refusal now carries the directory the run would have used. RED: the outsideSource unit at 9d71756 (pool tags r57m-shard3, r57m-diag-outside). Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a default-directory refusal is judged on its shape, not its directory The cold refusal law required the report in the driver's --out, but a run with the default directory has none the driver knows: the CLI chooses it. The law now requires the directory only for an explicit --out. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the daemon's refusal is judged on its directory as the cold one is Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a default-directory refusal names the directory the default rule predicts The driver no longer skips the refusal's directory check for the default output mode: the report must sit in the default directory's shape, and a logged run's note must name the same directory. The explicit gate is gone. The default directory is normalized in observations, so the random run name does not make two runs differ. RED: the outsideSource unit at 92d7ab0 (the default-mode gate narrowed it). Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a default run's report may be named by the scratch placeholder <out> Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the static oracle normalizes a default output directory's random name A refusal of the arguments now carries the directory its run would have used (args.mjs), so its ending names a random default directory. The static determinism units recorded that name, and two runs differed at the arguments observation (CI, Gap report oracles 2/3). Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): delta check: claim cause and fix, mid-list inserts, README, stray file - Claim write failures give the claim's own cause and fix (run.pid cannot be written), not the stream log's. - The driver judges a claim refusal's fix line exactly: the claim's directive, if no run uses <out>, delete <out>/run.pid. The reviewer's mutant (f775dff) printed the stream-log fix and now fails. - The eleven #57 constructed histories moved to the end of the list, so main's 26 names are an exact prefix in order: every existing index and seed keeps its history. - README: the diff3 marker and the stale HOLD_EXIT_MS paragraph are removed. - Removed the stray packages/oracles/.dbg4.tmp.mjs; a dangling JSDoc is fixed. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the supervised child exits when its supervisor is killed; a hold the CLI cannot release is marked - The supervisor records its child's pid beside the started marker, and a test that SIGKILLs the supervisor requires the child to be gone within 5 s (the child's watchdog, one tick). POSIX only: win32 has no supervisor until #79 (platform-scoped reach). The supervisor's own claim is left as SIGKILL leaves it, a stale claim, which the next run refuses. - A hold that never releases writes a .unreleased marker beside the hold file; the CLI keeps the exit its done line stated, and the driver throws when the marker exists (a test-only hook: the failure belongs to the test). Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the death witness waits for a silent child, so only the watchdog can end it The supervised child pauses before it writes anything (testPause child), with its watchdog armed. The driver kills the supervisor once the child is paused. A child that writes would die of a broken pipe, which would hide the watchdog. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): the death witness measures the child, and a held child keeps its watchdog running The driver waited for the supervisor's `close` event, which fires only after the child (which inherits the supervisor's stdout pipe) has ended, so the 5 s check ran after the child's own 60 s hold had ended it. The witness passed without the watchdog. The child's pause was also a synchronous wait, which stopped the watchdog's timer, so the watchdog could never end a held child. The driver now waits for `exit`, checks the recorded child pid within 5 s, and kills that pid when the check fails. A supervisor that ends before its child pauses fails the witness. The child's hold at `child` polls on timers. Co-authored-by: Isaac <no-reply@databricks.com> * docs(oracle-gaps): the supervisor death witness names its platform scope win32 has no supervisor until #79, so the death state is not reached there and the law has no observation to judge. Comment only. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): progress durations are not compared; the supervisor death law is stopping A progress line's wall-clock duration ("in 0.2s") is live progress, not outcome, so the shared stream normalizer names it `<t>`. Every other field stays. The supervisor death violation is tagged with the stopping law, not output namespace. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): claim fix lines print a posix path; the supervisor-death state is platform-scoped Windows CI showed the claim refusals' fix line printed a backslash path (`delete <out>\run.pid`), where the owed line names the claim as `<out>/run.pid`. The fix line now prints the claim in posix form. The supervisor-death state exists only where a supervisor does, so it is required and entered only off win32 (platform-scoped reach, register "Platform-scoped reach (#73)"). On win32 the history runs without the death witness and enters no state; that run is not judged by the death law. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): a daemon-served check writes to the directory its run named A run without --out names its output directory per call (args.mjs), so the client's stream log and the daemon's results.json were in two different directories. Windows CI showed exit 0 with results.json missing beside the stream log (run 43778158734, job 133009093614: OBSERVER-DIAG names the client's and the daemon's directories, one second and a random suffix apart). The request now carries the directory the run named, unless the arguments already name one. RED: Windows CI at 8120cb8 (run 43725828219), the daemon units on shards 1, 7, 8 and 9, and the diagnostic run 43778158734. GREEN: local POSIX default-output daemon unit (1 passed); Windows CI pending on this head. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the POSIX oracle reaches the daemon request that names no directory On POSIX the supervisor appends `--out` to the child, so every daemon request carries the directory the run named and the default-directory state is never produced there. On win32 there is no supervisor, the request has no `--out`, and the daemon names another directory than the stream log's. A test-only hook (ORACLE_GAPS_TEST_UNSUPERVISED) runs a POSIX check unsupervised, the path win32 takes. The driver sets it on the daemon runs that name no directory. RED: the fix reverted (the request without `--out`) on POSIX fails the default-output daemon unit with results.json missing beside its stream log, the same observation as Windows CI. GREEN: the fixed head passes. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the eight #57 drawn fields are constructed-only, and the unsupervised hook is a constructed field F2 (generator rule, 2026-11-14): outMode, logBlocked, overlap, stopWitness, spawnStop, spawnSignal, spawnDelay and spawnOnStart were drawn with values that change what a history runs. They are now constants with neutral values, set only by constructed histories, and the take-back maps a constructed value to the neutral one through one table (constructedNeutral). concurrentStart, tornRows, pauseEngages and warmReplaced are main's and are exempt. F1: ORACLE_GAPS_TEST_UNSUPERVISED is set only for a history whose unsupervised field is true. The drawn record holds false. The new constructed history "a daemon check with the default output directory, unsupervised" is appended last. The supervised default-directory history runs sequential checks, so the supervisor-drops-`--out` mutant is killed by it. RED (forwarding reverted, POSIX, the unsupervised unit): "through the daemon at start: the run completed, but <out>/results.json is missing". Mutant (supervisor drops --out for daemon runs without one, the supervised unit): same observation. GREEN: the six default-directory units pass. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): place the unsupervised default-directory history in the partitions Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the cli observation compares gap lines as a sorted set, not in streamed order Determinism flake (pool runs r57c-det-6, job of the unsupervised default-directory history): two runs of one history printed the same 43 gap lines in a different order, so the cli observation's lead differed at its first gap line. The order of streamed gap lines is not part of the contract (C, 2026-11-13; live progress, results.json and the gap-line set are the record). The lead keeps every other line in order, and stdoutGaps, which holds each gap line once, is sorted. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): merge-gate fixes: limits, changeset, and nits F3: the declared limits say the supervisor-death state is required only off win32 (win32 has no supervisor until #79; register C entry (e) of #58), and name the scope rather than an OS gap. F4: the declared limits no longer say a default --out is judged only by the static oracle, or that a SIGKILL of the CLI is judged by inspection: a SIGKILL of a supervisor is witnessed on POSIX by its constructed history. F6: the changeset lists the run.pid refusals (live, stale, empty, unreadable) and the default-directory refusal. F7: the daemon request always carries the directory the run named (no `--out=dir` special case); the claim refusals print the claim directory in posix form on win32; the stranded optionsOf JSDoc sits with its function. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the stderr gap lines are compared as a sorted set, like stdout's The determinism flake's full difference (pre-merge pool run r57f-det-1, observation cli, first difference at stderrGaps): two runs of one history streamed the same gap lines on stderr in a different order. stderrGaps is the streamed gap lines, which the register counts as live progress (C, 2026-11-13); results.json is the record. The list is now sorted, as stdoutGaps already is, and every line is kept. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): draw #57's fields, judge win32 stops, and witness the start holds, ps and a dangling claim The oracle side of the review of #57 under the maintainer's decision "Randomness wins for new fields" (2026-11-16): - outMode, logBlocked, stopWitness, claimScenario, spawnSignal and spawnDelay are drawn. Fields that switch a test-only hook (overlap, spawnStop, spawnOnStart, logLost, supervisorDeath, unsupervised) stay constants that constructed histories set. - A drawn stop no longer takes `after`, which switches ORACLE_GAPS_TEST_HOLD_UNTIL; only a constructed history sets it, and the take-back drops it. - The done line of every win32 stop is judged again: each goes through the test stop hook. A new constructed history stops an unsupervised run, so POSIX reaches stopNow too. - A spawn stop is sent only once the CLI is held at its start, then after its drawn delay: after the started marker until the CLI keeps the stop, or before the CLI's first statement through the driver's own preload (outcome a, now judged on stopped runs). New constructed histories: a stop before the CLI starts and a SIGTERM stop as the CLI starts (POSIX only). "A stop as the CLI starts" is entered only when the CLI kept the stop before its child started. - A drawn and constructed run with no command on its PATH (no ps), and a claim scenario whose run.pid is a dangling link. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the unsupervised witness stops after its note, and a drawn field never switches a hook On POSIX an unsupervised run has no group to end, so a stop while the witness's test spins leaves the test's own child (C 2026-11-13, Windows (e)); the unsupervised witness now stops after its note instead, with its stream log lost, the path every win32 stop takes. The history grammar's canary also checks that no drawn history switches a test-only hook, and that no drawn stop is the stop after the completed line. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the unsupervised witness is a stop held at the run's note A stream log lost after the note makes the unsupervised run refuse before its stop is taken, so that history never reached stopNow. The witness now holds the run at its note (ORACLE_GAPS_TEST_PAUSE_AT), sends the stop, and releases the hold: a constructed-only stop kind, `note`, which the take-back drops as it drops `after`. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): a start hold released by the stop, no ps, and a dangling claim reads as unreadable - The test-only hold after the started marker (ORACLE_GAPS_TEST_PAUSE_AFTER_STARTED) is released by the stop the CLI keeps (or, on win32, by the test stop file), never by time: a 200 ms hold let a late stop land after the supervisor had started its child (register C 2026-11-14, "Determinism under load covers witnesses too"). - The supervised child tells that it leads its own process group by a signal 0 to the group its pid names, not through `ps -o pgid= -p`, which a slim container lacks and BusyBox's ps does not accept: every supervised check exited 3 there. - A run.pid that the claim's link finds and no read does (a link to no file) refuses as an unreadable claim, without writing its report into the directory; the "keeps changing" refusal wrote results.json and summary.md beside another run's claim. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a blocked log refuses after the arguments and before the check With logBlocked drawn again, a random seed (-118669542) drew a blocked log beside a declaration fault: the driver claimed "refused <fault>", a state the property does not declare, and the model owed the runtime faults' refusal (brokenVitest, projects) before the log's and the log's before an argument refusal (outsideSource). The CLI refuses its arguments first, then a log it cannot write, then what the check finds (P10 as amended in v21.2: an argument refusal owes the done line alone, and once a check starts its first line is the note). The model's refusal now follows those stages, and the driver claims a fault's refusal only when the run reached its fault. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): split the warm and abort stops of every outcome class into two histories On Windows the one history that held both stops took 557 s, and its job 10m16s, past P12's 10 minutes. Each stop now has its own constructed history over the same base, appended last, and each sits in its own Windows shard (4 and 5). The required states and the union of "every outcome class" are unchanged; the output directory stays drawn. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the warm and abort stop histories run over the stop witness alone Split over the first half's heavy base, each stop history cost that base again: Linux fixed-seed 2/3 took 10m26s, and on Windows the abort stop, which fails there on main's win32 witness (#79), shrank through three heavy histories (592 s, job 12m41s). Each now runs over the stop witness alone, the base of "a stop with the default output directory": the same two states, cheaper histories (R7). Locally each passes in under 15 s, and a mutant of the stopped line and one of the abort's error line are each killed. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the rejected batch's note names its log on one line After #128 the rejected-batch history's output directory holds a line break, and the stream law judges that run. Its observation now carries the stream log, and the note it owes spells the path as every line of the stream does: a control character as its \uXXXX escape (C 2026-11-16 "The internal-error line"). RED: the note line splits in two. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): the note line escapes a control character in its log path A --out whose name holds a line break printed the note on two lines, on both streams; every other line of the stream names a path with its control characters escaped (#128). The note now does too. Co-authored-by: Isaac <no-reply@databricks.com> --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #125.
Summary
The gap CLI's internal-error ending printed two lines with no
oracle-gaps:prefix:The e2e oracle skipped the stream law for the one history that reaches an internal error: the constructed history "a batch whose run rejects" (#61). This PR fixes the oracle first, and then the CLI. The second round answers the review in
pr128-review.md(F1 to F7) under the register's 2026-11-16 entries. It also merges origin/main (fe3a6b8, #121).Oracle
Files:
gap-report-oracle.property.test.mjs,gap-report-model-oracle.mjs,gap-report-static-oracle.property.test.mjs.The stream law applies to every run. Law 15 now judges every run, including an internal error. The model gains the stream kind
error.One error line, tied to the log (general law, F6). A run whose done line is
done error, log <log>(anendRunending) owes exactly one error line, on stderr. That line must name the same log, and the log must be its output directory'serror.log, spelled with escapes. The run owes no error line on stdout. Every other run owes none, including the supervisor's own exit 3.What the rejected batch must still show (law 12). Exit 3, no stack frame on stderr, and an
error.log. A new state,a run ended in an internal error, is required, and the rejected-batch history enters it.Exempt lines are told by their own shapes (F1). Indentation alone exempts nothing. The reader (
exemptBlocks) exempts:--trace-warningsframes and its hint;✘ [ERROR]and▲ [WARNING]blocks;[vite]line.A blank line before the end of the stream is a stray, unless it is inside one of these blocks or comes right before one.
Four new calibrations. Each runs the rejected batch, puts lines before its done line (
foreign), and requires the law to name our own line alone:Witnesses for the escaping (F4 and F5).
retention-hook.mjs) contains a line break, followed by a forged done line.--outname contains a line break.Limits and the static oracle (F2).
--helpis exempt under the register's M entry "--helpis exempt". The static oracle's Help law now cites that entry.--stop-daemonis read the same way (C, pending ratification).Nits (F7). The law prose and the README now list every exempt form. The prose says the stdout-summary exemption is unused. The stale
output: <dir>text in the model is corrected.Production
Files:
ending.mjs,check.mjs,report.mjs.oracle-gaps: error <message>: a bug in oracle-gaps, not in your oracle; its stack is in <log>oracle-gaps: done error, log <log>\uXXXXescapes (escapeControls). These paths are:oracle-gaps:no longer repeats the CLI's name.error.log, and the exit status stays 3.Evidence
oracle-gaps: <kind>prefix, except the declared exempt lines. The exempt lines are the register's M entries of 2026-11-16: "P10 exemptions ratified", "--helpis exempt" and "Other tools' output is exempt". Exactly one done line comes last on both streams, and it matches the exit code.oracle-gaps: error …line on stderr and none on stdout, thendone error, log <path>, exit 3. Line breaks in the message and the path are escaped, and the stack is inerror.log. The supervisor's own exit 3 owes no error line.a run ended in an internal error. On the final head 3b11762 it was reached once by each of the fixed-seed and random-seed units of "a batch whose run rejects" (r125-green3-1). In the three Linux fixed-seed shards, 58 reach lines report 0 and 1 reports 1 (the rejected batch). No generated history reaches an internal error.cd packages/oracles && pnpm exec vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "a batch whose run rejects"(r125-red-1). The fixed and random seed units fail withgreppable stream: "not a line of the stream: This is a bug in oracle-gaps, not in your oracle. …"-t "a batch whose run rejects|our own"(r125-red2-1). 6 failed: the two seed units, plus the four calibrations, whose clean run fails. The difference isthe run wrote 1 lines on stderr that are not lines of the stream: ["break/error.log"]: the unescaped log path split the line.r125-green3-1);r125-g3-shard{1,2,3}, exit 0);test:oracle-gaps:fast(33,893 tests) andtest(115 tests) (r125-g3-fast, exit 0).oracle-gaps: errorfrom the error line (r125-m3-prefix). The law names that line as a stray.r125-m3-esc): "stderr has 2 done lines". The forged done line from the message comes out as a second done line.r125-m3-path): straybreak/error.log.r125-mut-b1). It was killed, and it passed with the loose reader (r125-mut-b-loose1).endRun, but no history reaches it; that path is checked by inspection.[vite]form, and no calibration covers it. A form of those tools thatexemptBlocksdoes not name is read as a stray, so it fails loudly.error.logsays is not judged. The streams of--stop-daemonare not observed. v21.2 P10's note-log first-line clause is not part of this PR.CI
The first round's run, at 7e9dd69, passed. Its only failures were Windows e2e shards 2/10 and 4/10. Those shards are non-blocking, and they fail the same way on main: the Windows idle gap, and the abort witness that waits for #79. Shard 2/10 took 11m59s. The review traces the overrun to the unit "every outcome class", which this PR does not touch. The same overrun is on #121's head (11m34s), and it is tracked in #118. The run for 3b11762 (43917930134) has the same result. Every Linux, framework, Vitest 5, static, unit and Windows unit job passes, and the longest Linux job took 7m28s. Windows e2e 2/10 and 4/10 fail with the same two pre-existing causes. Shard 2/10 took 10m30s, again over P12's 10-minute budget, and the overrun is tracked in #118.
This pull request and its description were written by Isaac.