[napetrov]

Summary

Second half of the exit-code work in #275 (stage A2). Input refusals now raise InputError, so cli.main prints one Error: <message> line and exits 2 from a single place. Two run failures move from exit 2 to exit 1, the code every other runtime failure uses.

before: refusal -> print(...) ; sys.exit(2)         (per command, 55 sites)
after:  refusal -> raise InputError(msg)            -> cli.main: "Error: msg", exit 2
run failure -> exit 1 after writing the report      (pairwise all-failed, dead --judges judge)

• Converted to raise InputError: commands/ answer_formats (4), answers (1), evaluate (13), headroom (16), ingest (out-dir write), judge_reliability (8), redteam (2), run (1), dataset (export), baseline (malformed records now propagate; RecordError subclasses InputError).
• Exit-site ratchet (tests/test_input_refusal_exit_sites.py) counts every exit 2 in the frontend: a literal, os._exit, quit, and a module-level constant holding 2 (EXIT_USAGE). The 138 EXIT_USAGE sites not converted yet are frozen per module in EXIT2_BASELINE (143 with the 5 literal exits that stay); each count drops as its site converts.
• Test helper tests/cli_refusal.py::refusal() runs a command in-process and reports exit code and output as cli.main would. About 12 test files now use it. tests/test_cli_refusal_parity.py runs two converted commands through the helper and through cli.main and requires the same code, stdout and stderr.

Part of #275. Not Closes: the items listed under "Left out" remain open.

Evidence

• Full suite on this branch (head 2d2f745, merged with intel/main 5ce3207): 4 failed, 15692 passed, 11 skipped. The same 4 fail on a clean intel/main (5ce3207) in this environment: tests/test_repo_typecheck_grader.py (_image_provides and the shadowing cases). No new failures.
• New pinned tests: tests/test_run_failure_exit_codes.py (pairwise all-failed writes the report and exits 1), tests/test_cross_judge_judge_errors.py and tests/test_judge_not_measured.py (dead judge exits 1), tests/test_input_refusal_exit_sites.py (ratchet: a fixture module with each spelling of exit 2 is flagged), tests/test_cli_refusal_parity.py (helper vs cli.main), tests/test_baseline_unreadable_input.py (unreadable record prints Error: and exits 2).
• ruff check, mypy dokimi, and make ci-local CI_LOCAL_ARGS="--offline --jobs static,typecheck" pass (rc 0). Two non-blocking docs-review warnings: docs/answer-formats.md and docs/ingestion-docling.md are not updated for the exit-code change.

Merge Danger

• Exit code change for two run failures, 2 → 1: eval score pairwise when every comparison fails (the errors are still written to --output), and a --judges judge that scores none of its answers (eval judge agreement --judges). Scripts that treat 2 as "bad input" for these must also check 1. Labelled cli-interface-change.
• Output text change: baseline compare on an unreadable eval record prints Error: <message> instead of ❌ <message>. Exit stays 2.
• Refusals now print Error: <message> through cli.main. A cross-judge run with several stored-material problems used to print one Error: line per problem; it now prints one Error: line, with the problems on the lines after it.

Left out (still open under #275)

• ~62 usage_exit() sites, and exit codes 3, 4, 5.
• 138 EXIT_USAGE refusals still written by hand in 23 frontend modules: frozen in the ratchet baseline, to convert in follow-up PRs.
• ingest "No supported documents found" stays exit 1 (a run outcome, not a refusal of the input shape); the reason is in the code.
• Typed-scorer ImportError and the missing-Docling refusal stay exit 2 (recorded in the ratchet baseline).
• cli.py / cli_routes.py route rows are not touched (owned by the M2 change).

🤖 Generated with Claude Code

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 35 selected item(s).

CI at e03d0d3: 15 passing, 2 skipped.

Review limit reached. This PR has had 6 reviews, so it will only be reviewed again when you ask.
To ask: re-request review from exai-bot in the Reviewers list.

Review details

• Commit reviewed: e03d0d3 against merge base 040f86d
• Coverage: 35 files in 10 groups
• Engine: OpenCodeReview v1.13.5-exai.3.d7112371, us.openai.gpt-6.1-sol
• Usage: 780k tokens (454k from cache), 24s
• Tool calls: 54 (33 file_read, 21 code_search)
• Shared CLI input-refusal contract and parity tests: 5 files
• Judge reliability and cross-judge error handling: 7 files
• Baseline comparison and benchmarking: 5 files
• Headroom command and tests: 2 files
• Evaluation output errors and typed scoring: 3 files
• Answer formats, dataset export, and record readers: 4 files
• Answers command and tests: 2 files
• Ingest command and CLI tests: 2 files
• Redteam command and tests: 2 files
• Run failure exit codes and quickstart guidance: 3 files

[Copilot]

🟡 Changes recommended

The changelog’s migration notices name two invalid commands, obscuring which runnable commands have changed output or exit codes.

2 open issues

• Use the runnable baseline compare command in the notice · New
• Use the registered eval judge agreement command in the notice · New

What changed in this PR

2 new findings.

This PR advances #275 by centralizing input refusals through InputError and distinguishing bad input from runtime failures in dokimi’s CLI.

Changes:

• Routes converted input refusals through cli.main for consistent error output and exit code 2.
• Changes all-failed pairwise runs and judges that score nothing to exit 1 after saving reports.
• Updates command tests, adds a refusal helper and exit-site ratchet, and documents the interface changes.

File	Description
dokimi/​tests/​test_typed_scorer.py	Adopts refusal helper and explicit exit assertions.
dokimi/​tests/​test_run_failure_exit_codes.py	Checks all-failed pairwise runs save reports and exit 1.
dokimi/​tests/​test_redteam.py	Updates refusal and verdict assertions.
dokimi/​tests/​test_record_readers.py	Updates malformed-record refusal checks.
dokimi/​tests/​test_judge_validity.py	Adapts validity refusal tests.
dokimi/​tests/​test_judge_sequential_design.py	Updates undecidable-gate refusal assertion.
dokimi/​tests/​test_judge_reliability.py	Updates refusal handling and pairwise failure expectations.
dokimi/​tests/​test_judge_not_measured.py	Expects exit 1 for a failed judge.
dokimi/​tests/​test_input_refusal_exit_sites.py	Adds a ratchet for direct exit-2 sites.
dokimi/​tests/​test_ingest_cli.py	Adapts ingest error checks.
dokimi/​tests/​test_headroom_command.py	Adapts headroom refusal checks.
dokimi/​tests/​test_eval_output_write_errors.py	Updates output-write error assertions.
dokimi/​tests/​test_datasets_export.py	Adapts export refusal checks.
dokimi/​tests/​test_cross_judge_judge_errors.py	Pins exit 1 for judges that score nothing.
dokimi/​tests/​test_baseline_unreadable_input.py	Checks centralized malformed-record error output.
dokimi/​tests/​test_baseline_compare.py	Updates comparison refusal and verdict assertions.
dokimi/​tests/​test_answers_command.py	Adapts configuration, input, and write-error checks.
dokimi/​tests/​cli_refusal.py	Adds in-process exit and output capture helper.
dokimi/​dokimi/​commands/​run.py	Raises InputError for unreadable gate bands.
dokimi/​dokimi/​commands/​redteam.py	Converts input and artifact-write refusals.
dokimi/​dokimi/​commands/​judge_reliability.py	Centralizes refusals and changes runtime failure exits.
dokimi/​dokimi/​commands/​ingest.py	Converts materialization errors to InputError.
dokimi/​dokimi/​commands/​headroom.py	Centralizes validation refusals.
dokimi/​dokimi/​commands/​evaluate.py	Converts scoring and validity input refusals.
dokimi/​dokimi/​commands/​dataset.py	Converts export refusals to InputError.
dokimi/​dokimi/​commands/​baseline.py	Lets malformed-record errors reach the central handler.
dokimi/​dokimi/​commands/​answers.py	Converts answer-write failures to InputError.
dokimi/​dokimi/​commands/​answer_formats.py	Converts import and reference-scoring refusals.
dokimi/​changelog.d/​20261104_120000_napetrov_input-refusal-exit-codes.md	Documents error-output and exit-code changes.

_{Review effort: Balanced}

[napetrov]

Fixed in 9059ad1: the notice now names baseline compare and eval judge agreement --judges, both checked against the registered routes.

[napetrov]

Fixed in 9059ad1: the notice now names baseline compare and eval judge agreement --judges, both checked against the registered routes.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.86%. Comparing base (0e993b9) to head (e03d0d3).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #393      +/-   ##
==========================================
+ Coverage   94.78%   94.86%   +0.07%     
==========================================
  Files         353      354       +1     
  Lines       46575    46539      -36     
  Branches     7974     7974              
==========================================
+ Hits        44146    44147       +1     
+ Misses       1541     1513      -28     
+ Partials      888      879       -9     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/answer_formats.py	84.33% <100.00%> (+17.67%)	⬆️
dokimi/dokimi/commands/answers.py	95.37% <100.00%> (ø)
dokimi/dokimi/commands/baseline.py	94.67% <100.00%> (-0.08%)	⬇️
dokimi/dokimi/commands/dataset.py	100.00% <100.00%> (ø)
dokimi/dokimi/commands/evaluate.py	90.88% <100.00%> (+0.53%)	⬆️
dokimi/dokimi/commands/headroom.py	98.90% <100.00%> (-0.09%)	⬇️
dokimi/dokimi/commands/ingest.py	78.57% <100.00%> (-1.43%)	⬇️
dokimi/dokimi/commands/judge_reliability.py	97.59% <100.00%> (+4.30%)	⬆️
dokimi/dokimi/commands/redteam.py	100.00% <100.00%> (ø)
dokimi/dokimi/commands/run.py	91.93% <100.00%> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 2d2f745. The conversion itself holds up: every converted site keeps its message and exit 2 through cli.main, and no raise InputError sits inside a try that would catch it as a ValueError. Three things are worth fixing before merge, one inline and two below.

What I ran, on a Linux review host at 2d2f745:

• make ci-local: 29 passed, 1 failed, 7 skipped; 15,694 tests; changed-line coverage 84%. The one failure is test_unbalanced_quote_in_hook_completes_quickly (2.6 s against a 1 s limit), plus two 60 s timeouts in test_audit_test_quality.py. They are timing failures under load and pass when run alone (173 passed); the PR touches none of that code.
• Mutation runs on the changed commands: 26 hand-written mutants and one per converted raise InputError. Both moved exits (pairwise all-failed, dead --judges judge) are pinned: mutants to exit 2 or 0 fail the tests.
• All commits carry Signed-off-by.

Findings that cannot be anchored inline:

1. benchmarking-and-comparison.md:555-557 still says exit 2 for both run failures this PR moves to exit 1 (outside this diff). It reads: a judge that "scored none of its answers exits 2 once the report is written, with or without --strict, as eval score pairwise does when every comparison fails". Both now exit 1. A script that follows this page and checks for 2 never fires. The older fragment changelog.d/20261028_120000_vlad-perevezentsev_cross-judge-judge-errors.md says the same, so it now contradicts the new fragment. Suggestion: change both to exit 1 in this PR; the exit-code change is the point of the PR.

2. Most converted refusals in judge_reliability.py, answer_formats.py and evaluate.py have no test that reaches them. Replacing each raise InputError(...) with pass survives the related test files for: judge_reliability.py:80 (cannot read questions), :213 (checklist without expected_topics), :237 (unknown wording), :288 (--grounded without --questions), :395 (bad --judges entry); answer_formats.py:28, :52, :54, :74; evaluate.py:302 (unknown degradation). The exit code is what these lines decide. At answer_formats.py:54, losing the raise would save an artifact with no evaluations and exit 0. Run through the CLI today, that case prints Error: no answered question has a grading block in --questions and exits 2, so the behaviour is right and only the pin is missing. Suggestion: add one refusal() case per site; the helper already makes it a few lines each.

[napetrov]

Both fixed: ad3ae50 and 7bad232.

Fixed in ad3ae50: the docs page and the cross-judge fragment now say exit 1.

Fixed in 7bad232: the 10 untested sites now have pinning tests.

[vlad-perevezentsev]

Review at ad3ae50. The conversion is right: every converted refusal keeps its message and exit 2 through cli.main, and the two moved run failures exit 1 after their report is written. Two small gaps are in the line comments: the exit-code table for a judge with no key, and what the ratchet counts.

What I ran, at ad3ae50:

• make ci-local --offline: 0 failed, 19 passed, 13 skipped (network and Docker jobs); full test suite, changed-line coverage, the changelog fragment check and the strict docs build pass.
• literal_exit_2_lines on a usage_exit(...) call and on sys.exit(code=2): both return [].

Checked, no issues

• Callers: every changed cmd_* is reached only through args.func in cli.main, so no in-process caller catches the old SystemExit. No new raise InputError sits inside a try that would catch it as a ValueError. In headroom, the helpers run outside the except ValueError around headroom().
• Pins: each of the ten sites @antonwolfy listed, and the pairwise "no comparable pair" refusal, now has a refusal() test that asserts exit 2 and the message, so dropping the raise fails it. test_run_failure_exit_codes.py asserts exit 1, so it fails on the base, which exits 2.
• Docs and changelog: benchmarking-and-comparison.md and the older cross-judge fragment now say exit 1. The new fragment names baseline compare and eval judge agreement --judges (both registered) and tells scripts to check for 1.
• New files: all carry the SPDX header.

[napetrov]

Thanks. Both gaps are fixed in e81c4a1 and 4646df5, details in the threads.

[xaleryb]

• Ran: the full suite, ruff, mypy, bandit, check_growth, the benchmark gate and the CLI snapshot on the head merged into main 040f86d, plus the real CLI with no provider key.
• tests/test_cli_refusal_parity.py case dataset-export-missing-input on head + main: stderr was moved: dokimi data export ..., so the test failed. Expected Error: cannot export .... refactor(cli): move skill, catalog, trace and data under their roots (#94 M2) #396 retired the dataset root, and CI passed only because it ran on a base from before refactor(cli): move skill, catalog, trace and data under their roots (#94 M2) #396.
• Changelog fragment lines 8-10 named headroom, judge, redteam and answer-formats, which are not commands (invalid choice), and dataset, which prints moved: after the merge.
• Pushed: e03d0d3 merges main. The parity test calls data export (id data-export-missing-input), and the fragment names the commands as main registers them. Suite, snapshot and parity test pass on it.
• Every earlier review thread is addressed at this head.

[vlad-perevezentsev]

LGTM

I checked e03d0d3. All three points from my last round are fixed. After the merge of #396, the refusal-parity test and the changelog fragment use the commands main registers now.

Merge: safe now: it merges cleanly into current main, 0 commits behind, and its tests pass there. Once it is in, #408 conflicts with it (evaluate.py, judge_reliability.py) and needs a rebase.

What I ran, at e03d0d3:

• make ci-local --offline: 0 failed, 19 passed, 13 skipped (network and Docker jobs); 15,898 tests passed, 63 skipped.
• --help on every command named in the changelog fragment and the quickstart line (eval score pairwise, eval judge agreement, data export, data import docling, baseline compare, ...): all are registered.

Checked, no issues

• Missing key: the quickstart exceptions paragraph now says eval judge agreement --judges and eval score pairwise exit 1 when every judge call fails, a missing key included (4646df5).
• Ratchet: usage_exit(...) calls now count, by plain name and as an attribute, and the 61 sites are in EXIT2_BASELINE with their reason (e81c4a1).
• eval judge agreement: the RecordError wrapper is gone, and the complexity baseline of cmd_eval_cross_judge went down to 29 (308882c).

[antonwolfy]

LGTM. All three points from my last review are fixed: the docs and the older cross-judge fragment now say exit 1, the untested refusals have refusal() pins, and the no-comparable-pair refusal in eval score pairwise is pinned. No new findings.

What I ran, on a Linux review host at e03d0d3:

• make ci-local: 29 passed, 1 failed, 7 skipped; 15,950 tests passed; changed-line coverage passes. The one failure is test_the_interval_excludes_zero_exactly_when_p_is_below_005 hitting the 60 s timeout during the full run. It passes alone (21 passed in 3.6 s), and it belongs to the compare_interval code that came in with the merge of main, not to this PR.
• Mutation run on the changed commands: each of the 45 converted raise InputError in judge_reliability, answer_formats, evaluate, headroom, redteam, ingest, answers, dataset and baseline replaced by pass, and both exit-1 moves changed to exit 2 and to exit 0. All are caught by the tests.
• All 12 non-merge commits carry Signed-off-by.

Checked, no issues

• Swallowed errors: no raise InputError or RecordError sits inside a try whose handler catches ValueError or Exception. No caller of a changed cmd_* catches SystemExit.
• Exit-1 moves: pairwise all-failed and the dead --judges judge write the report first, then exit 1. The docs and both changelog fragments agree.
• Ratchet: EXIT2_BASELINE is enforced in both directions and counts usage_exit(...) calls, as asked in the earlier round.