feat(repo): repo compare intervals invert the sign-flip test (#292)

dokimi/changelog.d/20261105_120000_antonwolfy_compare-sign-flip-interval.md

@@ -0,0 +1,17 @@
+### Changed
+
+- **Every `repo compare` interval now inverts the sign-flip test** — a row's 95% interval was a
+  percentile bootstrap (gate, score) or a t interval (the other measures) next to a sign-flip p, and near
+  0.05 the two disagreed. It is now, for every measure, every shift the row's sign-flip test does not
+  reject at the uncorrected 5% level (exact up to 16 changed tasks, Monte Carlo above with the draws of p,
+  made once from the seed, so the ends are the same on every run), and it excludes 0 exactly when p is
+  below 0.05. Under a shift tied tasks count too, so both ends are bounded from 6 tasks on and open at 5
+  or fewer: `delta.lo` / `delta.hi` are then `null` in `--json` (before, only a single task had `null`
+  ends) and print as `-inf` / `+inf`, in `repo compare` and in the `--ledger` text. There, a non-zero end
+  that 3 decimals would print as `0.000` prints at 2 significant digits (`+0.0002`), so an interval that
+  excludes 0 no longer reads `[+0.000, +0.000]`; in `repo compare --json`, `delta.mean` stays inside the
+  ends (six deltas of `4e-5`: mean `0.0001`, not `0.0`). Each `--json` row, and each rule result of the ledger,
+  names its method, `delta.method` `sign_flip_inversion` (`null` for a rule result with no pairs); the ledger's ends
+  stay at 4 places, as before. `interval_p_disagree` now fires only where
+  Holm's correction lifts a gate or score row's p across 0.05, and its note says the interval is
+  uncorrected and `p_holm` decides the row. Printed p, `p_holm`, `mde` and `p_holm_floor` do not change.

dokimi/docs/architecture.md

@@ -359,9 +359,9 @@ next release.
 - `dokimi/repos/compare.py` — `repo compare`: treatment arms against a
   baseline arm over one or more batches, one value per task and arm (gate
   share, score, cost, turns, chosen `grade.json` metrics, `result.json`
-  `process` counts), with an interval
-  over tasks (bootstrap for gate and score, t for the rest), a sign-flip
-  permutation p and Holm's correction. `dokimi/repos/compare_tasks.py` holds
+  `process` counts), with a sign-flip permutation p, an interval over tasks
+  that inverts it (`dokimi/repos/compare_interval.py`) and Holm's correction.
+  `dokimi/repos/compare_tasks.py` holds
   what its `--task` selection keeps; `dokimi/repos/compare_reach.py` how small
   an effect a row can find (`mde`, an uncorrected t-based estimate, and the
   lowest `p_holm` Holm's family leaves it).

dokimi/docs/repo-evaluation-contract.md

@@ -2063,10 +2063,21 @@ python cli.py repo hits judge repo-runs/replay --judge openai/gpt-5.1 --labels l
   task's cells that observed the first useful result, §4.1), `first_useful_rate` (the share of a task's
   declared cells with a verdict that observed one), each `--metric` from `grade.json`, and each `process` count a
   compared cell reports as `process:<key>` or `counter:<name>`) it gives the mean per-task
-  delta against the baseline with a 95% interval over tasks (a bootstrap for
-  gate and score, a t interval for the descriptive measures, `-` for a single
-  task), the tasks up, down and tied, and a two-sided sign-flip permutation p (exact up to 16
-  changed tasks; at one cell per task and arm it equals McNemar's exact test).
+  delta against the baseline with a 95% interval over tasks, the tasks up, down and tied, and a two-sided
+  sign-flip permutation p (exact up to 16 changed tasks, Monte Carlo above with a fixed seed; at one cell per
+  task and arm it equals McNemar's exact test). Every measure's interval inverts that test
+  (`dokimi/repos/compare_interval.py`; `delta.method` is `sign_flip_inversion` in `--json`): it is every
+  shift theta at which the test on the per-task deltas minus theta does not reject at the uncorrected two-sided
+  5% level, each end found by bisection between the mean and a far bound. It excludes 0 exactly when p is below
+  0.05: the row's own p places 0, and above 16 changed tasks the interval's Monte Carlo draws are those of p, drawn
+  once from the seed and reused at every theta, so the ends do not move between runs. Under a shift every task
+  counts, the tied ones included (their delta becomes -theta), and the smallest p over n tasks is `2 / 2^n`, so
+  both ends are bounded from 6 tasks on, whatever the ties, and open (`null` in `--json`, `-inf` / `+inf` in the
+  text) at 5 or fewer, one task included. The test stays exact, with no mid-p: p(theta) is a step function and the
+  ends are its step points, so an end can be a delta itself, or 0. The text prints an end at 3 decimals, and a
+  non-zero end that would print as `0.000` at 2 significant digits (`+0.0002`), so it keeps its side of 0.
+  `delta.mean` (4 places) stays inside those ends: six deltas of `4e-5` give a mean of `0.0001`, not `0.0`, as
+  both ends are `0.0001`.
   A `process` count, `first_prompt_tokens` or `first_useful_*` only one arm reports for the shared tasks of a
   (treatment, model) is not compared; a warning names it. When neither arm
   reports it (a batch run before `repo run` recorded it), the row is absent.
@@ -2093,12 +2104,11 @@ python cli.py repo hits judge repo-runs/replay --judge openai/gpt-5.1 --labels l
   0.05, `mde` is `null` and the note names Holm instead (`with Holm over 2
   rows, p_holm cannot go below 0.0625 here: no effect can reach p_holm <
   0.05`); a family of one row is the raw rule. `p_holm_floor` decides only
-  whether `mde` is given, not its size. A row whose interval and p disagree (the
-  interval excludes 0 while p is 0.05 or more, or includes 0 while p is below
-  0.05; for gate and score that p is `p_holm`, the one that decides them, and the
-  note quotes it) has `interval_p_disagree` `true` and a note: they come from two
-  methods, and for gate and score `p_holm` is the one to read, since a percentile
-  bootstrap runs narrow over few tasks; the inconclusive note gives way to
+  whether `mde` is given, not its size. A gate or score row whose interval
+  excludes 0 while its `p_holm` is 0.05 or more (Holm's correction lifted its p
+  across 0.05; the interval agrees with the raw p, and Holm's step-down has no
+  interval to match) has `interval_p_disagree` `true` and a note: the interval
+  is uncorrected; `p_holm` decides this row. The inconclusive note gives way to
   it. Descriptive rows with p below 0.05 are named on one line: their p is
   uncorrected and outside Holm's family. `--task ID` (repeatable) compares only
   those tasks: an id no finished cell of the batches has is refused, the tasks

dokimi/dokimi/repos/compare.py

@@ -36,11 +36,11 @@
 are listed as unpaired, with the reason (no cells in one arm, a task kind that
 differs between the arms, a measure one arm lacks). Per measure the comparison
 gives the mean per-task delta (treatment minus baseline) with a 95% interval
-over tasks (a percentile bootstrap for gate and score, a t interval for the
-continuous measures; none for a single task), the tasks that went up, down or
-stayed, and a two-sided sign-flip
+over tasks, the tasks that went up, down or stayed, and a two-sided sign-flip
 permutation p on the per-task deltas: exact up to :data:`EXACT_MAX` non-zero
-deltas, Monte Carlo above. At one cell per task and arm the deltas are -1, 0
+deltas, Monte Carlo above. The interval inverts that test (:mod:`.compare_interval`, ``delta.method``): every
+shift theta the test on the deltas minus theta does not reject at 5%, uncorrected, so it excludes 0 exactly when
+p < 0.05; an end is open (``None``) at 5 or fewer tasks. At one cell per task and arm the deltas are -1, 0
 or +1 and the exact p equals McNemar's exact test. Holm's correction runs over
 the gate and score comparisons of one invocation; cost, turns and metrics are
 descriptive and keep their raw p.
@@ -54,10 +54,9 @@
 effect of any size reaches p < 0.05: ``mde`` is ``None`` there too, and the note says how low p can go instead.
 ``mde`` is ``None`` too where a gate or score row's ``p_holm_floor`` (its ``p_holm`` with its p at that floor and the
 family's other p as observed, :mod:`.compare_reach`) is not below 0.05, and the note names Holm: that floor decides
-whether ``mde`` is given, not its size. A row whose interval and p disagree (the interval excludes 0 while p is 0.05
-or more, or includes it while p is below; for gate and score the p is ``p_holm``, the one that decides them) has
-``interval_p_disagree``: the two come from different methods, and for gate and score ``p_holm`` is the one to read,
-since the percentile bootstrap runs narrow over few tasks (the inconclusive note then gives way to this one). A
+whether ``mde`` is given, not its size. A gate or score row whose interval excludes 0 while its ``p_holm`` is 0.05
+or more (Holm lifted its p across 0.05; the interval agrees with the raw p) has ``interval_p_disagree``: the
+interval is uncorrected, and ``p_holm`` decides the row (the inconclusive note then gives way to this one). A
 descriptive row with p below 0.05 is named on a line saying its p is uncorrected and outside Holm's family.
 
 ``--task`` limits a comparison to some task ids: the others are left out, not listed as unpaired, and the
@@ -111,9 +110,11 @@
 from typing import Any
 
 from dokimi.context.consistency import Run, run_from_events
-from dokimi.eval.stats import bootstrap_ci, correct_family, mean_ci
+from dokimi.eval.stats import correct_family
 from dokimi.utils import same_model
 
+from .compare_interval import (EPS, EXACT_MAX, METHOD, N_RESAMPLES, format_interval, includes_zero,
+                               sign_flip_interval, within)
 from .compare_reach import ALPHA, holm_floor, min_detectable
 from .compare_tasks import CompareError, select_tasks
 from .compare_values import finite_values, task_values
@@ -122,12 +123,6 @@
                      unmeasured_note, uptake_counts)
 from .useful import USEFUL_MEASURES, useful_arms_line, useful_counts
 
-#: Largest number of non-zero deltas whose sign-flip distribution is enumerated (2**16 patterns).
-EXACT_MAX = 16
-#: Resamples of the Monte Carlo sign-flip test above :data:`EXACT_MAX`.
-N_RESAMPLES = 20_000
-#: Deltas closer to zero than this are ties.
-EPS = 1e-9
 #: Measures Holm's correction runs over.
 PRIMARY = ("gate", "score")
 #: Descriptive measures compared only when both arms report them for some shared task (an older batch has none).
@@ -183,7 +178,8 @@ def compare(batch_dirs: list[Path], baseline: str, treatments: list[str] | None
             models: list[str] | None = None, metrics: tuple[str, ...] = (), seed: int = 0,
             pass_k: int | None = None, path_stability: bool = False, tasks: list[str] | None = None,
             *, exact: bool = False) -> dict[str, Any]:
-    """``exact``: the statistics at full precision (the ledger's verdicts read them); otherwise rounded as printed."""
+    """``exact``: p and the mean delta at full precision (the ledger's verdicts read them); otherwise rounded as
+    printed. The interval's ends are at 4 places either way: bisection only knows them within 1e-7."""
     metrics = tuple(dict.fromkeys(metrics))
     if pass_k is not None and pass_k < 1:
         raise CompareError(f"--pass-k must be at least 1, got {pass_k}")
@@ -363,7 +359,7 @@ def _holm(rows: list[dict[str, Any]]) -> None:
     for i, (row, (p, below)) in enumerate(zip(primary, correct_family(ps), strict=True)):
         d = row["delta"]
         row["p_holm"] = p
-        row["interval_p_disagree"] = d["lo"] is not None and below is not None and (d["lo"] <= 0 <= d["hi"]) == below
+        row["interval_p_disagree"] = below is not None and includes_zero(d["lo"], d["hi"]) == below
         if p is not None:
             row["p_holm_floor"], reachable = holm_floor(ps, i, min_sign_flip_p(row["up"] + row["down"]))
             row["mde"] = row["mde"] if reachable else None
@@ -555,28 +551,29 @@ def paired_summary(pairs: list[dict[str, Any]], measure: str = "gate", seed: int
 
     The values are taken at full precision and rounded only in the output: rounded rates such as 1/3 and 2/3
     no longer sum to the same totals, which would move sign patterns across the observed one in the
-    permutation test. The interval is a percentile bootstrap for the bounded gate and score shares and a t
-    interval (:func:`~dokimi.eval.stats.mean_ci`) for the continuous measures, where the bootstrap
-    under-covers at these task counts; one task has no interval (``lo``/``hi`` None), not a zero-width one.
-    ``mde`` and ``interval_p_disagree`` are the reading aids of the module docstring; both read the raw p here,
-    and :func:`compare` revisits both on a gate or score row once Holm's p is known.
+    permutation test. Every measure's interval inverts the row's sign-flip test (:func:`.sign_flip_interval`,
+    ``delta.method``), so it excludes 0 exactly when the raw p is below 0.05; an end is ``None`` where it is open,
+    as both are at 5 or fewer tasks, one task included. ``mde`` and ``interval_p_disagree`` are the reading aids of
+    the module docstring; both read the raw p here, and :func:`compare` revisits both on a gate or score row once
+    Holm's p is known. ``measure`` is not read: every measure gets the same statistics.
     """
     deltas = [p["delta"] for p in pairs]
-    ci = (bootstrap_ci if measure in PRIMARY else mean_ci)(deltas, seed=seed)
-    lo, hi = (float(ci["lo"]), float(ci["hi"])) if len(deltas) > 1 else (None, None)  # mean_ci gives numpy floats
     p = sign_flip_p(deltas, seed=seed, exact=exact)
+    lo, hi = sign_flip_interval(deltas, p, seed=seed)
+    mean = sum(deltas) / len(deltas)
     sd = statistics.stdev(deltas) if len(deltas) > 1 else 0.0
     reachable = min_sign_flip_p(sum(abs(d) > EPS for d in deltas)) < ALPHA  # else no effect can be detected
     return {
         "n_tasks": len(pairs),
         "baseline_mean": _round(_mean([p["baseline"] for p in pairs])),
         "treatment_mean": _round(_mean([p["treatment"] for p in pairs])),
-        "delta": {"mean": sum(deltas) / len(deltas) if exact else ci["mean"], "lo": lo, "hi": hi},
+        "delta": {"mean": mean if exact else within(round(mean, 4) + 0.0, lo, hi), "lo": lo, "hi": hi,  # no -0.0
+                  "method": METHOD},
         "p": p,
         "p_holm": None,
         "p_holm_floor": None,
         "mde": _round(min_detectable(sd, len(deltas))) if sd > EPS and reachable else None,
-        "interval_p_disagree": lo is not None and hi is not None and p is not None and (lo <= 0 <= hi) == (p < ALPHA),
+        "interval_p_disagree": p is not None and includes_zero(lo, hi) == (p < ALPHA),
         "up": sum(d > EPS for d in deltas),
         "down": sum(d < -EPS for d in deltas),
         "tied": sum(abs(d) <= EPS for d in deltas),
@@ -626,7 +623,7 @@ def format_compare(summary: dict[str, Any]) -> str:
                   f"{'delta [95% CI]':<26}  {'p':>6}  {'p_holm':>6}  up/down/tied"]
         for r in rows:
             d = r["delta"]
-            delta = f"{d['mean']:+.3f} " + ("-" if d["lo"] is None else f"[{d['lo']:+.3f}, {d['hi']:+.3f}]")
+            delta = f"{d['mean']:+.3f} " + format_interval(d["lo"], d["hi"])
             lines.append(f"  {r['measure']:<{width}}  {r['n_tasks']:>5}  {_num(r['baseline_mean']):>9}  "
                          f"{_num(r['treatment_mean']):>9}  {delta:<26}  {_p(r['p']):>6}  {_p(r['p_holm']):>6}  "
                          f"{r['up']}/{r['down']}/{r['tied']}")
@@ -657,14 +654,9 @@ def _reading_notes(rows: list[dict[str, Any]], family: int) -> list[str]:
     out = []
     for r in rows:
         d, primary = r["delta"], r["measure"] in PRIMARY
-        if r["interval_p_disagree"]:
-            inside = d["lo"] <= 0 <= d["hi"]
-            how = "a percentile bootstrap, too narrow over few tasks," if primary else "a t interval"
-            name = "p_holm" if primary else "p"  # a gate or score row is decided by its Holm p (_holm)
-            out.append(f"  {r['measure']}: the interval [{d['lo']:+.3f}, {d['hi']:+.3f}] "
-                       f"{'includes' if inside else 'excludes'} 0 but {name} {r[name]:.4g} is {'<' if inside else '≥'} "
-                       f"{ALPHA}; they come from two methods, {how} and a sign-flip test"
-                       + ("; read p_holm" if primary else ""))
+        if primary and r["interval_p_disagree"]:  # the interval agrees with the raw p: only Holm lifts p past ALPHA
+            out.append(f"  {r['measure']}: the interval {format_interval(d['lo'], d['hi'])} excludes 0 but p_holm "
+                       f"{r['p_holm']:.4g} is ≥ {ALPHA}: the interval is uncorrected; p_holm decides this row")
         elif primary and (note := _inconclusive_note(r, family)):
             out.append(note)
     low = [r["measure"] for r in rows if r["measure"] not in PRIMARY and r["p"] is not None and r["p"] < ALPHA]
@@ -676,8 +668,7 @@ def _reading_notes(rows: list[dict[str, Any]], family: int) -> list[str]:
 
 def _inconclusive_note(r: dict[str, Any], family: int) -> str | None:
     """The note of a gate or score row of 2+ tasks whose interval includes 0: its ``mde``, or why it has none."""
-    d = r["delta"]
-    if d["lo"] is None or not d["lo"] <= 0 <= d["hi"]:
+    if r["n_tasks"] < 2 or not includes_zero(r["delta"]["lo"], r["delta"]["hi"]):
         return None
     head = f"  {r['measure']} inconclusive: the interval includes 0; "
     if r["mde"] is not None: