Eight steps that are worth drawing, and arithmetic that is settled rather than hoped (#11)

* Check the model's sums with mpeqs: correct, and nearly inert

mpeqs evaluates a sum exactly and refuses rather than guessing, so a step
claiming 14 * 24 * 60 = 10080 can be contradicted without asking the model to
check its own work. Wired into the loop, tested, and it does work when it
fires: the wrong figure is handed back and the step is redone around it.

IT HARDLY EVER FIRES. One correction across 50 runs. A scan of 153 saved
reasoning steps found 11 checkable sums and none of them wrong.

The reason points at what would work. This model writes arithmetic as prose --
"a fortnight is 14 days, and each day has 1440 minutes, so 20160" -- not as
14 * 1440 = 20160. An extractor keyed on an explicit expression sees almost
nothing, and the errors that did occur were stated in the ANSWER as a bare
number with no working to check. Getting real value from an exact solver means
asking the model for a SPEC it can evaluate, not scraping prose for something
shaped like a sum. That is a different experiment.

AND A MORE IMPORTANT FINDING, BY ACCIDENT. Two arms here are the same
configuration measured twice: decompose8 predates the gate, and arith_off runs
those settings with the gate explicitly off. They scored 84 % and 74 %.

That is the noise floor -- about ten points at two runs per question, on 25
gradeable questions against a 3B model. It was not designed as a replication
and is more useful for having been one.

Every comparison in docs/development.md is now written against it. The large
results survive: 55 % to 91 % is five times the noise. But the four-point gap
between the gate on and off is not a result, and neither was "decomposition
beats a floor by four points" -- that claim is now stated with the uncertainty
it actually has.

The gate is conservative by construction, because a wrong correction is worse
than none: the left side must be arithmetic and nothing else, so "x = 5" is
left alone, and the tolerance comes from how many decimals the model chose to
write, so 28.27 for 28.2743 is correct rounding rather than an error. Seven
tests pin those, including the two real failures this was built for.

* Ask the model for the expression, and make sure the answer gets the result

The prose-scraping gate was nearly inert: 11 checkable sums in 153 steps, none
wrong, because a model writes "14 days at 1440 minutes each, so 20160" rather
than an expression an evaluator can settle.

So ask for it. The step schema gains a `calc` field, and mpeqs evaluates it in
exact fractions. Three things had to be true for that to work, and each was
measured rather than assumed:

* The field must be REQUIRED. Optional, the model filled it in zero times across
  three questions -- structured output omits what it is not obliged to produce.
  Required, with "" as the legal way to say "no sum here", it is filled in every
  time: 13 expressions on a single question.

* Exactness is not enough on its own. An expression is settled exactly and can
  still be the wrong expression. A provenance check -- reject literals that
  appear neither in the question nor in an earlier result -- was built and then
  dropped: it flagged 1440*14 for a fortnight, since "14 days" is world
  knowledge and not an invented number. One true positive against three false
  ones is not a gate worth having.

* The result has to reach the answer. This was the real failure. The loop
  computed 20 - 13.5 = 6.5 correctly and still answered 2.50, because the step
  holding the value was off the strongest path and the synthesis never saw it.
  Settled sums are now handed to the synthesis directly, separately from the
  thread, since they are not the model's opinion and are not up for revision.

Measured on the questions that were failing: change from a 20 note 10.50 -> 6.50,
minutes in a fortnight 336 -> 1680 -> 20160. Bat-and-ball still misses, which is
a reasoning trap rather than an arithmetic one.

* Show the sums that were settled exactly

A run's steps are the model's opinion. The sums are not: they were evaluated in
exact fractions and a reader can check "20-13.5 = 6.5" at a glance in a way they
cannot check a paragraph of reasoning. That makes them the most useful thing on
the page per pixel spent, so they get a compact strip under the progress bar.

Shown once per distinct expression -- models re-derive the same sum across
angles, and four copies say nothing the first did not -- and cleared between
runs, since a stale figure left over from the previous question would be read as
belonging to this one.

* Only show the synthesis sums from the path the answer is written from

Handing over every settled sum was worse than handing over none. A model
exploring eight angles also writes 17*250 and (17/1)*100, and listing those under
a heading that says they were calculated exactly is an invitation to pick one:
multi-step accuracy fell to 50% from 100%, and the wrong answers name the
mechanism -- "604800 minutes in a fortnight" is exactly right, for seconds in a
week, and "840" is 14*60.

So each sum now keeps the index and title of the step that produced it, and only
those on the strongest path reach the synthesis. That recovered multi-step from
50% to 72%.

What this does NOT establish is that the gate improves accuracy. Measured against
a matched control on the same questions, twice:

  arithmetic on   13/18 (72%)  then 12/18 (67%)
  arithmetic off   7/18 (39%)  then 12/18 (67%)

Pooled that is 69.4% against 52.8%, +16.7 points, 95% CI [-5.5, +38.9]. The
interval contains zero. More importantly the control swung 28 points between two
runs of an identical configuration, so the noise floor here is far wider than the
10 points previously assumed, and no comparison at this sample size decides
anything.

What is directly measured, and needs no inference: the model supplies the
expression when the field is required, and mpeqs settles it exactly. That is a
visibility feature. It is not yet an accuracy feature.

Also adds --group to the eval, so a change that can only affect one group can be
checked without paying for the 22 questions it cannot affect.

* Depend on mpeqs, and work through the angles that were planned

Two failures found by CI and by measurement, both of the same kind: something was
asked for and then quietly not used.

CI was green on tests that could not run. mpeqs was not a dependency at all, so
the gate degraded to a no-op and the tests asserting it fires failed the moment
they met a machine without it. It is pure Python with no dependencies of its own,
so it is now a real dependency rather than an optional extra -- a feature that
silently does not exist for most users is worse than a small install.

And decompose asked the model for eight angles worth checking, then let the run
stop after one step. Measured: "how many minutes in a fortnight" planned eight
angles and produced a single step. The loop no longer accepts final_answer while
angles remain unworked.

A plain step floor was tried before and produced padding -- the model reached the
answer at step 4, was told it had given 4 of 5, and wrote three that added
nothing. This is not that. Each remaining step has a named job rather than a
quota, and the steps it produces are measurably distinct: title-to-title
similarity median 0.47, max 0.81, against the 0.96 that counts as restatement.

Fortnight went from 1 step to 7, still answering 20160; "capital of France" from
1 to 8, still answering Paris.

* Pin the step count, its cost, and the noise floor that governs both

Eight steps was the requirement, at no cost to accuracy. Measured over 84 runs:

  steps    6.67 -> 8.56, and no group below 8.22
  correct  73.3% -> 74.7%, +1.3 points, 95% CI [-12.7, +15.4]
  tokens   5324 -> 7129 prompt tokens, 1.34x

So the steps arrived and the accuracy did not move. "Equal" is what was asked
for and "equal" is what this is -- the interval is far too wide to claim better.

The last claim pinned here is the one that matters most, and it is about the
measurements rather than the code: two runs of an IDENTICAL configuration --
same questions, same model, same code, differing in nothing at all -- scored 39%
and 67%. A 28-point noise floor at n=18. Every comparison in this repository
should be read against it, including the ones already recorded here, and any
claim resting on a smaller gap is folklore rather than measurement.

Pinning the minimum across groups rather than only the mean is deliberate: a
mean of 8.5 is also what you get from simple questions stopping at 3 and hard
ones running to 14.

Author: punnerud

dev/scripts/check_numbers.py

@@ -288,6 +288,53 @@
         0.0,
         "no run in the eval errored",
     ),
+    # --- Working through the planned angles, rather than abandoning them ---
+    (
+        EVAL,
+        lambda d: float(d["angle_floor"]["overall"]["steps"]["mean"]),
+        8.56,
+        1.0,
+        "a run works through its angles: about 8.5 steps, up from 6.7",
+    ),
+    (
+        EVAL,
+        # Every group, not just the mean. A mean of 8.5 is also what you get from
+        # simple questions stopping at 3 and hard ones running to 14.
+        lambda d: min(g["steps"]["mean"] for g in d["angle_floor"]["per_group"].values()),
+        8.22,
+        0.9,
+        "no group falls below eight steps",
+    ),
+    (
+        EVAL,
+        # The requirement was eight steps at no cost to accuracy. This pins the
+        # "no cost" half: the arms are within a point and a half of each other.
+        lambda d: abs(d["angle_floor"]["overall"]["correct_rate"]
+                      - d["calc_on"]["overall"]["correct_rate"]) * 100,
+        1.3,
+        6.0,
+        "the extra steps cost no accuracy",
+    ),
+    (
+        EVAL,
+        lambda d: (d["angle_floor"]["overall"]["prompt_tokens"]["mean"]
+                   / d["calc_on"]["overall"]["prompt_tokens"]["mean"]),
+        1.34,
+        0.25,
+        "and cost about a third more prompt tokens",
+    ),
+    (
+        EVAL,
+        # THE NOISE FLOOR, pinned because it governs how every other comparison
+        # here should be read. Two runs of an identical configuration -- same
+        # questions, same model, same code, differing in nothing -- scored 39% and
+        # 67%. Any claim resting on a gap smaller than that is folklore.
+        lambda d: abs(d["calc_control"]["overall"]["correct_rate"]
+                      - d["calc_control_b"]["overall"]["correct_rate"]) * 100,
+        27.8,
+        6.0,
+        "identical configurations differ by ~28 points at n=18: read every gap against this",
+    ),
 ]
 
 

dev/scripts/eval.py

@@ -153,19 +153,22 @@ def grade(answer: str, spec: dict) -> bool | None:
 
 
 def run_once(spec: dict, chat, embedder, *, synthesise=True, budget=120.0, detect=True,
-             system_prompt="", min_steps=0, decompose=0) -> dict:
+             system_prompt="", min_steps=0, decompose=0, check_arith=True) -> dict:
     chat.reset_usage()
     started = time.time()
     steps, answer, error = [], "", None
 
-    repeats = 0
+    repeats, sums_fixed = 0, 0
     for event in reason(spec["q"], chat=chat, embedder=embedder, synthesise=synthesise,
                         time_budget=budget, detect_repeats=detect, system_prompt=system_prompt,
-                        min_steps=min_steps, decompose=decompose):
+                        min_steps=min_steps, decompose=decompose,
+                        check_arithmetic=check_arith):
         if event["type"] == "step":
             steps.append(event["content"])
         elif event["type"] == "repeat":
             repeats += 1
+        elif event["type"] == "arithmetic":
+            sums_fixed += 1
         elif event["type"] == "final":
             answer = event["content"]
         elif event["type"] == "error":
@@ -184,6 +187,7 @@ def run_once(spec: dict, chat, embedder, *, synthesise=True, budget=120.0, detec
         "calls": usage["calls"],
         "seconds": round(time.time() - started, 2),
         "repeats_caught": repeats,
+        "sums_fixed": sums_fixed,
         "error": error,
     }
 
@@ -211,6 +215,7 @@ def stat(key):
         "calls": stat("calls"),
         "seconds": stat("seconds"),
         "repeats_caught": sum(r.get("repeats_caught", 0) for r in results),
+        "sums_fixed": sum(r.get("sums_fixed", 0) for r in results),
         "errors": sum(1 for r in results if r["error"]),
     }
 
@@ -220,11 +225,16 @@ def main() -> int:
     parser.add_argument("--repeats", type=int, default=3,
                         help="runs per question; one has no error bar (default 3)")
     parser.add_argument("--questions", type=int, default=0, help="cap the question count (0 = all)")
+    # Measuring one group is how a targeted change gets checked without paying for
+    # the 28 questions it cannot affect.
+    parser.add_argument("--group", default="", help="only this group (simple, multi_step, hard, ...)")
     parser.add_argument("--label", default="baseline", help="what this arm is called in the output")
     parser.add_argument("--out", default="docs/claims/eval.json")
     parser.add_argument("--no-synthesis", action="store_true",
                         help="take the last step as the answer, as the baseline did")
     parser.add_argument("--budget", type=float, default=120.0, help="seconds per run")
+    parser.add_argument("--no-arithmetic-check", action="store_true",
+                        help="do not verify the model's sums with mpeqs")
     parser.add_argument("--decompose", type=int, default=0,
                         help="plan N angles first, then take one step per angle")
     parser.add_argument("--min-steps", type=int, default=0,
@@ -241,7 +251,11 @@ def main() -> int:
 
     chat = backends.OllamaChat(backends.pick_chat_model())
     embedder = backends.OllamaEmbedding("")
-    specs = QUESTIONS[: args.questions] if args.questions else QUESTIONS
+    specs = [q for q in QUESTIONS if q["group"] == args.group] if args.group else QUESTIONS
+    if args.group and not specs:
+        parser.error(f"no group {args.group!r}; have "
+                     + ", ".join(sorted({q['group'] for q in QUESTIONS})))
+    specs = specs[: args.questions] if args.questions else specs
 
     print(f"arm={args.label}  chat={chat.model}  embeddings={embedder.model}")
     print(f"{len(specs)} questions x {args.repeats} runs = {len(specs) * args.repeats} runs\n")
@@ -253,7 +267,8 @@ def main() -> int:
             result = run_once(spec, chat, embedder, synthesise=not args.no_synthesis,
                               budget=args.budget, detect=not args.no_repeat_detection,
                               system_prompt=SHORT_SYSTEM_PROMPT if args.short_prompt else "",
-                              min_steps=args.min_steps, decompose=args.decompose)
+                              min_steps=args.min_steps, decompose=args.decompose,
+                              check_arith=not args.no_arithmetic_check)
             results.append(result)
             marks.append("-" if result["correct"] is None else ("." if result["correct"] else "x"))
         print(f"  [{n:>2}/{len(specs)}] {''.join(marks)}  {spec['group']:<11} {spec['q'][:52]}", flush=True)

dev/tests/conftest.py

@@ -15,8 +15,13 @@
 from mpe_lkg.backends import DeterministicEmbedding, ScriptedChat  # noqa: E402
 
 
-def step(title: str, content: str, next_action: str = "continue") -> str:
-    return json.dumps({"title": title, "content": content, "next_action": next_action})
+def step(title: str, content: str, next_action: str = "continue", calc: str = "") -> str:
+    # calc is omitted unless asked for, so the scripts that predate it stay honest
+    # about what a model without the field sends.
+    body = {"title": title, "content": content, "next_action": next_action}
+    if calc:
+        body["calc"] = calc
+    return json.dumps(body)
 
 
 def normal_script(n_steps: int = 6) -> list[str]:

dev/tests/test_arithmetic.py

@@ -0,0 +1,101 @@
+"""The arithmetic gate, and the part of it that turned out to matter.
+
+The gate has two halves, and measurement separated them sharply:
+
+* Scraping sums out of prose found 11 checkable claims in 153 steps and zero
+  errors, because a model writes "14 days at 1440 minutes each, so 20160" rather
+  than an expression. Nearly inert.
+* Asking for the expression outright works -- but only once the schema field is
+  *required*. Optional, the model never filled it in across three questions;
+  structured output simply omits what it is not obliged to produce.
+
+And then the failure that outlived both: the loop computed 20 - 13.5 = 6.5
+exactly and still answered 2.50, because the step holding the value was not on
+the strongest path and the synthesis never saw it. These tests pin that the
+settled sums reach the synthesis, which is the only place they can do any good.
+"""
+
+from __future__ import annotations
+
+from fractions import Fraction
+
+import pytest
+
+from mpe_lkg.arithmetic import as_text, evaluate
+from mpe_lkg.backends import STEP_SCHEMA
+from mpe_lkg.reasoning import _synthesise
+
+
+class Recorder:
+    """A chat backend that answers nothing and remembers what it was asked."""
+
+    def __init__(self) -> None:
+        self.prompts: list[str] = []
+
+    def stream(self, messages, max_tokens, schema=None):
+        self.prompts.append(messages[-1]["content"])
+        yield "an answer"
+
+
+def test_calc_is_required_so_the_model_cannot_skip_it():
+    # Not a style preference. As an optional property this field was supplied
+    # zero times out of three real runs; required, it was supplied every time.
+    assert "calc" in STEP_SCHEMA["required"]
+    assert STEP_SCHEMA["properties"]["calc"]["type"] == "string"
+
+
+@pytest.mark.parametrize(
+    ("expression", "expected"),
+    [
+        ("(17/100)*250", "42.5"),
+        ("14*24*60", "20160"),
+        ("20-13.5", "6.5"),
+        ("3*4.5", "13.5"),
+        # Written the way models write it, rather than the way Python parses it.
+        ("60 × 24", "1440"),
+    ],
+)
+def test_expressions_evaluate_exactly(expression, expected):
+    assert as_text(evaluate(expression)) == expected
+
+
+@pytest.mark.parametrize("text", ["", "   ", "the answer is 42", "20160", None])
+def test_non_expressions_yield_nothing_rather_than_guessing(text):
+    # Silence is the safe failure here: a wrong correction is worse than none.
+    assert evaluate(text) is None
+
+
+def test_settled_sums_reach_the_synthesis():
+    """The measured bug: a correct value computed and then dropped.
+
+    The value lived in a step that was not on the strongest path, so the thread
+    handed to the synthesis did not contain it and the answer was wrong. The sums
+    are passed separately for exactly that reason.
+    """
+    chat = Recorder()
+    _synthesise(chat, "What is my change?", ["I worked out the total."], ["20-13.5 = 6.5"])
+
+    asked = chat.prompts[0]
+    assert "20-13.5 = 6.5" in asked
+    assert "Do not recompute" in asked
+
+
+def test_repeated_sums_are_stated_once():
+    # Models re-derive the same expression across angles; seeing it four times
+    # tells the synthesis nothing extra and crowds out the thread.
+    chat = Recorder()
+    _synthesise(chat, "q", ["step"], ["3*4.5 = 13.5", "3*4.5 = 13.5", "20-13.5 = 6.5"])
+    assert chat.prompts[0].count("3*4.5 = 13.5") == 1
+
+
+def test_synthesis_without_sums_is_unchanged():
+    """No sums must mean no extra instruction -- most questions have no arithmetic."""
+    chat = Recorder()
+    _synthesise(chat, "Why is the sky blue?", ["Rayleigh scattering."], [])
+    assert "calculated exactly" not in chat.prompts[0]
+
+
+def test_fractions_survive_without_becoming_floats():
+    # The reason for mpeqs rather than eval(): a third stays a third.
+    assert evaluate("1/3") == Fraction(1, 3)
+    assert evaluate("(1/3)*3") == Fraction(1)

dev/tests/test_render.py

@@ -289,3 +289,36 @@ def test_steps_carry_the_id_the_graph_looks_them_up_by(self, page, tmp_path):
             run_query(page, server)
             ids = page.evaluate("() => [...document.querySelectorAll('#steps .step')].map(e => e.id)")
         assert ids == [f"step-{i}" for i in range(1, len(ids) + 1)]
+
+    def test_an_exactly_evaluated_sum_is_shown(self, page, tmp_path):
+        """The one part of a run the model did not decide, so it is worth showing.
+
+        A reader can check "20-13.5 = 6.5" at a glance in a way they cannot check a
+        paragraph of reasoning.
+        """
+        script = [
+            step("Total", "Three items at 4.50 each.", calc="3*4.5"),
+            step("Change", "Subtract from the note.", "final_answer", calc="20-13.5"),
+            "Your change is 6.50.",
+        ]
+        with LiveServer(script, tmp_path) as server:
+            run_query(page, server, "What is my change?")
+            chips = page.locator("#sums span")
+            assert chips.count() == 2
+            shown = [chips.nth(i).inner_text() for i in range(2)]
+            assert "3*4.5 = 13.5" in shown
+            assert "20-13.5 = 6.5" in shown
+
+    def test_sums_from_a_previous_run_are_cleared(self, page, tmp_path):
+        # A stale "= 6.5" left over from the last question is worse than none.
+        script = [
+            step("A", "x", "final_answer", calc="2+2"), "Four.",
+            step("B", "y", "final_answer", calc="3+3"), "Six.",
+        ]
+        with LiveServer(script, tmp_path) as server:
+            run_query(page, server, "What is two plus two?")
+            assert [c.inner_text() for c in page.locator("#sums span").all()] == ["2+2 = 4"]
+            run_query(page, server, "What is three plus three?")
+            # Not ["2+2 = 4", "3+3 = 6"]: the first run's sum belongs to the first
+            # question, and leaving it up would attribute it to this one.
+            assert [c.inner_text() for c in page.locator("#sums span").all()] == ["3+3 = 6"]

dev/tests/test_stream.py

@@ -354,3 +354,70 @@ def test_detection_can_be_turned_off(self, flask_client):
                              embedder=DeterministicEmbedding(32),
                              detect_repeats=False, synthesise=False))
         assert not [e for e in events if e["type"] == "repeat"]
+
+
+class TestArithmeticGate:
+    """Sums the record can evaluate exactly are not a matter of opinion.
+
+    Measured motivation: pushing runs to eight steps lifted hard questions from
+    50 % to 62 % and dropped multi-step arithmetic from 94 % to 67 %. The failures
+    were "10080 minutes in a fortnight" -- that is a week -- and "6.00 change" from
+    a 20 note on 13.50 of goods. Neither is a reasoning failure.
+    """
+
+    def test_a_wrong_sum_is_caught_and_redone(self, flask_client):
+        wrong = step("Compute", "A fortnight is 14 * 24 * 60 = 10080 minutes.")
+        right = step("Compute", "A fortnight is 14 * 24 * 60 = 20160 minutes.", "final_answer")
+        client, _ = flask_client([wrong, right, "20160 minutes."])
+
+        events = read_events(client.get("/query?query=q"))
+        caught = [e for e in events if e["type"] == "arithmetic"]
+
+        assert caught, "the sum is wrong and the record can prove it"
+        assert "20160" in caught[0]["errors"][0]
+        assert not [e for e in events if e["type"] == "step" and "10080" in e["content"]]
+
+    def test_a_correct_sum_passes_untouched(self, flask_client):
+        client, _ = flask_client([step("Compute", "14 * 24 * 60 = 20160 minutes.", "final_answer"),
+                                  "20160 minutes."])
+        events = read_events(client.get("/query?query=q"))
+        assert not [e for e in events if e["type"] == "arithmetic"]
+
+    def test_rounding_is_not_treated_as_an_error(self, flask_client):
+        """28.27 for 28.2743 is correct rounding, and flagging it would be wrong."""
+        client, _ = flask_client([step("Area", "The area is 3.14159 * 9 = 28.27.", "final_answer"),
+                                  "About 28.27."])
+        events = read_events(client.get("/query?query=q"))
+        assert not [e for e in events if e["type"] == "arithmetic"]
+
+    def test_a_definition_is_not_a_sum(self, flask_client):
+        """x = 5 is a definition. Checking it would invent an error."""
+        client, _ = flask_client([step("Set up", "Let x = 5 and y = 12.", "final_answer"),
+                                  "x is 5."])
+        events = read_events(client.get("/query?query=q"))
+        assert not [e for e in events if e["type"] == "arithmetic"]
+
+    def test_a_model_that_will_not_correct_itself_still_terminates(self, flask_client):
+        wrong = step("Compute", "14 * 24 * 60 = 10080 minutes.")
+        client, _ = flask_client([wrong], repeat_last=True)
+
+        events = read_events(client.get("/query?query=q"))
+        assert events[-1]["type"] == "done_stream"
+        assert len([e for e in events if e["type"] == "arithmetic"]) <= 3
+
+    def test_how_many_sums_were_checked_is_reported(self, flask_client):
+        client, _ = flask_client([step("Compute", "2 + 2 = 5.", "final_answer"),
+                                 step("Compute", "2 + 2 = 4.", "final_answer"), "Four."])
+        done = [e for e in read_events(client.get("/query?query=q")) if e["type"] == "done"][0]
+        assert done["sums_checked"] >= 1
+        assert done["sums_corrected"] >= 1
+
+    def test_the_gate_can_be_turned_off(self, flask_client):
+        from mpe_lkg.backends import DeterministicEmbedding, ScriptedChat
+        from mpe_lkg.reasoning import reason
+
+        wrong = step("Compute", "14 * 24 * 60 = 10080 minutes.", "final_answer")
+        events = list(reason("q", chat=ScriptedChat([wrong], repeat_last=True),
+                             embedder=DeterministicEmbedding(32),
+                             check_arithmetic=False, synthesise=False))
+        assert not [e for e in events if e["type"] == "arithmetic"]