The model is the lever: qwen3:4b at 82.5% against llama3.2:3b at 40% - #18
Merged
Merged
Conversation
The failure has a name in the literature -- Result-Ignore: the final answer states something other than what the tool returned. ToolFailBench measures it at 30% on an 8B Llama, and llama3.2:3b sits below that model on every grounding benchmark that covers it (50.79% against Qwen3-4B's 88.19%). Five runs in this repository's own battery computed the right value exactly and then answered something else. Persuasion was already exhausted: a dedicated heading, an explicit instruction not to recompute, deduplication, and restricting which facts are shown. So the model no longer writes the number. It picks a LETTER, and the value comes back from our record -- what it chose and what is reported cannot drift apart. Four things had to be true, and each was measured rather than assumed: * A LABEL. "(86400*378) = 32659200" does not say it is seconds in 54 weeks. Offering unlabelled expressions as candidates scored 0 of 5 -- the model rejected all of them -- while the same values labelled scored 9 of 12. The step schema now asks what a result IS. * A label that DESCRIBES. Asked what its result was, the model wrote "604800/7" and "seconds". The first is the sum again, the second fits any of four values. Labels that are expressions, or that fit two different values, are dropped: a label that does not discriminate is worse than none, because it looks like one. * NO NUMBERS IN THE CHOICE. Letters alone. Showing the values alongside was tried and measured worse, which fits the design rather than contradicting it. * A DETERMINISTIC GATE. An enum forces a choice, and asked "what is the capital of France?" with a settled number in front of it the model chose the number 3 times out of 3 -- and 3 of 3 again when the options carried labels. So the code decides whether a numeric answer is on the table; NONE is the second line, not the first. Comparisons are settled here too, as Fraction comparisons, and handed over as relations. Choosing between 3801755 and 3169068 is not a judgement. Also adds a Result-Ignore detector: long numbers in a synthesised answer that no tool computed. A measurement rather than a rewrite -- it turns a silent wrong answer into a count we can watch. The selection path is immune by construction. And groups thousands in every fact the model reads back. Llama-3 chunks a digit run into threes LEFT to right, so 13910400 tokenises as [139][104][00] and comes back as "139,104,000" -- the model faithfully copying a value it mis-segmented. Three test doubles in here were wrong before the code was: hash() is randomised per process, a 512-bucket sha1 still collided, and a four-digit guard flagged the year 1989. The vocabulary is now built exactly and collisions are impossible. 22 new tests, 310 pass. The battery number is NOT claimed: three runs of near identical configurations gave 4/6, 1/6 and 2/6, which is the noise floor at n=6 rather than an effect. A pooled measurement is running.
The regression check earned itself immediately. Three groups went backwards --
long_addition 2/2 to 0/2, long_multiplication and powers 1/2 to 0/2 -- and the
answers name the cause:
Q: what is 17,433,193,048 * 3,565?
A: "Calculator_verification: 42.5."
42.5 is (17/100)*250, the example in my own schema description. Another run
answered the same question with "Second in 23 week: 13910400" -- the example from
the convert field. The model was not reasoning badly. It was copying the only
concrete numbers it had been shown, which is exactly what a small model does with
a worked example sitting in its context.
Every example in every prompt is now a placeholder: "using the numbers from THIS
question and no others", "<quantity> <unit> to <unit>". A test walks the system
prompt, the answer prompt and every schema description and fails on any bare
number of three digits or more, allowing only numbers bound to a noun about the
output ("under 700 characters"), which cannot be mistaken for a question's data.
This was invisible before selection existed: a leaked example used to be one
wrong number among the model's own, and now it is a labelled candidate answer
competing on equal terms. Making the failure legible is what surfaced it.
It lost. Full battery, same questions, same model:
overall computed-but-unused
synthesis (0.6.0) 9/20 2
selection 7/20 7
The second column is the indictment. Selection exists to remove exactly that
failure -- a value computed exactly and then not used -- and it more than tripled
it. The guarantee it does provide is real and holds: the answer can no longer be
a number no tool computed, because the model returns a letter and the value comes
from our record. But it trades writing the wrong number for choosing the wrong
fact, and on this model that trade is bad.
On the narrow case it was built for it does win -- unit questions, pooled over 18
questions from three seeds, 6/18 against 0/18 with the field off, 95% CI
[+11.6, +55.1]. That is a real effect on a group where the model is otherwise
helpless. It is not enough to pay for the rest.
So `select_answer` defaults to False and the machinery stays, tested, behind a
flag and a `--select` switch on the bench. The research says the ceiling here is
the model's grounding -- 50.79% for llama3.2:3b against 88.19% for qwen3:4b-
instruct-2507 -- so this is worth turning on when the model changes, and worth
leaving off until it does.
Recording a design that did not work, with the number that says so, is worth more
than turning it on and hoping. The measurements are in docs/claims/battery_*.json
and the harness is one flag away from re-running them.
Same 40 generated questions, same code, same day:
seed 20260811 seed 555 pooled
qwen3:4b-2507 18/20 90% 15/20 75% 33/40 82.5%
llama3.2:3b 10/20 50% 6/20 30% 16/40 40.0%
+42.5 points, 95% CI [+23.3, +61.7], replicated on an independent battery, and
no group went backwards. Unit questions -- the group this repository has spent
the most effort on, through labels, gates, compression and exact relations --
went 0/2 to 2/2 by changing the model. It also uses FEWER steps: 5.5 against 7.2.
That is larger than every design change here put together, and it retires the
open question from the selection work: the research said the ceiling was the
model's grounding rather than the prompt, and this is that claim measured on our
own harness rather than on a leaderboard. Selection was tried on qwen3 too and
still adds nothing, which is the same answer from the other direction.
Both directions are pinned, on two batteries, because the comparison is only
worth anything if it cannot quietly become a comparison of two different things.
One pinned claim moved rather than broke: unaided accuracy on the default battery
was 2 of 20 and is now 5. The battery was re-run after concrete examples were
removed from the prompts, so dropping copyable numbers helped the arm WITHOUT
arithmetic as well. The old figure measured a different prompt, so the new one
is recorded with the reason instead of the tolerance being widened to hide it.
Adds --model to the bench, so any future comparison is one flag rather than a
rebuild.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started as an attempt to fix the last known failure — the model computing a value exactly and then answering with a different number. It ends somewhere else.
The headline
Same 40 generated arithmetic questions, same code, same day:
qwen3:4b-instruct-2507llama3.2:3b+42.5 points, 95% CI [+23.3, +61.7], replicated on an independent battery, no group backwards, and in fewer steps (5.5 against 7.2).
Unit questions — the group this repository has spent the most effort on, through labels, gates, compression and exact relations — went 0/2 → 2/2 by changing the model.
The design that lost, and is kept off
Answer-by-selection: the model picks a letter, the value comes from our record, so it cannot state a number no tool computed. The guarantee is real and holds. It still lost:
The second column is the indictment — it exists to remove that failure and tripled it, trading writing the wrong number for choosing the wrong fact. It wins on the narrow case it was built for (unit questions, 6/18 against 0/18 pooled over three seeds, CI [+11.6, +55.1]) and that is not enough to pay for the rest.
So
select_answerdefaults to False, with the machinery, 23 tests and a--selectswitch kept. Tried on qwen3 as well: still adds nothing.The real bug it uncovered
The regression check earned itself immediately:
42.5 is
(17/100)*250— the example in my own schema description. Another run answered with "Second in 23 week: 13910400", the example from the convert field. The model was copying the only concrete numbers it had been shown. Every prompt example is now a placeholder, and a test fails on any bare three-digit number in the system prompt, the answer prompt or any schema description.This was invisible before: a leaked example used to be one wrong number among many, and selection made it a labelled candidate competing on equal terms.
Also
13910400tokenises as[139][104][00]and returns as "139,104,000" — faithful copying of a mis-segmented value.--modelon the bench, so the next comparison is one flag.hash()is randomised per process, a 512-bucket sha1 still collided, and a four-digit guard flagged the year 1989.311 tests pass; all pinned numbers check.