Repository navigation
Description
When benchmarking stage routing on TB-Lite with a frontier strong tier and a local small model weak tier, I reached for capable_first and left the threshold at its default.
The runs came back perfectly healthy: every request 200, no errors. But nothing ever went to the weak tier. Not a single turn.
After digging into the scorer I think this is a property of the shipped defaults rather than anything specific to our setup, so it seemed worth writing up in case others hit the same thing.
The short version
picker: capable_first (switchyard/lib/profiles/stage_router_config.py:76) and confidence_threshold: 0.5 (:81) are both defaults, and in combination the scorer can never select the efficient tier. You get there by omitting both knobs, which we suspect is the common path.
Where it comes from
In crates/libsy/src/algorithms/util/stage.rs (SIGNAL_UNIT=0.10, SCORE_GAIN=5.0, HARD_SEVERITY=0.7):
let raw = SIGNAL_UNIT
* (d.severity / HARD_SEVERITY + d.spinning + d.exploring - d.production_intensity);
let score = (SCORE_GAIN * raw).tanh();
// confidence = score.abs(); gate is `if scored.confidence >= confidence_threshold` (stage.rs:319)Each maxed signal contributes one unit, so with k agreeing signals the confidence lands on a discrete ladder:
confidence = tanh(SCORE_GAIN × SIGNAL_UNIT × k) = tanh(0.5k)
k=1 → 0.4621 k=2 → 0.7616 k=3 → 0.9051
0.5 happens to sit in the gap between the k=1 and k=2 rungs.
That's fine in the escalation direction, because three terms push toward capable (severity, spinning, exploring) and any two of them reach k=2 = 0.7616. It's the de-escalation direction where it gets stuck: only one term is negative — production_intensity — and since it comes from ratio(...) it's bounded to [0, 1]. There's no second negative signal to stack with it, so the efficient side can never climb past the first rung:
max |confidence_negative| = |tanh(5.0 × 0.10 × (−1))| = tanh(0.5) = 0.4621171573
And capable_first only leaves its default tier when the score is confidently negative. So the 3-vs-1 asymmetry between the two directions is really what's going on here — which is also why efficient_first@0.5 behaves exactly as documented and only capable_first@0.5 gets stuck.
There's a near-miss that makes it easy to miss: before the squash, one maxed signal is exactly 5.0 × 0.10 = 0.5, which would satisfy the inclusive >= gate. The tanh compression pulls it down to 0.4621 — about 7.6% short.
What we measured
Two seeds of TB-Lite (20 tasks each, Hermes agent) at the default capable_first@0.5:
| seed | tasks | weak-tier turns | weak tokens |
|---|---|---|---|
| 1 | 20 | 0 / 217 | 0.0% |
| 2 | 20 | 0 / 222 | 0.0% |
439 consecutive turns, none routed to the efficient tier, with zero upstream errors in either run.
Changing only the threshold to 0.45 (same profile, same tasks) -- offload becomes 13.9% (token-weighted), and efficient_first@0.45 reaches 58.0%. So the tier plumbing is working fine; it's specifically the capable_first + >0.4621 combination that can't fire.
Worth noting the hard de-escalate shortcut (tests_passed && recent write/edit >= 1) didn't rescue it either — that path bypasses the scorer entirely, but across those 439 turns it never fired, so what you observe is a complete no-op rather than just a dead scorer branch.
The tricky part for users
The cliff sits at tanh(0.5) = 0.46211715726:
| threshold | 0.40 | 0.45 | 0.46 | 0.4621 | 0.4622 | 0.47 | 0.48 | 0.5 |
|---|---|---|---|---|---|---|---|---|
| scorer offload fires? | yes | yes | yes | yes | no | no | no | no |
Someone wondering "why isn't anything offloading?" would very reasonably nudge 0.5 → 0.48 → 0.47, still get nothing, and conclude the feature doesn't work. Nothing in the logs or in routing_decisions.stage_router separates "the signals were weak this turn" from "this branch cannot fire at all" -- every turn just shows up as no_signal / fall_open.
The docs get very close to this
docs/routing_algorithms/stage_router_routing.md:109 already spells out the key fact — one full signal scores ~0.46, "just under 0.5" — and draws exactly the right conclusion for escalation: a decisive escalation needs a strong signal plus corroboration. That reasoning is sound in the positive direction.
The subtlety is that the same sentence doesn't carry over to de-escalation, because there's only the one negative signal available to corroborate with. A few nearby places read a little differently once that's taken into account:
- :86 — "only drops to the efficient tier when the signals say 'efficient' and clear the threshold" — accurate in general, but at the recommended
0.5that path can't be taken. - :90 — "Raise it and only strong, decisive signals drop a turn to efficient" — reads as a continuous dial; above
0.4621the rate isn't lower so much as fixed at zero. - :101 — "Set
0.5explicitly. It's the recommended starting point." - :175 — "a
0.5threshold takes ~1.5 signals of agreement — a policy that escalates ~20% of tasks maps roughly toconfidence_threshold: 0.5withcapable_first" — withcapable_firstthe scorer-driven action is de-escalation, so we weren't sure how to square this one with the above. Quite possibly it was calibrated with a classifier attached, in which case it may just need a note.
Possible fixes
Any one of these would have saved us the detour:
- Lower the default
confidence_thresholdto0.45. - Warn at config load when
picker == capable_first && confidence_threshold > tanh(0.5). - Add a second negative signal so the efficient direction can reach the k=2 rung.
There's already a test pinning the constant — one_signal_scores_below_half (stage.rs:570) asserts 0.4621 < 0.5 — so the value is known and deliberate; it just doesn't currently flag the interaction with the default picker.
Happy to put up a PR for whichever direction you prefer, and happy to run any repro that would be useful. Thanks.
Activity
[-][Bug] stage_router: default capable_first + confidence_threshold=0.5 makes scorer offload mathematically unreachable[/-][+]stage_router: capable_first can't offload at the default confidence_threshold=0.5[/+][-]stage_router: capable_first can't offload at the default confidence_threshold=0.5[/-][+][Bug] stage_router: capable_first can't offload at the default confidence_threshold=0.5[/+]ayushag-nv commentedon Aug 10, 2026
++ @sabhatinas to look and comment
WUKUNTAI-0211 commentedon Aug 13, 2026
Opened #394 for this. It takes option 2 (warn at config load).
Notes on the other two, since they read differently against current
main:Option 1 is no longer available.
confidence_thresholdis a required TOML field with no default — seedocs/reference/toml_schema.md:178. The Python paths this issue cites (switchyard/lib/profiles/stage_router_config.py,switchyard/lib/processors/reasoning_hint.py) no longer exist;switchyard/lib/was removed when routing moved to Rust. So there is no shipped default to lower, and the "you get there by omitting both knobs" path is gone. The trap is now "0.5is a natural number to type", which is arguably worse, since nothing tells you it is unreachable.Option 3 was left alone deliberately. Adding a second negative signal changes routing behaviour and needs recalibration, which did not seem to belong in a bug fix. Happy to re-scope #394 if that is the direction you prefer.
What #394 does:
build_routewarns whencapable_firstis paired with a threshold above that ceiling, naming the ceiling and the reason. No routing behaviour changes. Your analysis of the negative side held up exactly —ratioboundsproduction_intensityto1.0, so the efficient direction has one signal unit and nothing to corroborate with. A test derives the ceiling from the dimension set rather than asserting the literal, so it will fail if a second negative signal is ever added.The docs lines you flagged are untouched — #288 is already rewriting that page and moving the starting recommendation to
0.3. A follow-up can point it atmax_efficient_confidence()once that lands.grahamking commentedon Aug 25, 2026
That seems important to fix. Thanks!
joshuacox commentedon Oct 16, 2026
This looks fixed for the reported case on current
main, so it may be closable.#984 (merged 2026-10-13) went in the direction of option 3: the stage scorer now adds two efficient-side terms, production weighted 1.5x plus a 1.2-unit clean-production term (writing or editing with no errored tool result in the recent window). Clean production now scores about -0.874, so it crosses a 0.5 threshold instead of stalling at the old -0.4621 ceiling. In the PR's offline replay of 1,221 decisions at
capable_firstwithconfidence_threshold = 0.5, the capable share moved from 100% to 74.9%, with no efficient turn carrying an error in its window.The other two options are accounted for as well. Option 1 is moot now:
confidence_thresholdis a required TOML field with no shipped default to lower. Option 2 (warn at config load) was implemented twice, in #394 and #711; #394 was closed for inactivity, and #711 was closed by its author as superseded by #984.A threshold far above the new ceiling (about 0.874) can still be unreachable, so if you would rather keep this open for a warning at that end, that is a narrower follow-up. Happy to be corrected if the
capable_firstat 0.5 case still misbehaves.ayushag-nv commentedon Oct 30, 2026
This issue is fixed on latest main as of today. Closing this..