Skip to content

Route casual threads Low and let a thread's Balanced tier decay - #732

Merged
rockfordlhotka merged 3 commits into
mainfrom
issue-729/apply-the-active-thread-tier-floor-only-when
Oct 22, 2026
Merged

rockfordlhotka merged 3 commits into
mainfrom
issue-729/apply-the-active-thread-tier-floor-only-when

Conversation

@rockfordlhotka

Copy link
Copy Markdown
Member

Summary

  • Removed the active-thread floor. A follow-up on a 3+-turn thread no longer goes to Balanced just because the thread exists. It inherits Balanced (the existing inherited-tier rule) only when a recent turn earned it, so small talk stays Low past turn 3.
  • IntrinsicTier is captured before the thread-state rules. The content floors (research question, balanced-floor keyword) still count toward it. The active-subagent floor and inheritance no longer do, so a Balanced thread decays after MaxTurnsPerSession turns that neither earn it nor do work.
  • Turns that do real work record Balanced. At the end of a user turn, UserMessageHandler.EarnedTier raises the turn's history entry to Balanced when its tool ledger has a call that counts as thread work (ToolSideEffects.IsThreadWork). That is any call except memory, working-memory, skill and tool-guide reads and the agent's own bookkeeping. Mid-turn escalation only fires on Low-routed runs, so without this a Balanced turn that edited a deck recorded Low and the thread decayed mid-task.
  • The active-subagent-floor is unchanged. TierRoutingRules.ActiveThreadFloor stays so older routing-log entries remain readable.

Routing-log analysis (prod, 10/10–10/21)

  • active-thread-floor decided 59 of 131 rule-logged user turns, more than any other rule. About 8 of those 59 were casual: the propane message, the health updates, the evening plans, and the trivia probes.
  • Raw tool use is not a usable signal. All 274 Low user turns with a toolCallCount made at least one call, because every turn opens with search_memory / search_working_memory / get_from_working_memory. That's why IsThreadWork excludes context reads.
  • The Agent replies to injected memory instead of recent prompt on short user messages #383 defenses that don't depend on tier stay in place: the short-message BM25 gate (≤30 chars) and the memory-summary reply guard (≤120 chars, on by default). The Low model is now gpt-5.6-luna.

Test plan

  • dotnet test RockBot.slnx: all pass
  • New: a casual session stays Low past turn 3; a Balanced turn's follow-ups inherit Balanced and then decay to Low; the active-subagent floor is not part of IntrinsicTier; IsThreadWork classification; EarnedTier
  • Live: a casual exchange plus "I'll find out soon" logs rule=score-band tier=Low and the reply continues the thread (Agent replies to injected memory instead of recent prompt on short user messages #383 shape)
  • Live: a work request, then short follow-ups, inherit Balanced and decay after three trivial turns

Closes #729

🤖 Generated with Claude Code

The active-thread floor raised every non-ack follow-up on a 3+-turn thread to
Balanced, and ran before IntrinsicTier was captured, so floored turns recorded
Balanced and a busy thread never decayed. Small talk went Balanced from turn 4.

- Remove the active-thread floor. A thread now stays Balanced through the
  inherited tier only while a recent turn earned it.
- Capture IntrinsicTier after the content floors (research question,
  balanced-floor keyword) but before the thread-state rules (active-subagent
  floor, inherited tier).
- At the end of a user turn, raise its history entry to Balanced when it did
  work on the user's behalf (ToolSideEffects.IsThreadWork: anything but memory,
  working-memory, skill and tool-guide reads and bookkeeping). Mid-turn
  escalation only fires on Low-routed runs, so without this a Balanced turn
  that edited a deck recorded Low and the thread decayed mid-task.
- Keep TierRoutingRules.ActiveThreadFloor for older routing-log entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Worker delegation is excluded from earned-tier tracking, allowing worker-backed tasks to lose Balanced inheritance despite continuing substantive work.

1 open issue
What changed in this PR

This PR updates RockBot’s thread-aware model routing so casual conversations stay Low and Balanced inheritance expires when recent turns neither earn a higher tier nor perform substantive work.

Changes:

  • Removes the active-thread floor and captures intrinsic tier before thread-state rules.
  • Records substantive tool work as earning at least Balanced for subsequent turns.
  • Adds routing, decay, and tool-classification tests and updates routing documentation.
File Description
tests/​RockBot.Llm.Tests/​ThreadAwareTierRoutingTests.cs Tests casual routing, inheritance, decay, and content floors.
tests/​RockBot.Llm.Tests/​KeywordTierSelectorTests.cs Updates expectations for earned-tier inheritance.
tests/​RockBot.Host.Tests/​ToolSideEffectsTests.cs Tests thread-work classification.
tests/​RockBot.Host.Tests/​SessionTierHistoryTests.cs Tests Balanced-to-Low decay.
tests/​RockBot.Agent.Tests/​TurnEarnedTierTests.cs Adds earned-tier calculation tests.
src/​RockBot.Llm/​KeywordTierSelector.cs Removes the active-thread floor and repositions intrinsic-tier capture.
src/​RockBot.Host/​ToolSideEffects.cs Adds substantive thread-work classification.
src/​RockBot.Host/​SessionTierHistory.cs Documents work-based history promotion.
src/​RockBot.Host.Abstractions/​ShortMessageHeuristics.cs Documents removal of the active-thread floor.
src/​RockBot.Host.Abstractions/​ILlmTierSelector.cs Clarifies routing contracts and retains the legacy rule identifier.
src/​RockBot.Agent/​UserMessageHandler.cs Raises recorded tiers based on completed-turn work.

Review effort: Balanced

Comment thread src/RockBot.Host/ToolSideEffects.cs Outdated
{
if (string.IsNullOrWhiteSpace(call.Name)) return false;
var name = string.Join('_', SplitWords(call.Name));
return !IsBookkeeping(name) && !ContextReadTools.Contains(name);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3fa3b4b. IsThreadWork now returns true for spawn_workers (snake_case and PascalCase). Its side-effect classification is unchanged; IsBookkeeping and IsSideEffecting still exclude it. Added regression coverage: ToolSideEffectsTests.ThreadWork rows for both name forms, and TurnEarnedTierTests.WorkerBatchOnly_EarnsBalanced, which also asserts that spawn_workers is still not side-effecting.

rockfordlhotka and others added 2 commits October 22, 2026 17:44
spawn_workers is bookkeeping for side effects, and its workers' calls run in
their own loops, so they never reach the turn's ledger. A turn whose only work
was a worker batch recorded Low and the thread could decay mid-task. Its
side-effect classification is unchanged.

Also replace two test messages with neutral small talk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live testing showed "what time is it in Amsterdam?" calling convert_time and
keeping the next three turns on Balanced. convert_time, get_agent_name and
ping answer a one-line question without touching anything the user is working
on, so a turn that only uses them no longer raises the session's tier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rockfordlhotka
rockfordlhotka merged commit 63f2148 into main Oct 22, 2026
2 checks passed
@rockfordlhotka
rockfordlhotka deleted the issue-729/apply-the-active-thread-tier-floor-only-when branch October 22, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Apply the active-thread tier floor only when the thread has earned Balanced

2 participants