API: Stream partial tool_calls argument deltas for qwen3_coder formats

[tengotengo]

What

Streaming responses with a qwen3_coder-family format (qwen3_coder, qwen3_5, step3_5,
step3_7) now report tool calls while the model generates them: tool_calls deltas with
partial arguments fragments, instead of one complete delta after generation finishes.
Closes #460.

How

QwenToolCallDeltaStreamer consumes the TOOL-channel text that TagStreamParser already produces
and turns the pseudo-XML structure into OAI deltas: an open frame (index, id, type, name,
arguments: "{"), argument fragments as parameters close, "}" when the function closes.

• String values stream live as soon as they provably cannot parse as a JSON literal, so
  coerce_param_value() cannot reinterpret them later. Values it may convert (objects, arrays,
  numbers, booleans, null, quoted strings) are emitted whole when their parameter closes.
• Malformed tag sequences are read the way the end-of-stream regexes read them: stray close tags
  and parameters outside a function block are dropped, a nested function open belongs to the outer
  call. The streamer never raises into the collector.
• The authoritative end-of-stream parse stays the fallback whenever the streamer produced nothing.
• verify() compares the streamed assembly against the end-of-stream parse at the end of every
  request. It is log-only, because fragments cannot be retracted.

Only the qwen3_coder family is wired up; harmony, muse_glimmer and the rest keep the current path.

Guarantee

For well-formed calls the concatenated arguments fragments per index are byte-identical to what
the end-of-stream parser produces, under random, one-character and whole-text chunkings — a client
that assembles deltas sees exactly what a non-streaming request returns.

Three inputs diverge on purpose, and are documented in the module docstring and the docs page:

1. A call generation was cut off inside. The regex needs a closing tag and drops the whole
   call, while the client already holds the partial arguments; verify() names that cause
   explicitly. Previously the same request returned finish_reason: "tool_calls" with an empty
   tool_calls list.
2. Duplicate parameter keys. Both are streamed (they parse to the same object); the parser
   keeps the last one.
3. A value containing a literal parameter close tag. Both sides end the value at the first
   occurrence but read the remainder differently.

Testing

• Unit: python -m unittest tests.test_toolcall_stream tests.test_tool_delta_streaming (61 tests).
  The whole repo suite — 182 tests — passes on Python 3.10 and 3.11.
• ruff check and ruff format --diff clean.
• Live: python tests/req_tool_delta.py runs 16 contract checks against a running server
  (Qwen3.8-Flash-Next, ExLlamaV3): frame shape and ordering, streamed vs non-streaming equality,
  tool_choice none/required/named, reasoning-then-tool and content-then-tool ordering, client
  disconnect mid-call, n>1, logprobs, non-streaming regression, truncated calls. Normal traffic
  produces no verify() errors in the server log.
• The tests were checked against deliberate breakage of each promised behaviour (24 mutations:
  21 caught, 3 provably equivalent), so the suite fails when the guarantee breaks. One caveat
  found that way: fixtures must take their wrapper tags from the format itself, otherwise the text
  never reaches the tool channel and every comparison silently passes on empty input.

Also included

One-line fix in tests/_common.py: tool_calls is a list of per-choice dicts, so the old
membership test was never true and the entry of a call was recreated on every delta, keeping only
the last argument fragment. Invisible while calls arrived as a single delta; visible as soon as
arguments are streamed in fragments.

Relationship to #378

#378 adds an XML tool-call parsing path (it predates the current toolcall_formats package) and
touches the same two files for that purpose. This change assumes the existing format parsers and
only changes how their input is streamed, so the two would need a rebase but address different
problems.

[turboderp]

I see a "what" and a "how", but I'm not entirely clear on the "why". Do clients benefit from streaming tool calls at all?

[fubyo]

I see a "what" and a "how", but I'm not entirely clear on the "why". Do clients benefit from streaming tool calls at all?

I like this.

It's more of a user experience improvement. Imagine the tool call writing a huge file. Now, you don't know what's happening before the tool call is complete, which might take a while.

Additionally, with this you could cancel a long tool call midway, if the user notices that the agent goes in the wrong direction.

[acasar]

+1 for this! This has been one of the things that's been bugging me the most with tabbyAPI compared to llama.cpp.

When an agent is writing a long file, there is no progress visible in the harness, sometimes for minutes. In llama.cpp I can always follow the progress of a long tool call and even abort it if it's not going in the right direction. It would be amazing to have the same in tabbyAPI.