Skip to content

API: Stream partial tool_calls argument deltas for qwen3_coder formats #462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tengotengo wants to merge 1 commit into
base: main
Choose a base branch
from
Open

API: Stream partial tool_calls argument deltas for qwen3_coder formats #462

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/10.-Tool-Calling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Below are the currently recognized tool call formats. Reasoning tags should be s

| tool_format | Aliases | Usual reasoning tokens | Model types
|----------------|----------------------------------------------|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| qwen3_coder | qwen3_5<br> step3_5<br> step3_7 | `<think>` `</think>` | Qwen3-Coder<br> Qwen3-Next<br> Qwen3.5 / Qwen3.6<br> Step-3.5 / Step-3.7
| qwen3_coder ⁵ | qwen3_5<br> step3_5<br> step3_7 | `<think>` `</think>` | Qwen3-Coder<br> Qwen3-Next<br> Qwen3.5 / Qwen3.6<br> Step-3.5 / Step-3.7
| minimax_m2 | | `<think>` `</think>` | Minimax-M2<br> Minimax-M2.1<br> Minimax-M2.5
| glm4_5 | glm4_6<br> glm4_7<br> laguna<br> poolside_v1 | `<think>` `</think>` | GLM4.5<br> GLM4.6<br> GLM4.7<br> Laguna XS/S (Poolside) GLM5.2, GLM5.3
| deepseek_v4 ⁴ | dsv4 | `<think>` `</think>` | DeepSeek-V4<br> DeepSeek-V4-Flash
Expand Down Expand Up @@ -59,3 +59,11 @@ template variables, settable per request via the top-level fields or `chat_templ
The template requires parallel tool results in the same order as the corresponding tool calls;
TabbyAPI sorts tool messages by `tool_call_id` before templating, so clients may send them in
any order.

**⁵** For the qwen3_coder family, streaming responses report tool calls incrementally: instead of a
single delta with the finished call at end of stream, the response carries `tool_calls` deltas with
partial `arguments` fragments (same `index`, `id` and `name` as before), which clients concatenate
as usual. The concatenated fragments are identical to what the end-of-stream parser produces, so
assembling deltas and reading a non-streaming response give the same result. A tool call that
generation cuts off in the middle is the one exception: its fragments stop early, while the
end-of-stream parser would drop the incomplete call entirely.
29 changes: 26 additions & 3 deletions endpoints/OAI/utils/chat_completion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@
from endpoints.OAI.utils.tools import (
get_toolcall_tags,
parse_toolcalls,
supports_delta_streaming,
)
from endpoints.OAI.utils.toolcall_stream import QwenToolCallDeltaStreamer
from endpoints.OAI.utils.common_ import aggregate_usage_stats, get_usage_stats


Expand Down Expand Up @@ -634,11 +636,13 @@ async def _chat_stream_collector(
# reasoning and tool format settings
tool_format = "harmony"
use_think = False
use_tool = False
parser = HarmonyStreamParser()
elif mc.muse_glimmer:
# Same for Muse Glimmer, with recipients in place of channels
tool_format = "muse_glimmer"
use_think = False
use_tool = False
parser = GlimmerStreamParser()
else:
tool_format = mc.tool_format
Expand All @@ -656,6 +660,14 @@ async def _chat_stream_collector(
tool_calls_in_reasoning=mc.tool_calls_in_reasoning,
)

# Incremental tool_calls deltas: for formats that support it, emit
# OpenAI-style tool call fragments as they are generated instead of one
# complete delta at end of stream. The end-of-stream parse stays as the
# fallback whenever the streamer produced nothing.
tool_streamer = None
if streaming_mode and use_tool and supports_delta_streaming(tool_format):
tool_streamer = QwenToolCallDeltaStreamer()

# Reasoning budget: when the reasoning phase exceeds the budget, force
# end-of-reasoning tokens into the output stream so the model answers
# with what it has. The injected text arrives as regular output, so the
Expand Down Expand Up @@ -704,6 +716,7 @@ async def _chat_stream_collector(

delta_reasoning = ""
delta_content = ""
tool_deltas: list = []
for channel, sub in events:
if channel == REASONING:
delta_reasoning += sub
Expand All @@ -713,6 +726,8 @@ async def _chat_stream_collector(
full_content += sub
else:
full_tool += sub
if tool_streamer is not None:
tool_deltas.extend(tool_streamer.feed(sub))

# Count reasoning tokens and force the end of the reasoning phase
# when the budget is exhausted. Attribution is approximate: a
Expand Down Expand Up @@ -757,10 +772,18 @@ async def _chat_stream_collector(
generation["delta_reasoning_content"] = delta_reasoning
generation["delta_content"] = delta_content
generation["delta_tool_calls"] = ""
if tool_deltas:
await gen_queue.put({"index": task_idx, "delta_tool_calls": tool_deltas})
if finish_reason and full_tool:
generation["delta_tool_calls"] = _parse_tool_calls(
full_tool, tool_format, request_id
)
if tool_streamer is not None and tool_streamer.emitted:
# Fragments were streamed; the client assembles the
# calls itself, so only cross-check against the
# authoritative parse and close the finish reason.
tool_streamer.verify(full_tool, request_id)
else:
generation["delta_tool_calls"] = _parse_tool_calls(
full_tool, tool_format, request_id
)
generation["finish_reason"] = "tool_calls"
await gen_queue.put(generation)

Expand Down
Loading