UN-4137 [FIX] Stream LLM completions under the hood so long generations complete instead of timing out

[praveen-formido]

What

Jira: UN-4137 · Slack thread: https://zipstack.slack.com/archives/C08PX3XALEA/p1789985766495129?thread_ts=1789726991.952109&cid=C08PX3XALEA

• LLM.complete() now streams under the hood (stream=True), collects the chunks and rebuilds the full response with litellm.stream_chunk_builder. Callers are unchanged: same dict, same raw response (finish_reason, usage incl. cache tokens, provider request-id headers).
• New collect_with_retry helper in retry_utils: a failure before the first content chunk is retried exactly as a non-streaming call would be; a failure after content started is raised immediately so a long generation is never replayed. Chunks from a failed attempt are discarded, not duplicated.
• New Enable Streaming checkbox on all 14 LLM adapter forms (enable_streaming), default on. Adapters stored before the field existed also stream. Untick it only for an endpoint that cannot stream. The flag is read from the raw adapter metadata and never reaches litellm.
• Anthropic adapter form: the Timeout description now says the value bounds the wait for the first token and any silence between chunks, since that is what it means once replies stream.

Why

A Prompt Studio prompt on an Excel document (claude-sonnet-4-6) "ran for an hour" and failed, while the same prompt completes in ~16 minutes in the Anthropic console. QA reproduced it with adapter timeouts of both 900 s and 1800 s.

Cloud Logging (staging + production) shows the mechanism:

Retry 1/3 for claude-sonnet-4-6 (anthropic): litellm.Timeout: AnthropicException - litellm.Timeout: Connection timed out after 900.0 seconds.
...
Retry 3/3 for sonnet 4.6 (anthropic): litellm.Timeout: ... Connection timed out after 1800.0 seconds.
Handler _handle_structure_pipeline failed after 7214.64s

• The non-streaming Anthropic endpoint sends no bytes until the last token, so the socket is silent for the whole generation. With 900 s the read timeout fires mid-generation; with 1800 s the response still never arrives on a 16-minute generation (Anthropic's own SDK refuses non-streaming requests that may exceed 10 minutes for exactly this reason; litellm has no such guard).
• is_retryable_litellm_error treats Timeout as retryable and max_retries defaults to 3, so the identical 15–30 minute request was replayed up to 4× and then killed by the Celery time limit, with nothing to show for it.

The console works because it streams. This PR makes the platform do the same.

How it was verified

• Local stack on this branch, same project export, same adapter settings that failed in staging (Sonnet 4.6, timeout 900 s, 3 retries):

  Run	Output cap	LLM call duration	Retries	Result
  via API	64k	720 s (64,000 completion tokens)	0	completed; output truncated at the cap
  via UI	128k	756 s (67,260 completion tokens)	0	completed; full 353-row table, verified against the source xlsx

• Backend test-connection against real providers on the branch: Anthropic and an OpenAI-compatible (custom_openai/gpt-5) adapter both pass with stream=True confirmed via a litellm spy. Bedrock could not be exercised locally (expired bearer token on the adapter).

• SDK1 unit tests: 33 new tests (test_anthropic_stream_complete.py, test_llm_schema_streaming_flag.py, utils/test_collect_with_retry.py) drive litellm's real mock-streaming path so stream_chunk_builder runs for real. Full SDK1 suite: 574 passed; the 4 pre-existing async test_llm_compat failures need pytest-asyncio in the venv and fail identically on main.

Staging regression checklist (all adapters, since streaming is now the default)

• Anthropic (the reported prompt: expect completion in ~13–16 min, no Retry line in executor logs)
• OpenAI, Azure OpenAI
• Bedrock (moves to converse-stream; includes inference-profile ARN routing)
• Vertex AI, Gemini
• OpenAI-compatible family: custom OpenAI, NVIDIA, MiniMax, OpenRouter, Azure AI Foundry, Ollama, Anyscale. A failure here shows as a 400 on stream_options or a zero-token usage line; the fix is to untick Enable Streaming on that adapter.
• Prompt caching metrics still reported (cache read/creation tokens come through the stream)
• Extended thinking on Anthropic (thinking blocks are reassembled by the chunk builder)

Notes for reviewers

• Retry semantics narrow by design: after the first content chunk, a mid-stream drop surfaces immediately instead of replaying the whole request. Before the first content chunk (Anthropic's message_start, 429/529/5xx/connection errors), behaviour is unchanged.
• The adapter Timeout now bounds the gap between chunks rather than the whole reply; the 900 s default is generous for that.

🤖 Generated with Claude Code

[greptile-apps]

via Greptile

Confidence Score: 5/5

The PR appears safe to merge; no actionable regression remains in the changes since the previous review.

Summary

This PR changes synchronous LLM completions to stream internally and reconstruct the existing consumer-facing response, preventing long generations from failing solely because the connection remains silent.

• Adds pre-content-only retry semantics while preventing replay after generated content begins.
• Adds an enabled-by-default per-adapter streaming switch across all LLM schemas.
• Preserves the deterministic non-streaming mock path used by the end-to-end test rig.
• Adds coverage for response reconstruction, retries, opt-out behavior, schemas, and mocked completions.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[LLM.complete] --> B[Validate completion arguments]
    B --> C{Mock response present?}
    C -->|Yes| D[Non-streaming completion with retry]
    C -->|No| E{Adapter streaming enabled?}
    E -->|No| D
    E -->|Yes| F[Create and consume stream]
    F --> G{Failure occurred?}
    G -->|Before content| H{Retryable and attempts remain?}
    H -->|Yes| F
    H -->|No| I[Raise error]
    G -->|After content| I
    G -->|No| J[Rebuild full LiteLLM response]
    D --> K[Return existing response contract]
    J --> K

Loading

_{Reviews (3) · Last reviewed commit: "UN-4137 [FIX] Keep mocked completions on..."}

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
✅	e2e-api-deployment	e2e	3	0	0	0	20.8
✅	e2e-coowners	e2e	1	0	0	0	1.8
✅	e2e-etl	e2e	1	0	0	0	6.8
✅	e2e-login	e2e	2	0	0	0	1.3
✅	e2e-prompt-studio	e2e	1	0	0	0	9.9
✅	e2e-smoke	e2e	2	0	0	0	1.5
✅	e2e-workflow	e2e	1	0	0	0	20.4
❌	frontend	unit	0	1	0	0	0.0
✅	integration-backend	integration	598	0	0	26	47.8
✅	integration-connectors	integration	1	0	0	7	8.1
❌	integration-workers	integration	159	5	0	1	47.7
❌	ui	e2e	0	1	0	0	0.0
✅	unit-backend	unit	1283	0	0	1	46.3
✅	unit-connectors	unit	63	0	0	0	10.1
✅	unit-core	unit	137	0	0	0	2.2
✅	unit-platform-service	unit	15	0	0	0	2.7
✅	unit-rig	unit	120	0	0	0	4.7
✅	unit-runner	unit	5	0	0	0	2.9
✅	unit-sdk1	unit	580	0	0	0	32.3
✅	unit-workers	unit	1362	0	0	1	125.2
	TOTAL		4334	7	0	36	392.4

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• platform-key-whoami — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment