feat: run TAC as N replicas with no shared datastore (supersedes #123) by wenzhu1587 · Pull Request #132 · twilio/twilio-agent-connect-python

wenzhu1587 · 2026-10-06T04:25:08Z

Summary

Makes TAC safe to run as N replicas behind a load balancer with no shared datastore. This PR carries @ryanrouleau's #123 (rebased onto current main, authorship kept, as the first commit) plus fixes for the gaps found reviewing it, listed below. This supersedes #123's diff. If you agree, close #123 in favour of this.

Base design, from #123

Messaging is stateless. Each webhook builds its own session from the payload, config and the participant list it already fetches.
Voice is instance-local. A WebSocket pins a call to one process, and teardown is guaranteed. Adds on_call_ended, the aclose() drain, and instance_public_domain for routing call events to the replica holding the call.

Gaps found in #123, and what this PR does about them

Correctness bugs

Gap	Problem in #123	In this PR
G1	A CO CLOSED arriving while the call is still live on the same instance removed the live session. `on_call_ended` then never fired, and the call's later prompts were dropped.	Fixed. CLOSED for a live call fires `on_conversation_ended` from a snapshot and keeps the session until teardown. After a mid-call close the call keeps the closed conversation's id; CO starts a new conversation for later traffic. Documented.
G2	Media Streams per-call `session_config` stays on the instance that served the TwiML webhook or placed the call; only a token travels with the call. A stream landing on another instance silently fell back to `default_session_config`, or failed when none was set.	Documented; fix deferred. With more than one instance, per-call configs require `instance_public_domain`. TAC now logs a warning naming it when a stream's config was stashed elsewhere. The planned fix, re-deriving inbound configs from TwiML fields forwarded as stream parameters, is recorded in `CLAUDE.md`.
G10	Rebuilt voice sessions got `call_sid=None`: the CLOSED handler read a top-level `channelId` that CO's conversation webhooks don't carry.	Fixed. `call_sid` comes from the VOICE participant's `channelId`, customer first, then agent.

Coverage gaps

Gap	Problem in #123	In this PR
G3	"Conversation Started"/"Ended" analytics from #127 hung on the session lifecycle that #123 removed for messaging. On `main` they over-count Started per instance and usually miss Ended with multiple instances.	Fixed. They come from CO webhooks: `PARTICIPANT_ADDED` for the customer, and CLOSED with `duration_ms = updatedAt − createdAt`. Each channel reports its own channel type, and one participant lookup per CLOSED is shared across channels. The lookup is skipped when analytics is off and no callback is registered. Relay-only and Media Streams voice still report at session start and teardown.
G4	Without per-pod routing, status, AMD and recording events landed on an instance that couldn't find the call's session.	Fixed. New `VoiceChannel.resolve_conversation_session_by_call_sid()` returns the live session if held here, otherwise an identity-only session rebuilt from CO with fresh participants.
G5	The app acting on a live call (`send_response`, reading the transcript) from an instance that doesn't hold it.	Documented as a limitation in `docs/horizontal-scaling.md` ("What needs instance affinity"). Do it from the call's own callbacks, or set `instance_public_domain`.
G6	GPT-Live, which postdates #123, wasn't covered by its session-config fixes.	Fixed. The token handoff and the bounded `ExpiringDict` live in the shared Media Streams base, so both providers get them. GPT-Live's redundant per-token timers are removed.
G11	An INACTIVE webhook reaching another instance doesn't refresh a live call's `memory_mode="once"` cache.	Documented in `docs/horizontal-scaling.md`. The call keeps the memory it fetched at the start.

Behavior changes #123 didn't call out

Gap	Problem in #123	In this PR
G7	Orchestrated ConversationRelay `on_conversation_ended` received a session rebuilt from CO (identity only, no app metadata), and `end_call()` no longer fired it.	Fixed. The instance that held the call keeps the ended session (bounded, 24 h) for CLOSED, which fires with it as on `main`; `end_call()` fires immediately again. Only a CLOSED on another replica gets a rebuilt, identity-only session.
G8	Messaging `session.metadata`, including outbound `options.metadata` and `direction`, no longer carried across turns, even on one instance.	Fixed. Outbound metadata that fits CO's limits is written onto the CO conversation (at creation; `PATCH` merge on reuse). A bounded, best-effort per-instance cache (24 h sliding TTL, 10k entries) keeps `session.metadata` across turns on the same instance, as on `main`. `await session.conversation_metadata()` reads the CO-stored metadata on any instance.
G9	Messaging `memory_mode="once"` raised at construction, which would break existing apps at startup.	Fixed. It logs a `DeprecationWarning` and runs as `"always"`; removal is planned for 3.0. Voice `"once"` is unchanged.
G12	`MessagingChannel.get_agent_address` took a session instead of `main`'s `conversation_id`, and `send_response`'s first parameter was renamed from `conversation_id`.	Fixed. Both keep `main`'s signatures. An override of the old `get_agent_address` keeps receiving ids (TAC's own paths go through a private `_agent_address(session)`), and `send_response` also accepts a session.
G13	Every inbound message whose participant lookup failed was dropped. `main` answered later turns from its in-memory session.	Fixed. A later turn on the same instance is answered from the participants it last reconciled, with a warning. With none cached, or if the webhook names a different sender, the message is dropped, as `main` dropped a first message.

Rebased onto #128 (multi-sender)

An inbound message's agent address is the sender the customer messaged: the webhook recipient, otherwise the sender the conversation's participants hold, otherwise the default.
That address drives reconciliation, replies, the CLOSED rebuild and digital handoff.
feat: multi–phone-number support (multiple senders per channel) #128's inbound tests were rewritten to observe the same behavior through the stateless paths. None were weakened.

Behavior changes reviewers should know

Breaking

None in signatures: get_agent_address(conversation_id) and send_response(conversation_id, ...) keep main's. send_response now also accepts the session in place of the id, which skips a participant lookup.

No longer carried across turns (except via the G8 cache and accessor)

Messaging session.metadata written during a turn. It's kept only on the same instance, best effort, in a cache with a 24 h sliding expiry and 10k entries. Unlike main, which kept it until CLOSED (and leaked it when CLOSED never came), a conversation idle for over 24 h, or pushed out by more than 10k active conversations, starts the next turn with empty session.metadata.

Changed semantics

Orchestrated ConversationRelay on_conversation_ended: unchanged on the instance that held the call (full session, end_call() fires it immediately). New: a CLOSED on another replica, which main ignored, now fires it with a session rebuilt from CO (identity only). With several replicas, read live state in on_call_ended.
Messaging on_conversation_ended: main fired it only if this process still held the session. It now fires for every closed conversation in the configuration with a participant on the channel, on whichever replica receives CLOSED, including after a restart. The session is rebuilt from the participant list (same profile_id and participants; metadata is CO's plus this instance's cache) without session.profile, cached_memory, or the original started_at.
Outbound metadata is stored in Twilio: entries with string values that fit CO's limits (8 keys including direction, keys up to 128 characters of letters, digits, ., _, -, values up to 512 characters) are written to the CO conversation; main kept them in process only. The rest stay local, with a warning. Request-body logs no longer include metadata.
Shutdown waits for live calls: TACFastAPIServer now runs VoiceChannel.aclose() on shutdown. With calls live it waits up to TACServerConfig.shutdown_grace_period (30 s) for them to end, then force-stops the rest within 5 s so their hooks fire; with none live it returns at once. main exited immediately and dropped them silently. Keep grace + 5 s below the platform's termination grace period (Kubernetes defaults to 30 s), or the process is killed mid-drain.
Media Streams per-call session configs: kept until the stream connects for up to 15 min, at most 1,000 per instance (main: Realtime unbounded, GPT-Live 120 s). Past 1,000 calls placed but not yet connected on one instance, the oldest fall back to default_session_config. The stream's TwiML carries a reserved _tac_session_config_token parameter.
Analytics: "Conversation Ended" duration_ms for orchestrated ConversationRelay and messaging is now the CO conversation's length (updatedAt − createdAt, so it includes CO's close timeout), not call or session length. "Started" is counted when CO adds the customer. Dashboards spanning versions will shift.
Telemetry cost: with analytics on (the default), each closed conversation costs one CO participant lookup. TAC_ANALYTICS_DISABLED=true removes it unless an on_conversation_ended handler needs it anyway.

Deprecated

Messaging memory_mode="once", to be removed in 3.0. It logs a DeprecationWarning at channel construction and runs as "always": memory is recalled on every message, with that message's text as the query (instead of once per conversation with an empty query), so each turn adds one recall and one profile-trait fetch. memory_config.fetch_profile_traits=False drops the trait fetch. Test suites that turn warnings into errors will fail at construction until they switch to "always".

Delivery guarantee

on_message_ready, on_conversation_ended and the lifecycle analytics are at-least-once across webhook retries and replicas.

Not yet verified on a live account

These checks block shipping, not review:

V1: an Actions send by address works for SMS and CHAT, and leaves a correctly typed AI_AGENT participant.
V2: what type CO's v2-native capture gives TAC's own number. That decides whether reconciliation is really v1-bridge-only.
V3: after a mid-call CLOSED, the new conversation can be found by CallSid. CO's docs say later traffic starts a new conversation.
V4: a VOICE participant's channelId is the CallSid. G10 depends on it.
V5: conversation metadata is accepted on create, merges on PATCH, and is returned on GET and in conversation webhooks. G8 depends on it, and so does every outbound conversation: TAC now always sends at least direction at creation, so a rejection fails outbound outright (logged with the metadata keys as a possible cause; no retry).

Type of Change

Checklist

Tests added/updated (1293 passing; ruff, mypy strict, and mkdocs build --strict clean)
Documentation updated (docs/horizontal-scaling.md, CLAUDE.md, public docstrings)
Tested E2E (blocked on V1–V5 above)

SDK Parity

This is the Python SDK. If this change affects shared functionality, ensure the TypeScript SDK is updated as well.

Tip: Use the /sync-to-ts-sdk skill in Claude Code to automatically generate and create a TypeScript SDK PR from your changes.

Change is Python-specific (no TypeScript update needed)
TypeScript SDK PR created:

🤖 Generated with Claude Code

Messaging and voice needed different answers. Messaging is now stateless. `MessagingChannel` has no session store at all: each webhook derives a `ConversationSession` from the payload, config, and one `list_participants` call it already needed for the self-message check and reconciliation. Any replica serves any webhook. Previously `CONVERSATION_UPDATED` was filtered on locally-tracked state, so with N instances `on_conversation_ended` usually never fired and the session leaked for the process lifetime; it now rebuilds the session from Conversation Orchestrator and fires on whichever replica Twilio picked. Voice state is legitimately instance-local — a WebSocket pins a call to one process — so the fix is guaranteed teardown instead. `_cleanup_connection` no longer skips cleanup in orchestrated mode waiting for a CLOSED webhook that may land elsewhere (the largest leak); it always releases, and a new `on_call_ended` hook carries the live session, which is the only place the transcript is still reachable. `VoiceChannel.aclose()` drains live calls at shutdown, wired to the FastAPI shutdown event. For a call's out-of-band webhooks (status/AMD/recording, `<Connect action>`), `TACConfig.instance_public_domain` makes TAC mint every callback URL against this process, so they return to the replica holding the socket. No Redis, no database, and no extra API calls in the default configuration — `memory_mode="never"` still costs 2 calls per message, and the budget is pinned by a regression test. Breaking changes: - `MessagingChannel.get_agent_address(session)` was `get_agent_address(conversation_id)`. - `MessagingChannel.send_response` takes `conversation` (a session or an id) as its first parameter, was `conversation_id`. Positional calls are unaffected. - Messaging channels reject `memory_mode="once"` at construction. It caches a recall on a long-lived session, which messaging no longer has; statelessly it would cost the same as `"always"` with worse relevance. Use `"always"`. `"once"` is unchanged on Voice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CO's CONVERSATION_UPDATED payload has no top-level channelId, so rebuilt sessions always got call_sid=None. Read it from the VOICE participant address's channelId instead, and make the fixture match the documented payload. (G10) Includes review fixes: - fix(voice): prefer the customer's VOICE channelId for a rebuilt call_sid Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

A CLOSED webhook for a call still live on this instance popped its session, so on_call_ended never fired and the call's later prompts were dropped. Fire on_conversation_ended now from a snapshot and leave the session for teardown to release. (G1) Updates two pre-existing tests in test_voice_channel.py that asserted the old behavior (session removed on CLOSED while the call was still live), which this change intentionally replaces. Includes review fixes: - fix(voice): document mid-call CLOSED on process_webhook; test end_call and raising callback Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Add VoiceChannel.resolve_conversation_session_by_call_sid: the live session when held locally, otherwise (orchestrated ConversationRelay) an identity-only session rebuilt from Conversation Orchestrator, preferring the active conversation. Lets CallSid-only call events find the conversation without per-pod routing. (G4) Includes review fixes: - fix(voice): document rebuilt-session limits and cost; parse created_at as datetime Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The shared Media Streams base now stashes session configs in a bounded ExpiringDict, so GPT-Live's asyncio sleep task per outbound token was a second TTL for the same entries. Unclaimed entries now expire on the shared default, same as OpenAI Realtime. (G6) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Also tidies a test docstring. Includes review fixes: - docs(voice): note the per-replica conversation id after a mid-call close Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Messaging channels accepted "once" on main, so narrowing the config to never/always and raising in the constructor broke existing apps at startup. Accept it again, emit a DeprecationWarning plus a logged warning, and run it as "always" (memory stays on, with each message's text as the query). Removal is planned for 3.0; Voice is unchanged. (G9) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Adds TAC._list_participants_shared (in-flight + 60 s cache, failures not cached) and analytics.analytics_enabled(), for CO-driven lifecycle analytics. Includes review fixes: - fix: evict failed shared participant lookups via a done-callback Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Conversation Started comes from PARTICIPANT_ADDED for the customer and Conversation Ended from CONVERSATION_UPDATED/CLOSED (duration = CO's updatedAt - createdAt), each channel reporting its own channel type. A CLOSED webhook's participant lookup is shared across channels. (G3) Includes review fixes: - fix(analytics): log unparseable PARTICIPANT_ADDED; document CLOSED reporting Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

With a CO conversation behind the call (orchestrated ConversationRelay), Conversation Started/Ended now come from PARTICIPANT_ADDED and CLOSED, like messaging, with CO's duration; the call's session start and teardown report them only in relay-only and Media Streams modes. The CLOSED rebuild shares the cross-channel participant lookup. (G3) Includes review fixes: - fix(voice): keep G4's CallSid lookup fresh; tidy lifecycle comments - test(analytics): pin voice sharing the CLOSED lookup; fix rebuild log text Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Includes review fixes: - docs: note telemetry lookup cost and at-least-once lifecycle reporting Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s configs Punts the G2 fix. A per-call session_config lives on the instance that served the TwiML or placed the call, so with several instances the stream can land elsewhere and fall back to default_session_config. Both providers now claim the stashed config through one shared helper that logs a warning naming instance_public_domain when the token isn't held locally. docs/deployment.md lists what needs instance affinity and corrects the per-pod guidance (a headless Service isn't reachable from Twilio). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Adds metadata to ConversationRequest/ConversationResponse and to create_conversation / create_or_reuse_conversation, plus get_conversation and patch_conversation_metadata (PATCH merge). (G8) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Reads a conversation's Conversation Orchestrator metadata: answered from what the session already knows, otherwise via a channel-supplied loader whose result is kept for the turn; failures return {} and retry. (G8) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Outbound initiation writes the metadata that fits CO's limits onto the conversation (at creation; PATCH merge on reuse). A bounded, per-instance cache shares each conversation's session.metadata across its turns, as on main, and session.conversation_metadata() reads the CO-stored metadata on any instance, fetching only when it isn't known locally. CLOSED carries the payload metadata and evicts the cache entry. (G8) Includes review fixes: - fix(messaging): keep fetched conversation metadata per turn; fetch on bare CLOSED Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ConversationResponse.metadata is typed dict[str, str] | None, which is strict in Pydantic v2. A conversation whose metadata another tool wrote with a non-string or null value (e.g. {"n": 1, "x": None}) raised ValidationError on every parse: the CLOSED webhook (losing on_conversation_ended and analytics), list_conversations (voice resolves by CallSid with it), update_conversation and get_conversation. The field did not exist before G8, so this was a regression from main. Add a before-validator that keeps only str/str entries and turns a non-dict value into None. ConversationRequest stays strict because TAC builds it itself. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

A CLOSED payload without metadata now uses the cache entry's co_metadata when this instance wrote it, instead of wiring a fetch. session.metadata for on_conversation_ended no longer depends on the replica: payload metadata, then local entries over it, then the rebuilt session's fields. Document that conversation_metadata() answered from local writes does not see later changes made elsewhere, and note the outbound reuse PATCH sets direction. Includes review fixes: - docs(messaging): don't overclaim CLOSED metadata consistency Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pin exact call lists for outbound create, outbound reuse and an inbound turn that reads conversation_metadata() on a non-initiating instance. Correct the docs cost sentence, and stop logging the CO response body from create_conversation when metadata was sent (CO may echo the value); log the status code instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

After rebasing onto #128: a reply goes out from the sender the customer messaged (webhook recipient, else the one the conversation's participants hold), now that each message builds its own session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

Voice shutdown can leave active work unfinished, and session reconstruction, snapshot isolation, stream parameters, and sender analytics have unresolved correctness issues.

Review effort: Balanced
Findings: 3 · 3

Open (6)

Detach mutable session data before invoking callbacks · New
Drain admitted handlers and in-flight teardown before shutdown · New
Grace expiry leaves WebSocket and provider resources active · New
Select the VOICE customer when rebuilding mixed-channel sessions · New
Recognize all configured sender numbers for UNKNOWN VOICE participants · New
Preserve inherited TwiML parameters when adding session tokens · New

What changed in this PR

Enables TAC to run across replicas without a shared datastore, extending #123 while preserving #128’s multi-sender behavior.

Changes:

Rebuilds messaging sessions per webhook and supports persisted conversation metadata.
Adds voice teardown hooks, shutdown draining, and replica-aware callback routing.
Moves orchestrated lifecycle analytics to webhooks and updates tests, examples, and deployment guidance.

File	Description
tests/voice_invariants.py	Adds shared teardown assertions.
tests/test_whatsapp_channel.py	Tests stateless WhatsApp routing.
tests/test_voice_channel.py	Updates voice lifecycle coverage.
tests/test_timestamps.py	Tests timestamp parsing and durations.
tests/test_tac.py	Tests shared participant lookups.
tests/test_sms_channel.py	Tests stateless SMS sessions and replies.
tests/test_rcs_channel.py	Tests stateless RCS routing.
tests/test_profile_retrieval.py	Verifies per-message profile refresh.
tests/test_participant_reconciliation.py	Updates reconciliation tests for supplied participants.
tests/test_outbound.py	Updates outbound session and metadata coverage.
tests/test_openai_realtime_provider.py	Tests token-based session configuration.
tests/test_media_streams_shared_provider.py	Tests configuration claims and warnings.
tests/test_integration.py	Tests rebuilt conversation-ended sessions.
tests/test_gpt_live_provider.py	Updates configuration and timer coverage.
tests/test_expiring_dict.py	Tests cache expiry and capacity.
tests/test_conversation.py	Tests conversation metadata APIs.
tests/test_conversation_session.py	Tests metadata access and caching.
tests/test_conversation_metadata.py	Tests metadata limits.
tests/test_chat_channel.py	Tests stateless chat replies and identity.
tests/test_analytics.py	Tests webhook-driven lifecycle reporting.
src/tac/utils/timestamps.py	Adds timestamp and duration helpers.
src/tac/utils/expiring_dict.py	Adds bounded expiring storage.
src/tac/utils/conversation_metadata.py	Filters metadata to API limits.
src/tac/server/fastapi_server.py	Registers voice shutdown draining.
src/tac/server/config.py	Adds shutdown grace-period configuration.
src/tac/models/session.py	Adds persisted metadata access.
src/tac/models/outbound.py	Documents outbound metadata persistence.
src/tac/models/conversation.py	Adds conversation metadata fields.
src/tac/core/tac.py	Shares participant lookups and makes trait fetching optional.
src/tac/core/config.py	Adds instance routing and trait-fetch settings.
src/tac/core/analytics.py	Exposes whether analytics is enabled.
src/tac/context/conversation.py	Adds metadata creation, retrieval, and updates.
src/tac/channels/whatsapp.py	Accepts sessions for agent-address resolution.
src/tac/channels/voice/twiml.py	Uses replica-aware WebSocket domains.
src/tac/channels/voice/provider.py	Defines conversation-ending ownership.
src/tac/channels/voice/media_streams/shared/openai_provider.py	Shares bounded configuration-token handling.
src/tac/channels/voice/media_streams/openai_realtime/provider.py	Uses shared configuration claims and session release.
src/tac/channels/voice/media_streams/gpt_live/provider.py	Removes expiry timers and updates teardown.
src/tac/channels/voice/conversation_relay/twiml.py	Uses replica-aware action URLs.
src/tac/channels/voice/conversation_relay/provider.py	Releases sessions on disconnect.
src/tac/channels/voice/channel.py	Adds lifecycle hooks, rebuilding, routing support, and draining.
src/tac/channels/voice/__init__.py	Exports the call-ended handler type.
src/tac/channels/sms.py	Accepts sessions for agent-address resolution.
src/tac/channels/rcs.py	Accepts sessions for agent-address resolution.
src/tac/channels/chat.py	Resolves chat addresses from supplied sessions.
src/tac/channels/base.py	Removes shared session storage and adds lifecycle reporting.
README.md	Documents lifecycle telemetry cost.
mkdocs.yml	Adds deployment-guide navigation.
getting_started/examples/features/dashboard/dashboard.py	Accepts application-tracked messaging sessions.
getting_started/examples/features/dashboard/app.py	Tracks and clears dashboard sessions.
getting_started/examples/features/chat/app.py	Passes the session when replying.
docs/deployment.md	Explains scaling, affinity, metadata, and shutdown.
CLAUDE.md	Updates architecture and lifecycle guidance.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

instance_public_domain pins status, AMD and recording callbacks only on calls TAC places; inbound call events come from the number's own configuration and still reach the load balancer. A mid-call CLOSED fires from a live snapshot only on the replica holding the call, and other replicas resolve the call to the new conversation only once CO has started one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The call budget counts turns that get a reply; one without makes no POST /Actions. - profile_id comes off the participant list once CO has resolved it; until then a memory fetch falls back to the address lookup. - Add the 128-character metadata key limit. - Only Media Streams records a transcript; on_conversation_ended fires on the instance holding the call outside orchestrated mode; only ConversationRelay has a text send_response. - VoiceChannel.initiate_outbound_conversation: both Media Streams providers place outbound calls too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…atible The stateless rewrite changed MessagingChannel.get_agent_address to take a session, which broke callers passing an id and subclasses overriding the old signature. Restore main's get_agent_address(conversation_id). TAC's own paths now call a private _agent_address(session) that defaults to get_agent_address(session.conversation_id), so an override of the old signature keeps receiving ids; the built-in chat channel overrides it to read channelId off the session, and answers get_agent_address by id from what this instance cached, as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The first parameter was renamed to conversation when send_response started accepting a session, which broke keyword calls passing conversation_id=. Name it conversation_id again, typed str | ConversationSession, so both positional and keyword calls from main keep working and a session can still be passed to skip the participant lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

On main, an orchestrated ConversationRelay session stayed live until CO's CLOSED, so on_conversation_ended got the full session, metadata included, and end_call() fired it straight away. The stateless rewrite released the session at hangup and rebuilt an identity-only one on CLOSED, and deferred end_call() to CLOSED. Keep each ended orchestrated session in a bounded per-instance store (24 h, 10k) until its CLOSED, which then fires with that session on the instance that held the call; other instances, and expired entries, still rebuild from CO. end_call() fires the hook now and marks the kept session so that instance's CLOSED doesn't refire. A conversation CO closed mid-call is neither kept nor refired at teardown. The bound replaces main's leak of sessions whose CLOSED never came. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…lookup fails On main a later turn reused its in-memory session, so a failed participant lookup didn't stop it being answered. The stateless rewrite fetched participants every turn and dropped the message whenever that failed. Keep the last reconciled participants (customer, agent, profile_id) in the per-instance conversation cache, and when the lookup fails answer from them with a warning. With nothing cached, or when the webhook names a different sender than the cached one, the message is still dropped, as main dropped a first message. The happy path is unchanged and costs no extra calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…data Outbound now sends conversation metadata at creation, which main never did, and the client logs no response body when it does. On a 4xx at creation, log a warning naming the metadata keys (not values) as a possible cause; the error still propagates as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Status, AMD and recording callbacks are pinned only on calls TAC places; an inbound call's come from the number's own configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ryanrishi · 2026-10-09T17:48:19Z

-    async def _expire_session_config_token(self, token: str) -> None:
-        """Purge a stashed session_config token if it's still unclaimed after the TTL.
-
-        Covers no-answer, busy, and other outbound-call outcomes that never
-        trigger ``_register_call`` — the only other place this token is popped.
-        Cancelled by ``_register_call`` as soon as it claims the token, so
-        this only actually runs to completion when the call never connects.
-        """
-        await asyncio.sleep(_SESSION_CONFIG_TOKEN_TTL_SECONDS)
-        self._call_session_configs.pop(token, None)


Why don't we need this anymore?

It was a second TTL for the same entries. This PR moves _call_session_configs in the shared Media Streams base to a bounded ExpiringDict (900 s TTL, 1,000-entry cap), so a token whose call never connects (no answer, busy) ages out there. GPT-Live's per-token asyncio.sleep task was the only other expiry, and it covered exactly that case. Dropping it puts GPT-Live on the same path as OpenAI Realtime, which had no expiry on main, and removes a background task per outbound call. The TTL goes from 120 s to 900 s. That fits better, since an outbound call can ring for up to 600 s and the old 120 s could expire a config before the stream connected. test_outbound_call_schedules_no_expiry_task pins it.

ryanrishi · 2026-10-09T17:53:38Z

        - `TWILIO_REGION`: Twilio region for data residency (e.g., 'au1', 'ie1')
        - `TWILIO_STUDIO_HANDOFF_FLOW_SID`: Studio Flow SID (FWxxx...) for handoff tool
        - `TWILIO_VOICE_PUBLIC_DOMAIN`: Public domain for voice routes (required for voice)
+        - `TWILIO_INSTANCE_PUBLIC_DOMAIN`: This process's own directly-routable host, for


Can we make sure this is documented in .env.example, readme, etc.?

Done. TWILIO_INSTANCE_PUBLIC_DOMAIN is the only env var added in this PR.

ryanrishi · 2026-10-09T17:56:24Z

+        """Whether any call is still live, being set up, or being torn down."""
+        return bool(self._conversations or self._handler_tasks or self._releases_in_flight)
+
+    async def aclose(self, *, grace_period: float = DEFAULT_DRAIN_GRACE_PERIOD) -> None:


Is this shorthand for "asynchronous close"? Can we just call this close instead since it's already marked async?

yeah and you are right about that but it's a moot point now as the method has been removed based another comment.

ryanrouleau · 2026-10-12T15:09:32Z

@@ -0,0 +1,276 @@
+# Horizontal scaling


Let's rm this before merging to follow current convention or make in the form of a "how to horizontally scale" guide for users

rm'd for now.

ryanrouleau · 2026-10-12T15:38:14Z

+state survive the drain.
+Fail your readiness probe *before* it runs so the balancer stops routing here,
+and keep the grace period, plus those 5 seconds, below your orchestrator's
+termination grace period or the process is killed mid-drain.


Should we be doing all this on behalf of the customer? Seems more of a platform concern. Perhaps we provide guidance?

Agreed and done.

ryanrouleau · 2026-10-12T15:44:34Z


-            await chat_channel.send_response(conv_id, llm_response)
+            # Hand back the session the callback received — no extra API call.
+            await chat_channel.send_response(context, llm_response)


This is a breaking change?

For chat, yes. Fixed now. Overall the send_response signature has been expanded to accept both str(conv_id) and a context object.

ryanrouleau · 2026-10-12T15:46:08Z

+Metadata you pass to `initiate_outbound_conversation` is stored on the
+conversation in Conversation Orchestrator (keys of up to 128 letters, digits,
+`.`, `_` or `-`; string values up to 512 characters; at most 8 keys including TAC's
+`direction`), so a reply can find it on any replica:


Does TAC need to add direction?

honestly no. Only added there because direction was stored in session metadata but seems no one is reading it. I removed it for now. Feel free to call out if you know a legit use case for it.

ryanrouleau · 2026-10-12T15:48:22Z

+`session.metadata` still carries values between turns, including ones you
+write during a turn, but only on the replica that handled the conversation
+before. Treat it as a per-replica scratchpad. Entries that don't fit Conversation Orchestrator's
+limits stay only in `session.metadata`, with a warning.


I'm wondering if we should rename session to turn and just drop trying to cache session data between turns. turn only lives between receiving a message and on_message_ready returning.

If dev wants to cache anything, they do it in their code via conversationId

Keeping session around between turns will cause a memory leak as it'll just grow

Agreed. Messaging no longer keeps anything between turns.

In terms of rename, I agree turn is more appropriate for messaging but probably not suitable for voice. I'll punt that for now.

ryanrouleau · 2026-10-12T15:53:44Z

+
+The one deliberate exception: an orchestrated call's ended session waits in
+`VoiceChannel._ended_sessions` for its conversation's CLOSED webhook. That
+store is bounded and expiring, and CLOSED drops the entry, so it can't leak.


Do we need this _ended_sessions cache?

Not really. removed it.

ryanrouleau · 2026-10-12T15:55:30Z

+  - **Lifecycle analytics** ("Conversation Started"/"Conversation Ended") come from CO webhooks wherever a CO conversation exists: `PARTICIPANT_ADDED` for the customer, and `CONVERSATION_UPDATED`/CLOSED with `duration_ms` = CO's `updatedAt − createdAt`. Each channel reports only its own channel type (`BaseChannel._track_conversation_started` / `_track_conversation_ended`), so one webhook handed to every channel counts once per channel (at-least-once across webhook retries and replicas, like the callbacks). The CLOSED participant lookup is shared via `TAC._list_participants_shared` (in-flight + 60 s), and is skipped when analytics is off and no `on_conversation_ended` is registered. Relay-only and Media Streams voice report at session start/teardown instead. On chat, where customers stay UNKNOWN (no customer reconciliation), each UNKNOWN non-agent participant counts as a start, so a conversation with several such participants over-counts.
 - **ConversationRelay-only mode**: When `conversation_configuration_id` is omitted from TACConfig, TAC runs with just the Voice channel (messaging channels raise at construction), `TAC.retrieve_memory()` returns an empty `TACMemoryResponse`, and the ConversationRelay callback handles session cleanup. Use `tac.is_orchestrator_enabled()` to check mode at runtime.
 - **Multi-sender support**: Each channel has a singular default sender plus a plural allowlist (`phone_number`/`phone_numbers`, `rcs_sender_id`/`rcs_sender_ids`, `whatsapp_number`/`whatsapp_numbers`). Inbound derives the agent address from the webhook's recipient (SMS/RCS/WhatsApp) or the dialed number (voice), validates it against the allowlist. Messaging (SMS/RCS/WhatsApp) drops messages addressed to an unconfigured number; voice falls back to the default sender (with a warning) for a call to an unrecognized dialed number. Outbound selects the sender via an optional `from_` (defaulting to the channel's default sender), and digital handoff sends From the session's active agent number.



An aside but we should probably trim the CLAUDE.md, not to necessary anymore with newer models

ryanrouleau · 2026-10-12T15:57:24Z

 TAC_ANALYTICS_DISABLED=true
 ```

+Reporting when a conversation ends costs one Conversation Orchestrator participant lookup per closed conversation. Disabling telemetry removes that call, unless you've registered an `on_conversation_ended` handler, which needs it anyway.


This will be confusing for readers

Agreed. Seems more like an internal detail. Removed.

TAC never reads `direction`; storing it on the CO conversation cost apps one of CO's 8 metadata keys and, on reuse, overwrote conversations that began inbound. Only app metadata is written now. session.metadata still carries `direction: outbound`, as on main. With no app metadata, creation sends none (as main did), a reused conversation isn't patched, and a rejected creation no longer names metadata as a possible cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

TAC shouldn't hold app state across turns, and a per-instance cache made session.metadata carry over on one replica but not another. On messaging a session now lasts one turn; apps keep cross-turn state themselves, keyed by conversation_id. Outbound metadata stays readable on any replica through conversation_metadata(). Removed with the cache: - the last-known-participants fallback: a failed list_participants now drops the turn (on_error, dropped_inbound) on every replica alike - CO metadata copied into session.metadata on CLOSED; read it through conversation_metadata(), answered from the payload when it has it - chat's get_agent_address(conversation_id) channelId, which came from the cache; TAC's own paths read it off the turn's session Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

On SIGTERM, uvicorn closes every open WebSocket with 1012 before the app's shutdown event runs, so VoiceChannel.aclose()'s grace period never kept a call alive; the handler's guaranteed teardown already fires on_call_ended on that disconnect. Keeping calls through a rollout needs SIGTERM delayed, which only the platform can do. Removes VoiceChannel.aclose(), TACFastAPIServer.aclose() and its shutdown hook, TACServerConfig.shutdown_grace_period, the providers' _force_close_call, and the admission/handler-task tracking only the drain used. None of it shipped on main. The scaling guide now gives platform guidance instead: fail readiness, delay SIGTERM with preStop, size the termination grace period. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

send_response(conv_id) rebuilds the session from list_participants, and Chat's required channelId only arrived on the inbound webhook, so a reply by id raised on Chat where it worked on main. The rebuilt session now takes channelId from the customer's or agent's CHAT participant address. Without one, the error says to pass the ConversationSession. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Drop VoiceChannel._ended_sessions. It only restored the full session to on_conversation_ended when CLOSED reached the replica that held the call; everywhere else the hook already got a session rebuilt from CO. on_call_ended is the hook that sees the live session. end_call() now releases the call like any hangup: on_call_ended fires right away and, when orchestrated, on_conversation_ended fires once on the CLOSED that follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Move docs/horizontal-scaling.md to getting_started/horizontal_scaling.md, where the other user guides live, so docs/ stays the API reference. Rewrite it as task-ordered steps for running TAC on several replicas, leaving the design rationale to CLAUDE.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

TACFastAPIServer acks webhooks before the handler runs, so Twilio only retries on failures and a cross-replica duplicate reply is rare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Drop the links to it; TWILIO_INSTANCE_PUBLIC_DOMAIN stays documented in the README, the getting-started env var list and .env.example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ryanrouleau and others added 22 commits October 5, 2026 21:21

docs: document mid-call CLOSED behavior and the CallSid fallback

803648d

Also tidies a test docstring. Includes review fixes: - docs(voice): note the per-replica conversation id after a mid-call close Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

refactor: add shared ISO 8601 helpers for CO timestamps

84bb518

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs: describe CO-driven lifecycle analytics

9829076

Includes review fixes: - docs: note telemetry lookup cost and at-least-once lifecycle reporting Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat: fit app metadata to CO conversation-metadata limits

5644930

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs: describe conversation metadata across turns

8db05ea

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

wenzhu1587 requested review from ryanrishi, ryanrouleau and xinghaohuang91 as code owners October 6, 2026 04:25

Copilot AI balanced review requested due to automatic review settings October 6, 2026 04:25

wenzhu1587 mentioned this pull request Oct 6, 2026

feat: make TAC safe to run as N replicas with no shared datastore #123

Open

7 of 11 tasks

wenzhu1587 added a commit that referenced this pull request Oct 6, 2026

docs: record §11 gap status after #132; add G11 and V5

c14ca5c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot started reviewing on behalf of wenzhu1587 October 6, 2026 04:27

View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

wenzhu1587 and others added 9 commits October 6, 2026 16:27

docs: describe messaging on_conversation_ended under several replicas

d9c2010

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(config): say which callbacks instance_public_domain pins

5b6e225

Status, AMD and recording callbacks are pinned only on calls TAC places; an inbound call's come from the number's own configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ryanrishi reviewed Oct 9, 2026

View reviewed changes

ryanrouleau reviewed Oct 12, 2026

View reviewed changes

wenzhu1587 and others added 10 commits October 12, 2026 21:15

docs: document TWILIO_INSTANCE_PUBLIC_DOMAIN

e79695f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(readme): drop the end-of-conversation lookup note from Telemetry

d44f2df

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: describe webhook retries instead of requiring idempotent handlers

ff15b3c

TACFastAPIServer acks webhooks before the handler runs, so Twilio only retries on failures and a cross-replica duplicate reply is rare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: remove the multi-replica guide for now

b40497e

Drop the links to it; TWILIO_INSTANCE_PUBLIC_DOMAIN stays documented in the README, the getting-started env var list and .env.example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

wenzhu1587 requested review from ryanrishi and ryanrouleau October 14, 2026 20:23

Conversation

wenzhu1587 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behavior changes reviewers should know

Not yet verified on a live account

Type of Change

Checklist

SDK Parity

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanrouleau Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanrouleau Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

wenzhu1587 commented Oct 6, 2026 •

edited

Loading

ryanrouleau Oct 12, 2026 •

edited

Loading

ryanrouleau Oct 12, 2026 •

edited

Loading