Skip to content

FallbackSpeechStream marks healthy STTs unavailable when an AgentTask handoff closes the stream #2349

Closed

Description

@albertocubeddu

In @livekit/agents@1.7.0, stt.FallbackAdapter can treat normal stream teardown as an STT provider failure.

When a FallbackSpeechStream is closed while its active child STT has an in-flight transcript:

  1. SpeechStream.close() closes the parent stream's input, event queue, and output queue and aborts its controller.
  2. The child STT yields a late transcript event.
  3. FallbackSpeechStream.run() calls this.queue.put(ev) after the parent queue has closed.
  4. AsyncIterableQueue.put() throws Error: Queue is closed.
  5. The surrounding provider-level catch treats this local lifecycle error as an STT provider error.
  6. The adapter emits stt_availability_changed with available: false and mutates its shared provider status.

This occurs during ordinary AgentTask activity handoffs, not only when a participant disconnects. Repeated handoffs can mark both healthy providers unavailable and lead to:

all STTs are unavailable, retrying..

Expected behavior

  • A transcript arriving after the parent stream has closed should be discarded as expected teardown.
  • It should not emit a provider-availability change.
  • It should not mark the child STT unavailable.
  • A normal AgentTask handoff should not poison the fallback adapter's provider health state.

For the minimal reproduction below, the expected final state is:

{"availability":[],"status":[true]}

Actual behavior

The late event raises Queue is closed, is logged as an unexpected STT error, and the healthy child is marked unavailable:

{"availability":[{"label":"primary","available":false}],"status":[false]}

Impact

This is not only log noise. The false availability state is stored on the shared FallbackAdapter, so subsequent streams skip or probe providers unnecessarily. With repeated activity handoffs, all children can become unavailable, creating transcription gaps and recovery latency.

In one captured session:

  • 6 agent activity handoffs occurred.
  • The turn-detector stream was reused 6 times.
  • The STT pipeline was reused 0 times.
  • 4 Queue is closed STT warnings occurred.
  • 4 all STTs are unavailable errors occurred.
  • VAD continued detecting speech during periods where no final transcript was produced.

The STTs otherwise returned accurate, high-confidence transcripts before and after the affected transitions.

Relevant log output

Sanitised production excerpt with all user transcripts and identifiers removed:

DEBUG: Agent handoff inserted into chat context
    previousAgentId: "default_agent"
    newAgentId: "default_agent"
DEBUG: reusing turn detector stream from previous activity

DEBUG: Agent handoff inserted into chat context
    previousAgentId: "default_agent"
    newAgentId: "default_agent"
DEBUG: reusing turn detector stream from previous activity

WARN: inference.STT unexpected error, switching to next STT
    stt: "inference.STT"
    err:
      type: "Error"
      message: "Queue is closed"
      stack:
        Error: Queue is closed
          at AsyncIterableQueue.put (.../@livekit/agents/dist/utils.js:217:13)
          at FallbackSpeechStream.run (.../@livekit/agents/dist/stt/fallback_adapter.js:357:24)
          at async FallbackSpeechStream.mainTask (.../@livekit/agents/dist/stt/stt.js:158:16)

ERROR: all STTs are unavailable, retrying..

The deterministic reproduction produces:

WARN: primary unexpected error, switching to next STT
    stt: "primary"
    err:
      type: "Error"
      message: "Queue is closed"
      stack:
        at AsyncIterableQueue.put (.../dist/utils.js:217:13)
        at FallbackSpeechStream.run (.../dist/stt/fallback_adapter.js:357:24)

{"availability":[{"label":"primary","available":false}],"status":[false]}

Describe your environment

The issue was reproduced against a clean npm installation of the official package, without application patches, credentials, network STT providers, or microphone input.

@livekit/agents@1.7.0 was the latest npm release when tested.

The original occurrence used:

  • Soniox stt-rt-v5 as primary.
  • LiveKit Inference deepgram/nova-3 as fallback.
  • voice.Agent.create() for the parent.
  • One voice.AgentTask.create() per interview question.

The provider-independent reproduction below confirms that neither provider caused the underlying error.

Minimal reproducible example

This reproduction is deterministic and requires no LiveKit or STT provider credentials:

mkdir livekit-stt-close-race
cd livekit-stt-close-race
npm init -y
npm install @livekit/agents@1.7.0 @livekit/rtc-node@0.13.34

node --input-type=module <<'EOF'
import { initializeLogger, stt } from '@livekit/agents';

initializeLogger({ pretty: false, level: 'warn' });

const primary = new stt.testing.FakeSTT({
  label: 'primary',
  fakeTranscript: 'late transcript',
  fakeTimeoutMs: 50,
});

const adapter = new stt.FallbackAdapter({
  sttInstances: [primary],
  maxRetryPerSTT: 0,
});

const availability = [];

adapter.on(
  'stt_availability_changed',
  ({ stt: child, available }) => {
    availability.push({
      label: child.label,
      available,
    });
  },
);

const stream = adapter.stream({
  connOptions: {
    maxRetry: 0,
    timeoutMs: 1_000,
    retryIntervalMs: 0,
  },
});

// Ensure FallbackSpeechStream has created the delayed child stream.
await primary.streamCh.next();

// Simulate normal parent/activity teardown while the child has an
// in-flight transcript.
stream.close();

await new Promise((resolve) => setTimeout(resolve, 150));

console.log(
  JSON.stringify({
    availability,
    status: adapter.status.map(({ available }) => available),
  }),
);

await adapter.close();
EOF

Actual result:

{"availability":[{"label":"primary","available":false}],"status":[false]}

Expected result:

{"availability":[],"status":[true]}

Additional information

Source-level cause

In the @livekit/agents@1.7.0 release commit:

  1. SpeechStream.close() closes this.queue and then aborts the stream:

    /** Close both the input and output of the STT stream */
    close() {
    if (!this.input.closed) this.input.close();
    if (!this.queue.closed) this.queue.close();
    if (!this.output.closed) this.output.close();
    if (!this.abortController.signal.aborted) this.abortController.abort();
    this.resampler?.close();
    this.closed = true;

  2. FallbackSpeechStream.run() forwards child events with an unguarded this.queue.put(ev):

    try {
    for await (const ev of child) {
    this.fallbackAdapter._setActiveStt(sttInstance);
    this.queue.put(ev);
    }
    } finally {
    child.close();
    }

  3. Its surrounding catch treats every non-APIError, including the local closed-queue exception, as an unexpected provider failure and marks the provider unavailable:

    } catch (e) {
    if (e instanceof APIError) {
    this._logger.warn(
    { stt: sttInstance.label, err: e },
    `${sttInstance.label} failed, switching to next STT`,
    );
    } else {
    this._logger.warn(
    { stt: sttInstance.label, err: e },
    `${sttInstance.label} unexpected error, switching to next STT`,
    );
    }
    if (status.available) {
    status.available = false;
    this.fallbackAdapter.emitAvailabilityChanged(sttInstance, false);
    }

Why AgentTask makes the problem frequent

AgentTask.run() pauses the parent activity and starts the task activity, then closes the task activity and resumes the parent:

await session._updateActivity(this, {
previousActivity: 'pause',
newActivity: 'start',
blockedTasks,
});
let runState = session._globalRunState;
if (runState && !runState.done()) {
if (speechHandle && runState._unwatchHandle(speechHandle)) {
suspendedHandles.push(speechHandle);
}
for (const task of blockedTasks) {
if (runState._unwatchHandle(task)) {
suspendedHandles.push(task);
}
}
// It is OK to call _markDoneIfNeeded here: _updateActivity has started
// the AgentTask activity, and any onEnter-generated speech is now watched.
// Only call it when something was actually suspended — a run created
// mid-transition has no watched handles yet (its generateReply is still
// deferred behind the activity lock), and marking done on an empty
// handle set would resolve it before it produced any events.
if (suspendedHandles.length > 0) {
runState._markDoneIfNeeded();
}
}
try {
return await this.future.await;
} finally {
// runState could have changed after future resolved
runState = session._globalRunState;
if (session.currentAgent !== this) {
this.#logger.warn(
`${this.constructor.name} completed, but the agent has changed in the meantime. ` +
`Ignoring handoff to the previous agent, likely due to AgentSession.updateAgent being invoked.`,
);
await oldActivity.close();
} else {
if (runState && !runState.done()) {
for (const handle of suspendedHandles) {
runState._watchHandle(handle);
}
}
const mergedChatCtx = oldAgent._chatCtx.merge(this._chatCtx, {
excludeFunctionCall: !this._preserveFunctionCallHistory,
excludeInstructions: true,
});
oldAgent._chatCtx.items = mergedChatCtx.items;
await session._updateActivity(oldAgent, {
previousActivity: 'close',
newActivity: 'resume',
waitOnEnter: false,
});

The SDK attempts to reuse an STT pipeline only when both agents' prototype sttNode methods are exactly Agent.prototype.sttNode:

https://github.com/livekit/agents-js/blob/c1bf4cdcab5c6fde963036f8e9ccb77f1dd38e0f/agents

====
A similar issue is being seen and fixed here: #839

LiveKit hit the identical bug on the sibling output.put() call inside monitorMetrics() (stt.js:214-227) and already patched around it — the code comment literally calls it "expected during disconnect" (GitHub #839, fixed by #861). That fix covers output.put(). It does not cover this.queue.put(ev) inside FallbackSpeechStream.run() — still unguarded, and 1.7.0 is the current latest release (checked npm, nothing newer fixes it).

=====

return new soniox.STT( {
...( apiKeyOption ? { apiKey: apiKeyOption } : {} ),
model: 'stt-rt-v5',
languageHints: [ 'en' ],
languageHintsStrict: true,
maxEndpointDelayMs: 3_000,
} );

Works perfectly.

Relevant log output

WARN: inference.STT unexpected error, switching to next STT
stt: "inference.STT"
err:
type: "Error"
message: "Queue is closed"
stack:
Error: Queue is closed
at AsyncIterableQueue.put (.../@livekit/agents/dist/utils.js:217:13)
at FallbackSpeechStream.run (.../@livekit/agents/dist/stt/fallback_adapter.js:357:24)
at async FallbackSpeechStream.mainTask (.../@livekit/agents/dist/stt/stt.js:158:16)

ERROR: all STTs are unavailable, retrying..


### Describe your environment


```text
System:
  OS: macOS 26.5.1
  CPU: (10) arm64 Apple M1 Pro
  Memory: 387.30 MB / 16.00 GB
  Shell: 5.9 - /bin/zsh

Binaries:
  Node: 24.19.0
  npm: 11.17.0
  pnpm: 11.13.1

npmPackages:
  @livekit/agents: 1.7.0
  @livekit/agents-plugin-soniox: 1.7.0
  @livekit/rtc-node: 0.13.34

Minimal reproducible example

No response

Additional information

No response

Activity

  1. swayamg20 commented on Aug 30, 2026

    @swayamg20
    Contributor

    Hi, I checked this against current main and the teardown path is still present. I’d like to take a narrow fix that treats a closed parent queue during expected stream teardown as a local lifecycle event, without changing genuine provider-failure handling. Please let me know if that direction conflicts with the intended fallback semantics.

  2. dtran26 commented on Sep 14, 2026

    @dtran26
    Contributor

    Still reproducible on @livekit/agents@1.8.1, using Node 22.23.2 on macOS. We independently reproduced the same queue-close race with the installed SDK and synthetic STT providers; no credentials, audio input, or network calls are required.

    The reproduction confirms that a late transcript after FallbackSpeechStream.close():

    1. Produces inference.STT unexpected error, switching to next STT with Error: Queue is closed.
    2. Marks the healthy primary provider unavailable, even though the parent stream is already closed and aborted.
    3. Creates a primary recovery stream and a secondary stream after closure.
    4. Causes a replacement stream sharing the adapter to select the secondary provider.

    No adapter error event is emitted in this reproduction, so a session can avoid a terminal STT error while its fallback state changes incorrectly.

    Production observation

    We also observe this warning in production around handoffs and call cleanup, with the same stack below. We have not quantified unintended production provider switches or established speech loss from this issue.

    Relevant log output

    inference.STT unexpected error, switching to next STT
    Error: Queue is closed
        at AsyncIterableQueue.put (.../@livekit/agents/dist/utils.js:261:13)
        at FallbackSpeechStream.run (.../@livekit/agents/dist/stt/fallback_adapter.js:357:24)
        at async FallbackSpeechStream.mainTask (.../@livekit/agents/dist/stt/stt.js:162:16)
        at async runMainTask (.../@livekit/agents/dist/stt/stt.js:138:9)
    

    Expected behavior

    Closing or aborting a stream should discard late events and finish child-stream cleanup without marking providers unavailable or starting new fallback/recovery streams. A subsequent stream should retain the preferred healthy provider. Actual provider failures should continue to trigger fallback.

    Reproduction

    Save the following as reproduce.mjs in a project with @livekit/agents@1.8.1 installed, then run yarn node reproduce.mjs. The assertions describe the current buggy behavior and pass on 1.8.1. The script first verifies normal primary recognition, then checks closure and provider selection on a replacement stream.

    Runnable reproduction
    import assert from 'node:assert/strict';
    import { setTimeout as delay } from 'node:timers/promises';
    import { initializeLogger, stt } from '@livekit/agents';
    
    initializeLogger({ pretty: false, level: 'warn' });
    
    // Synthetic provider boundary; all fallback and queue lifecycle code is the
    // installed @livekit/agents 1.8.1 implementation. No network calls are made.
    class ControlledStream extends stt.SpeechStream {
      async run() {
        if (this.abortSignal.aborted) return;
        await new Promise((resolve) => {
          this.abortSignal.addEventListener('abort', resolve, { once: true });
        });
      }
    
      emitText(text) {
        this.queue.put({
          type: stt.SpeechEventType.FINAL_TRANSCRIPT,
          alternatives: [{ language: 'en', text, confidence: 1, startTime: 0, endTime: 1 }],
        });
      }
    }
    
    class ControlledSTT extends stt.STT {
      streams = [];
    
      constructor(label) {
        super({ streaming: true, interimResults: true });
        this.label = label;
      }
    
      stream({ connOptions } = {}) {
        const stream = new ControlledStream(this, undefined, connOptions);
        this.streams.push(stream);
        return stream;
      }
    }
    
    async function waitUntil(condition) {
      for (let attempt = 0; attempt < 100; attempt += 1) {
        if (condition()) return;
        await delay(10);
      }
      throw new Error('Condition did not settle within one second');
    }
    
    const primary = new ControlledSTT('inference.STT');
    const secondary = new ControlledSTT('secondary.STT');
    const adapter = new stt.FallbackAdapter({
      sttInstances: [primary, secondary],
      maxRetryPerSTT: 0,
    });
    const errors = [];
    const availability = [];
    adapter.on('error', (error) => errors.push(error));
    adapter.on('stt_availability_changed', (event) => {
      availability.push({ label: event.stt.label, available: event.available });
    });
    
    const original = adapter.stream();
    let replacement;
    try {
      await waitUntil(() => primary.streams.length === 1);
      primary.streams[0].emitText('Recognized normally while the original stream is open');
      assert.equal((await original.next()).value.alternatives[0].text,
        'Recognized normally while the original stream is open');
      assert.equal(secondary.streams.length, 0);
      assert.equal(availability.length, 0);
      console.log(JSON.stringify({ phase: 'open_control', primaryWorked: true, fallbackStreams: 0, availability }));
    
      original.close();
      assert.equal(original.closed, true);
      assert.equal(primary.streams[0].closed, false);
      primary.streams[0].emitText('Late final transcript after the wrapper was closed');
      await waitUntil(() => secondary.streams.length === 1);
      assert.equal(adapter.status[0].available, false);
      assert.equal(primary.streams.length, 2);
      console.log(JSON.stringify({
        phase: 'closed_wrapper_with_late_event',
        closed: original.closed,
        aborted: original.abortSignal.aborted,
        availability,
        primaryStreams: primary.streams.length,
        fallbackStreams: secondary.streams.length,
        sessionErrors: errors.length,
      }));
    
      replacement = adapter.stream();
      await waitUntil(() => secondary.streams.length === 2);
      secondary.streams[1].emitText('Replacement stream uses the secondary provider');
      assert.equal((await replacement.next()).value.alternatives[0].text,
        'Replacement stream uses the secondary provider');
      assert.equal(errors.length, 0);
      console.log(JSON.stringify({
        phase: 'replacement_stream',
        receivedText: true,
        selectedProvider: 'secondary',
        primaryAvailable: adapter.status[0].available,
        sessionErrors: errors.length,
      }));
    } finally {
      original.close();
      replacement?.close();
      for (const provider of [primary, secondary]) {
        for (const stream of provider.streams) stream.close();
      }
      await adapter.close();
    }

    Relevant observed output, with the warning shown separately above:

    {"phase":"open_control","primaryWorked":true,"fallbackStreams":0,"availability":[]}
    {"phase":"closed_wrapper_with_late_event","closed":true,"aborted":true,"availability":[{"label":"inference.STT","available":false}],"primaryStreams":2,"fallbackStreams":1,"sessionErrors":0}
    {"phase":"replacement_stream","receivedText":true,"selectedProvider":"secondary","primaryAvailable":false,"sessionErrors":0}

    The 1.8.1 source still forwards child events into this.queue without checking cancellation, then classifies the resulting exception as a provider failure and updates shared availability: fallback_adapter.ts, lines 476–516. SpeechStream.close() has already closed that queue and aborted the stream: stt.ts, lines 538–545.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions