Skip to content
This repository was archived by the owner on Nov 7, 2026. It is now read-only.

fix(relay): bound shutdown with a deadline and report drain state - #105

Merged
kitlangton merged 1 commit into
mainfrom
fix/relay-drain-deadline
Oct 25, 2026
Merged

kitlangton merged 1 commit into
mainfrom
fix/relay-drain-deadline

Conversation

@kitlangton

Copy link
Copy Markdown
Collaborator

Why

On Oct 10, the production relay on :19989 (pid 40439) stopped accepting work for about 45 minutes. Every execute returned Relay is draining for an explicit restart; retry after it completes, doctor still reported Status: ok, and SIGTERM did nothing; it had to be SIGKILLed. Its lifecycle.jsonl has no Requested entry for that drain, so it wasn't a restart request. The relay had been signalled and was stuck in teardown: cleanup() waited without any limit for accepted work to settle and sessions to disconnect. Node's runtime turns every later SIGTERM into another interrupt of a fiber that is already finalizing, so repeat signals never stopped it.

What Changes

Before, while any accepted work never settled (reproduced with a stuck execute next to an idle, connected named session):

$ kill -TERM <relay>        # relay keeps serving 409 "draining" indefinitely
$ browser-control doctor    # Status: ok
$ kill -TERM <relay>        # no effect; only SIGKILL works

After:

$ kill -TERM <relay>
[browser-control relay] shutdown deadline (10000ms) passed; forcing close while waiting on: 1 accepted request(s), session stuck executing
# exits; the next command starts a fresh relay and both sessions reattach from the catalog

$ browser-control doctor    # while it is still inside the grace period
Status: fail
Relay: reachable (0.8.4, <build>), closing
[fail] relay admission: Relay has been shutting down for 4s (since 2026-10-10T18:10:00.000Z); waiting on 1 accepted request(s), session stuck executing.
Situation Behavior
First SIGTERM/SIGINT Admission closes. Accepted work gets BROWSER_CONTROL_SHUTDOWN_TIMEOUT_MS (default 10s), then teardown is forced: sockets terminated, lingering HTTP connections closed. ShutdownForced (with holders) is written to lifecycle.jsonl
Second signal during shutdown serve exits immediately (143/130)
Teardown still running 5s after the deadline serve exits itself
Idle connected session Never blocks the drain. Persisted ownership is frozen before teardown, so it reattaches on the next relay
Explicit relay restart Unchanged: the existing pre-commit drain timeout still cancels the restart and leaves the old relay running
/version while admission is closed New drain: { phase: draining|stopping|closing, since, elapsedMs, holders }. status prints a warning and doctor fails a new relay admission check, recommending next steps with the relay pid

Requests rejected during a signal-driven shutdown now say Relay is shutting down; retry once a fresh relay has started, not that a restart is draining. lifecycle.jsonl also gains a Closing entry, so a signalled shutdown leaves a trace even without a Requested entry.

Relay Shutdown

sequenceDiagram
  participant S as Signal (or self-SIGTERM after a committed restart)
  participant R as Relay cleanup
  participant G as Graceful phase (detached fiber)
  S->>R: SIGTERM
  R->>G: close admission, drain work, close sessions and root lifecycle
  alt settles within the deadline
    G-->>R: done
  else deadline passes
    R->>R: report holders, log ShutdownForced
  end
  R->>R: freeze catalog writes, terminate sockets, close servers
Loading

The graceful phase runs in a detached fiber, and the cleanup waits on it with an interruptible, time-limited join. sessions.closeAll() is uninterruptible on purpose, and a plain timeout around it would wait forever. Catalog writes stop before teardown detaches targets, so neither a forced nor a clean shutdown can drop session ownership on its way out.

Scope

This covers the relay's shutdown path, drain reporting, and the serve signal backstop. I couldn't identify which accepted request or session held the production relay, because the old relay recorded nothing about it; the new drain field and ShutdownForced entry will name the holder next time. The installed ~/.config/opencode/skills/browser-control/skill.md has local edits and isn't synced here; sync it once this ships.

Verification

./node_modules/.bin/tsc --noEmit --noUnusedLocals --noUnusedParameters
./node_modules/.bin/knip
./node_modules/.bin/jscpd src extension/src --format typescript --min-lines 10 --min-tokens 100 --mode weak
./node_modules/.bin/vitest run --maxWorkers=3
./node_modules/.bin/vitest run test/relay-drain.test.ts test/relay-shutdown.test.ts test/doctor.test.ts
  • Typecheck, knip and jscpd are clean, and all 999 tests pass.
  • The new test/relay-drain.test.ts runs an in-process relay with a fake extension that is complete enough for Playwright to open a relay-owned page:
    • Explicit restart while an idle named session (instant-hover) is connected: the shutdown is accepted and /version reports stopping. After restart, the session is restored from the catalog.
    • A stuck execute next to the idle session: /version reports closing with the holder session stuck executing and doesn't list the idle session. Teardown is forced within the 300ms test deadline, and both sessions reattach.
  • Real-process proof: I spawned src/cli.ts serve with an isolated HOME, the fake extension, the idle session and a stuck execute, then sent SIGTERM.
    • With a 1500ms deadline, it exited after 1519ms and logged the forcing line. The catalog kept instant-hover and stuck.
    • With a 60s deadline, a second SIGTERM 300ms later exited it immediately (143).
    • The same harness against origin/main was still running 20s after SIGTERM, with drain undefined.

@kitlangton
kitlangton force-pushed the fix/relay-drain-deadline branch from 15b5aa4 to aa384e0 Compare October 25, 2026 22:37
@kitlangton
kitlangton merged commit b05a6f2 into main Oct 25, 2026
2 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant