This repository was archived by the owner on Nov 7, 2026. It is now read-only.
Repository navigation
fix(relay): bound shutdown with a deadline and report drain state - #105
Merged
Merged
Conversation
kitlangton
force-pushed
the
fix/relay-drain-deadline
branch
from
October 25, 2026 22:37
15b5aa4 to
aa384e0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On Oct 10, the production relay on
:19989(pid 40439) stopped accepting work for about 45 minutes. Every execute returnedRelay is draining for an explicit restart; retry after it completes,doctorstill reportedStatus: ok, and SIGTERM did nothing; it had to be SIGKILLed. Itslifecycle.jsonlhas noRequestedentry for that drain, so it wasn't a restart request. The relay had been signalled and was stuck in teardown:cleanup()waited without any limit for accepted work to settle and sessions to disconnect. Node's runtime turns every later SIGTERM into another interrupt of a fiber that is already finalizing, so repeat signals never stopped it.What Changes
Before, while any accepted work never settled (reproduced with a stuck
executenext to an idle, connected named session):After:
BROWSER_CONTROL_SHUTDOWN_TIMEOUT_MS(default 10s), then teardown is forced: sockets terminated, lingering HTTP connections closed.ShutdownForced(with holders) is written tolifecycle.jsonlserveexits immediately (143/130)serveexits itselfrelay restart/versionwhile admission is closeddrain: { phase: draining|stopping|closing, since, elapsedMs, holders }.statusprints a warning anddoctorfails a newrelay admissioncheck, recommending next steps with the relay pidRequests rejected during a signal-driven shutdown now say
Relay is shutting down; retry once a fresh relay has started, not that a restart is draining.lifecycle.jsonlalso gains aClosingentry, so a signalled shutdown leaves a trace even without aRequestedentry.Relay Shutdown
sequenceDiagram participant S as Signal (or self-SIGTERM after a committed restart) participant R as Relay cleanup participant G as Graceful phase (detached fiber) S->>R: SIGTERM R->>G: close admission, drain work, close sessions and root lifecycle alt settles within the deadline G-->>R: done else deadline passes R->>R: report holders, log ShutdownForced end R->>R: freeze catalog writes, terminate sockets, close serversThe graceful phase runs in a detached fiber, and the cleanup waits on it with an interruptible, time-limited join.
sessions.closeAll()is uninterruptible on purpose, and a plain timeout around it would wait forever. Catalog writes stop before teardown detaches targets, so neither a forced nor a clean shutdown can drop session ownership on its way out.Scope
This covers the relay's shutdown path, drain reporting, and the
servesignal backstop. I couldn't identify which accepted request or session held the production relay, because the old relay recorded nothing about it; the newdrainfield andShutdownForcedentry will name the holder next time. The installed~/.config/opencode/skills/browser-control/skill.mdhas local edits and isn't synced here; sync it once this ships.Verification
test/relay-drain.test.tsruns an in-process relay with a fake extension that is complete enough for Playwright to open a relay-owned page:instant-hover) is connected: the shutdown is accepted and/versionreportsstopping. After restart, the session is restored from the catalog.executenext to the idle session:/versionreportsclosingwith the holdersession stuck executingand doesn't list the idle session. Teardown is forced within the 300ms test deadline, and both sessions reattach.src/cli.ts servewith an isolated HOME, the fake extension, the idle session and a stuck execute, then sent SIGTERM.instant-hoverandstuck.origin/mainwas still running 20s after SIGTERM, withdrainundefined.