[bug]: one blocked chainWatcher disables UtxoSweeper and TxPublisher node-wide — unrelated BTC channels stop being swept

[bufo24]

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

v0.21.3-beta (in lightning-terminal v0.17.5-alpha)

LND Configuration

Integrated lnd+tapd inside litd.
lnd.db.backend=postgres
lnd.bitcoin.node=neutrino
lnd.protocol.simple-taproot-chans=true

Backend Version

n/a — neutrino

Backend Configuration

neutrino light-client backend, mainnet

OS/Distribution

Linux x86_64, official lightninglabs/lightning-terminal image

Bug Details & Steps to Reproduce

One chainWatcher whose ProcessBlock never returns stops on-chain sweeping for
every channel on the node, including plain BTC channels that have nothing to do
with the stuck one.

ChainArbitrator, UtxoSweeper and TxPublisher are registered in a single blockbeat
queue and notified via DispatchSequential, which returns on the first consumer error.
UtxoSweeper is one node-wide instance, so there is no per-channel fallback. Once
ChainArbitrator.ProcessBlock exceeds DefaultProcessBlockTimeout (60s), the sweeper
and fee-bumper are skipped for that block — and every block after, indefinitely.

Worth noting the inner dispatch is concurrent: ChainArbitrator fans the beat out to
its watchers with DispatchConcurrent, so healthy ChannelArbitrators do receive
it and do run their resolvers — they hand inputs to the sweeper normally. But
DispatchConcurrent waits for all of them. Two wedged watchers never return,
ChainArbitrator blows its own timeout, and the outer sequential queue aborts before
the sweeper is ever reached. Inputs go in; nothing comes out.

Our trigger is aux-side and filed separately (lightninglabs/taproot-assets#2312):
litd's taproot-assets AuxChanCloser.FinalizeClose is called synchronously from
dispatchCooperativeClose on the chainWatcher.closeObserver goroutine, and blocks
forever retrying a funding-input proof that the universe server does not have. We are
not asking lnd to fix that. The lnd question is why one consumer's fault is fatal to
unrelated channels' funds.

Steps to reproduce:

1. Run a node with a channel whose chainWatcher can block in an aux callback — any
   AuxChanCloser doing unbounded, retrying network I/O in FinalizeClose.
2. Have ordinary BTC channels in force-close resolution on the same node.
3. Make the aux callback unable to complete.
4. Every block: process block timeout for that watcher, queue=1 aborts, and no
   sweep is published for any channel on the node.

Scope: multiple mainnet nodes, 2 wedged channels each, 6+ days, across 3 restarts.
Reproduces on litd v0.17.0, v0.17.4 and v0.17.5, all on lnd v0.21.3-beta.

Expected Behavior

One misbehaving blockbeat consumer should not prevent unrelated consumers in the same
queue from receiving the beat. A single stuck chainWatcher should not disable
UtxoSweeper and TxPublisher for every other channel on the node.

Any of these would have bounded the damage:

1. Isolate a failing or timed-out consumer instead of aborting the rest of the queue —
   skip it and continue dispatching.
2. Treat a consumer that repeatedly exceeds DefaultProcessBlockTimeout as faulted,
   log at ERR naming it, and keep the remaining consumers running.
3. Fail loudly. Beat [2/4]: implement blockbeat #8894 describes the intent as "if any of the consumers fail to process
   the block under 30s ... the system will shut down as it's critical to handle blocks."
   These nodes did not shut down — they ran for days with sweeping silently
   disabled. If that was deliberately relaxed when the timeout became 60s, the
   silent-degradation path seems worth revisiting: "no sweeping at all, indefinitely"
   is a worse failure than a crashloop, because nothing surfaces it.

Related: #10389 (blockbeat retry mechanism). This is the production consequence of
that gap.

Debug Information

Every block for 6+ days, heights contiguous. At 968003:

[ERR] CHIO: Height[968003]: Consumer=ChainWatcher(CHAN_A:0) failed to process block: consumer ChainWatcher(CHAN_A:0): process block timeout
[ERR] CHIO: Height[968003]: Consumer=ChainWatcher(CHAN_B:0) failed to process block: consumer ChainWatcher(CHAN_B:0): process block timeout
[ERR] CHIO: Height[968003]: Failed to process block: consumer ChainArbitrator: process block timeout
[ERR] CHIO: Height[968003]: Notify block failed: queue=1 got err: consumer ChainArbitrator: process block timeout

Identical at 967981–967988 — 31 of 31 blocks in a sampled 6h window, unbroken since.

[INF] CHIO: Registered queue=1 with 3 blockbeat consumers
[INF] CHIO: BlockbeatDispatcher is starting with 1 consumer queues

Where the watcher is parked. Shutting the daemon down cancels the context and unwinds
the stack, naming the exact frame — note it took a process shutdown to end it:

[ERR] CNCT: Failed to handle confirmed spend: handle coop close: finalize coop close:
  unable to import output proofs: unable to fetch funding input proofs:
  unable to fetch input proof 0 (PrevID(outpoint=<redacted>, id=<redacted>, script_key=<redacted>)):
  error fetching proof provenance: fetching single proof failed:
  proof backoff receive attempt has failed: backoff wait: back off handler context done

The second wedged watcher was parked one frame earlier, in the proof-archive lookup:

[ERR] CNCT: Failed to handle confirmed spend: handle coop close: finalize coop close:
  unable to import output proofs: unable to check if proof exists: context canceled

Neither ever completed on its own across three restarts spanning six days.

Resolvers run and hand off; the sweeper never publishes. Captured ~90s after a restart,
at height 968002:

[INF] CNCT: ChannelArbitrator(<redacted>): commitSweepResolver(<redacted>): sweeping commit output using budget=<redacted> BTC   (x19)
[INF] SWPR: Registered sweep request at block 968002: witness_type=CommitmentTimeLock,          amount=<redacted> BTC, deadline=970547, state=Init
[INF] SWPR: Registered sweep request at block 968002: witness_type=CommitmentToRemoteConfirmed, amount=<redacted> BTC, deadline=968985, state=Init
[INF] SWPR: Registered sweep request at block 968002: witness_type=CommitmentAnchor,            amount=<redacted> BTC, deadline=968985, state=Init
...                                                                                              (38 total)
→ 0 published, 0 fee bumps

80 sweep requests registered / 0 published over 3 days on one node. All ordinary BTC
commitment outputs on non-asset channels. Deadlines in the set ranged from 968013
(18 blocks out at the time) to 970986.

Goroutine profile available on request — --profile is not enabled on these nodes, but
the wedge reproduces within ~90s of every restart, so capturing one is straightforward.

Environment

No response