Found while fixing the v1 instance (#2806). v2 has the delete-side half of the same bug, even though #2807 already fixed its publication ordering.
The code
pkg/lifecycle-poc/service.go:1658 — the recovery-failed arm falls through to an unconditional:
s.runningPipelines.Delete(rp.pipeline.ID)
and v2's Set at :1702 has no rollback by design — its comment at :1697 explicitly delegates cleanup to the goroutine "including when the UpdateStatus below fails".
The scenario
Identical to the one that shaped the v1 fix: in a recovery chain, the outer cleanup goroutine runs recoverPipeline -> StartWithBackoff -> Start(rp2) synchronously on its own goroutine. If the nested run's UpdateStatus fails, the error unwinds back into the outer cleanup, which falls through to its terminal block and deletes by key — erasing rp2, whose nodes are alive.
Result: rp2 is unreachable via Stop/WaitPipeline, and WaitPipeline returns a false nil off terminalErrors.
The fix
Port v1's deleteRunningPipelineIfCurrent plus its publishMu: compare-and-delete, serialized against publication so it is a real CAS rather than a TOCTOU. csync.Map has no CAS primitive, which is why the mutex is needed rather than just the comparison.
Note on the plans
The v0.20 plan's AC 6 for #2806 said the two packages would end up sharing one rule. They now share publication ordering but have diverged on delete discipline — v1 compares, v2 does not. That divergence should not outlive this issue.
Tier 1. Related: #2806, #2807.
Found while fixing the v1 instance (#2806). v2 has the delete-side half of the same bug, even though #2807 already fixed its publication ordering.
The code
pkg/lifecycle-poc/service.go:1658— the recovery-failed arm falls through to an unconditional:and v2's
Setat:1702has no rollback by design — its comment at:1697explicitly delegates cleanup to the goroutine "including when the UpdateStatus below fails".The scenario
Identical to the one that shaped the v1 fix: in a recovery chain, the outer cleanup goroutine runs
recoverPipeline -> StartWithBackoff -> Start(rp2)synchronously on its own goroutine. If the nested run'sUpdateStatusfails, the error unwinds back into the outer cleanup, which falls through to its terminal block and deletes by key — erasingrp2, whose nodes are alive.Result:
rp2is unreachable viaStop/WaitPipeline, andWaitPipelinereturns a false nil offterminalErrors.The fix
Port v1's
deleteRunningPipelineIfCurrentplus itspublishMu: compare-and-delete, serialized against publication so it is a real CAS rather than a TOCTOU.csync.Maphas no CAS primitive, which is why the mutex is needed rather than just the comparison.Note on the plans
The v0.20 plan's AC 6 for #2806 said the two packages would end up sharing one rule. They now share publication ordering but have diverged on delete discipline — v1 compares, v2 does not. That divergence should not outlive this issue.
Tier 1. Related: #2806, #2807.