Seal payloads only once every live worker can read them

[ppXD]

Summary

• Fleet capability gate. Every Hangfire job server of this build lists the marker cap-sealed-payload-v1 among its queues. Nothing is ever enqueued to it, and a previous build never lists it.
  • The gate covers every new row that hands a payload to another run: an agent task, a flow.subworkflow child's payload, and a replay or rerun fork of a sealed run.
  • While a live server (heartbeat within 2 minutes) lacks the marker, or no server is live, the sealer writes the legacy unsealed shape. That is the row the previous build wrote and the only one it executes correctly.
  • The gate reads hangfire.server at most once per 30 seconds. An unreadable table answers no.
  • Forks used to copy the original's ciphertext verbatim. They now open it and seal it again through the gate (WorkflowSensitivePayloadSealer.SealCopy, WorkflowService.cs StageForkedRunAsync).
• Rows already running keep their shape. The executor re-persists an auto run's resolved model onto its Running row through Reseal, which skips the gate. Unsealing that row made a later re-attach fail the handle's redactor fingerprint and lose the run's tail. No previous-version worker launches a Running row.
• Pre-rollback unseal. POST api/admin/sealed-payloads/unseal (global admin, UnsealNonTerminalPayloadsCommand):
  • Restores every non-terminal sealed run payload and every queued sealed agent task.
  • Re-stakes a restored first attempt's acceptance contract.
  • Each row is a compare-and-swap on its ciphertext.
• Recurring-job keeper. A keeper on the worker role re-runs recurring-job registration every 5 minutes. It re-enables the entries a previous-version scheduler disables after it fails to load their job types.

Deploy and rollback

• Rolling worker deploy, any order, RollingUpdate included: no action needed.
  • While old workers are live, current pods write the unsealed shape.
  • Sealing resumes at most 30 s after the last old server leaves hangfire.server, or after 2 minutes without a heartbeat.
• Worker rollback:
  1. Stop every current worker. Wait until none is live (its servers are gone from hangfire.server, or 2 minutes without a heartbeat), then 30 s more.
  2. As global admin, POST api/admin/sealed-payloads/unseal. Send it again if raced > 0.
  3. Start the previous-version workers. Keep the API on the current build.
• After either: check that the four new recurring jobs have a non-empty NextExecution. The keeper restores them within 5 minutes of a current worker being live.

Known limits

• While the gate holds back, new hand-offs keep the plaintext at rest in task_jsonb and normalized_payload_json, and the receiving run does not mask the inherited value. Both are exactly the pre-0247 behaviour.
• A previous-version API pod does not know the sealed column. A replay or rerun it stages of a sealed run copies only the masked column, so the fork runs with [REDACTED].
• At a rollback, terminal rows and running agent runs stay sealed. A previous-version worker's re-attach of a running run lands marker-only either way.
• A fork whose ciphertext this pod's Data Protection keys cannot open now fails at the API call, not when the forked run executes.
• No migration.

Test plan

• Unit.
  • Gate rules: mixed fleets, stale and missing heartbeats, empty and only-stale fleets, marker case.
  • Snapshot lifetime; unreadable fleet; host with no job storage.
  • Seal, Reseal and SealCopy theories; AgentTaskSecrets.Reseal theory.
• Integration (real Postgres and real Hangfire server table).
  • Mixed fleet writes the legacy shape, and the previous build's read gets the secret. An all-current fleet seals.
  • Fork theory: Replay and RerunFromNode against three fleets (all current, previous-version worker live, no worker live).
  • Unseal restores queued and non-terminal rows, leaves terminal and running ones sealed, and re-stakes acceptance.
  • Re-attach theory with a real detached process: the tail is captured even when the gate was closed at launch.
• E2E (worker host).
  • Markers are advertised.
  • The gate answers no once a previous-version server joins, and once the host stops gracefully.
  • The keeper reschedules disabled entries.
• Previous build's real worker binary.
  • It ran forks this build staged, both in the rollback gap and with the previous worker live, and used the real secret.
  • Control: an ungated fork ran with [REDACTED].
• Full suites.
  • Unit 13527 passed. One real-socket flake in broker tests, which this diff does not touch, passed 3 of 3 times when rerun.
  • Integration: every folder green. One MCP endpoint flake passed when rerun alone.
  • E2E: 534 passed.
• Mutations. Switching the executor's write back to the gated Seal, or removing the no-live-worker rule, makes the matching tests fail.