Skip to content

fix(iam): retain revocation versions through replay and recovery - #192

Merged
Vonng merged 5 commits into
mainfrom
codex/iam-revision-tombstones
Sep 15, 2026
Merged

Vonng merged 5 commits into
mainfrom
codex/iam-revision-tombstones

Conversation

@Vonng

@Vonng Vonng commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

Problem and resulting behavior

An offline site can replay deleted IAM identities and policy mappings after their deletion versions have disappeared from storage. Periodic healing can then restore access that an administrator revoked. This change retains source-ordered revocations across storage reloads and rejects older identities and grants while allowing deliberate newer recreation.

Depends on #191; merge that first. The first phase alone does not close R3.

Implementation

  • Persist same-path tombstones and retained user/group revocation boundaries on both object and etcd backends; compare and write under a distributed path lock.
  • Preserve source timestamps, per-member group grant times, and signed parent-generation claims. Commit revocations before dependent cleanup and refresh sibling caches from committed state.
  • Reconcile offline deletions using a versioned protocol, bounded batches and per-path acknowledgements that account for receiving-node restarts. Reuse the normal IAM loading index instead of adding a full storage scan each healing cycle.
  • Distinguish immutable STS expiration from permanent identity history. Preserve service-account status, absolute expiration and revocation boundaries when a newer snapshot replaces the same access key.
  • Resume healing after leadership loss and start only one loop across replication configuration reloads.

No dependency changes or unrelated deadline changes are included.

Validation and review

  • Real object/etcd persistence, replay, cache reload, deletion/recreation, policy/group boundaries, lock cancellation and committed-cleanup failure tests.
  • Focused randomized race regression: 78 leaf cases repeated twice, 156 passes, no skips or races.
  • Full local go test -p 4 -tags kqueue,dev ./... passed: 6142 leaf tests, 166 existing conditional skips, zero failures; a disposable real etcd endpoint was enabled.
  • Linux acceptance passed for both object and etcd IAM backends: three sites, two processes and four drives per site, signed S3 GET and STS, offline deletion, cold reload, same-name user/service recreation, and old-secret denial/new-secret acceptance. Observed offline convergence was 41.7 s / 18.8 s; this is not an SLA. Goroutine profiles show one healing loop per process after configuration reloads.
  • golangci-lint, generated files, compatibility baseline, and Docker entrypoint checks passed. The only differences after the functional race run are SPDX notice alignment; the full suite and network binary use the published source tree.
  • Claude Code claude-opus-5, max effort, read-only reviews recommend merging the code with no remaining P1/P2 findings in the covered scope. Runtime validation is separate from that review.

Upgrade and limits

Requires coordinated upgrades across participating sites and sibling nodes. Mixed old/new servers sharing the IAM backend are unsupported. Preserve deletion history in backups; clearing tombstones is not a safe downgrade procedure. The 30-second healing interval is not a convergence SLA. An intermediate object-backend experiment exceeded its original 95-second observation window; that failure was retained. Diagnosis identified initial leader-lock backoff and independently reproduced permanent healing-loop exit on lease loss. The latter is fixed; final experiments use a 240-second observation window without changing the production retry timing.

Ordinary live-group member-removal conflicts, per-token cross-site RevokeTokens, and arbitrary clock/equal-timestamp conflicts are not new guarantees. See the design and operational notes.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Retain source-ordered tombstones and parent grant boundaries across both IAM backends, cache reloads, and deliberate identity recreation. Reconcile deletions through a versioned, bounded replication protocol with restart-aware acknowledgements.

Cover inherited group grants, STS retention, same-key service recreation, absolute expiration, and failures after the durable commit. Document coordinated upgrades and the remaining consistency boundaries.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep one healing loop per process across replication configuration reloads. Reacquire leadership after a lease is canceled and allow shutdown while waiting, so temporary quorum loss cannot permanently stop revocation propagation.

Signed-off-by: Feng Ruohang <rh@vonng.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
@Vonng
Vonng force-pushed the codex/iam-peer-delete-reload branch from 230464c to dff81f2 Compare September 15, 2026 14:59
@Vonng
Vonng changed the base branch from codex/iam-peer-delete-reload to main September 15, 2026 14:59
@Vonng
Vonng force-pushed the codex/iam-revision-tombstones branch from 785323d to 7b4cacc Compare September 15, 2026 14:59
@Vonng
Vonng marked this pull request as ready for review September 15, 2026 15:11
@Vonng
Vonng merged commit 9ebe81c into main Sep 15, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant