Remove access-frequency pool tiering and reconcile version DELETE

[Vonng]

Remove the default-off GET-frequency pool-tiering feature and its dedicated fixes. Ordinary addressed-version DELETE now reconciles resolved copies across pools, preventing a successful deletion from leaving another readable pool copy under the supported quorum conditions. Preserve the independent #178 Object Lock, conditional-delete, metadata, rebalance/decommission, replication-write and shared remote-tier protections.

Qualification

Production code is 41aa84609754769cfb1861d7fd060c2e84182b98, based on 89637554d60c27cfc51d2281d0a4fe15e415f06d. Final delivery head 4d0693cb8c560e88e85d78094c73d208fe9180b6 changes only the history and migration documents relative to that tested code; every other tracked blob is identical. The unrelated local IAM/deadline commit ebc9937d9 is excluded and preserved.

• Full cmd/internal: 6,428 test/subtest passes, 166 skips; 50 packages with tests passed.
• Related race checks: 283 test/subtest passes. Build, vet, lint, generated files and compatibility checks passed. The final delivery head also passed all 11 CI checks, including Go, DCO, VulnCheck and release snapshot validation. No formal artifacts were released.
• R-Upgrade-2: 3 effective PASS / 3 total attempts, with the unmodified candidate Linux binary (SHA-256 28e1339d630a22fa5a0e4659b6182224e81f6cd7cd4079856534390e40a697b1). Runs: 8eb016db (90.20s), 2a2b7b64 (80.83s), 371e7b9f (96.28s).
• Every new run confirmed all 32 coordinator-to-drive paths for three rounds before and after DELETE. Immediate HEAD/GET ran before the post-delete readiness check, so waiting could not hide a transient failure. DELETE204, all observed HEAD/GET404, all-eight-drive target absence, other-version readback from four nodes, old ILM/lifecycle handling and persisted cache v9→v8 all passed.
• Separate pool-fault run 40a59b3b passed: offline DELETE503 SlowDownRead preserved the version on all four source drives; recovered DELETE204, three-node HEAD/GET404, all-eight-drive absence and other-version readback passed.

Claude Code Opus5/max independently verified the final evidence and gave conditional MERGE GO. All four prerequisites are now satisfied: final CI is green, source equivalence is verified, the prior ledger/harness/all three run checksums were recomputed, and the five old main-worktree files were restored only after matching their archived hashes. Their complete snapshot remains recoverable through refs/archive/access-tiering-main-five-files-20260915; the independent IAM commit is preserved. This approves merging, with release and deployment remaining separate deliverables.

Merged on 2026-09-15 as 9df0f4aba. The fetched main merge tree is byte-for-byte identical to the qualified PR tree; no additional production changes entered during merge. The merge commit also passed all seven automatically triggered checks in Go CI and VulnCheck.

Why the initially blocked run was not overwritten

R1 remains three attempts: pre-candidate fixture failure 83f88c59, effective PASS ea58d0c5, and effective FAIL 2059bc6b (DELETE204 followed by one node's HEAD503/GET503). The later all-disk snapshot and successful retries cannot reconstruct that failed request's disk state. Both 2059bc6b and historical 83676ca2 remain OPEN; neither is described as fixed, reproduced exactly or retrospectively passed.

The previous readiness rule (404 samples plus admin disk summaries) admitted incomplete coordinator disk views. A later diagnostic directly recorded DELETE204 with [nil,nil,nil,drive not found] in each pool, and one physical copy retained per pool. That satisfies existing 3-of-4 write quorum, but does not establish every-drive removal. Unique per-coordinator GetObjectTagging probes plus existing storage trace now verify actual ReadVersion responses from every drive; no production error handling was changed.

A matching-write control used identical stopped real-rebalance fixtures: the baseline's existing conditional DELETE and the candidate's ordinary version DELETE both deleted across both pools. Each arm had 96 HEAD/GET observations, all404, with eight-drive absence. It is one matched pair, not proof that all possible mechanisms are excluded. Its 0.60s/17.08s readiness difference did not recur consistently in R2 (17.04s/0.62s/0.60s); no candidate-specific startup slowdown is claimed.

A separate controlled fixture positively reproduced the mechanism: stop destination disk7, DELETE204 at write quorum, verify its residual version, reconnect it, then make two already-cleared destination drives unavailable while retaining namespace quorum in pool0. The deleted version returned503 while a never-written version returned404; the same failed request recorded [drive not found, drive not found, file version not found, nil]. This establishes a causal mechanism class, not the exact cause of the historical instance. This fixture used explicitly manufactured duplicate shards; real rebalance is covered separately. Partial-node restart did not recover every path within35s, so that experiment retains its FAIL. A separately recorded coordinated restart restored all40 paths and five-node HEAD/GET404.

R2 is explicitly a new qualification round under corrected preparation conditions. R1 records and their checksums remain unchanged. Diagnostic logging stayed outside the PR; all disposable runtime resources were archived and cleaned.

Compatibility and operational boundaries

• Ten obsolete ILM keys remain accepted but ignored; ordinary worker settings remain.
• Read v8/v9 usage caches; write v8 and omit retired fields. Historical fixtures cover ordinary fields; runtime version-header checks do not prove complete statistics conservation.
• Ignore retired lifecycle XML. Remove access-only rules before editing; retain ordinary actions in mixed rules.
• Use coordinated stop/replace/restart with matching binaries and environments. Rolling replacement failed the existing bootstrap binary check; partial fault recovery also must not be assumed complete from HTTP404 or admin summaries alone.
• Moved objects stay in their pools. No automatic move-back, all-version cleanup service or object metadata rewrite is added.
• Batch DELETE already fans out. Movement internals, incoming replication, expiration and free-version cleanup retain existing scopes.
• Pending outbound replication may retain VersionPurgePending. Existing per-pool quorum rules remain; success does not guarantee immediate physical removal from every drive.
• Insufficient read quorum returns503 SlowDownRead; other failures keep their mappings. No read error is downgraded to404. Audit event count per HTTP request/target is unchanged.

Tests use one Linux Docker VM and tmpfs. Cross-host, persistent-disk, pressure, all distinct duplicate UUIDs, full runtime statistics conservation, formal releases and production deployment are separate evidence boundaries, not claims supplied by these results.

Full introduction, subsequent fixes, preservation audit, review corrections, failed experiments and the qualified new round: history. Operations: migration guide.

Contribution licensing

Code is contributed under AGPL-3.0-or-later; docs retain CC BY4.0. Original attribution and license notices remain. All seven commits carry author-matching DCO sign-offs. No CLA or separate license grant is introduced.