Repository navigation
[Operations] Prepare a read-only audit and remediation plan for historical replica state #201
Description
Outcome
Separate prevention of new errors from recovery of state already affected by the R1/R5/R6/R7 defects. Establish a read-only, version-aware inventory and a tested remediation procedure before attempting any production rewrite. No affected production installation or object inventory has been identified by the current review.
R7 explicitly does not rewrite old objects: stored-metadata remediation design. Ordinary COPY can preserve polluted source metadata, and a self-COPY can change versions/timestamps.
Scope
Prioritize persisted aws-chunked Content-Encoding: record exact bucket/key/version, original and proposed headers, trusted source/replica relationships, ETag, size/content checksum, tags, Object Lock and SSE constraints without recording credentials or SSE-C keys. Include non-current versions. Use token-aware classification; ambiguous data remains an investigation item.
Also define bounded operator checks for past tag loss/resurrection and failed marker purge. Do not claim these exist in production without evidence; missing historical revisions cannot be reconstructed from absence alone. IAM historical revocations are covered by the separate IAM upgrade task.
Acceptance
- Deliver a read-only inventory procedure/tool and test it on seeded affected and unaffected fixtures, including non-current versions and mixed Content-Encoding values.
- Produce a reproducible, redacted manifest format that distinguishes confirmed, ambiguous and unaffected records.
- Validate remediation on a clone with raw-byte/checksum preservation, exact-version handling, Object Lock/SSE and replication convergence checks.
- Document rollback and conflict detection; do not recommend blind bulk self-COPY or direct edits to xl.meta.
- Keep production inventory and writes explicitly unexecuted until the deployment and reviewed per-object change list are available. Publish the readiness result and leave any environment-specific execution to a separately authorized run.
This is readiness/tooling and an operator plan, not an assertion that upgrading the binary repaired historical data.
Tracking boundary
Baseline: 40220bd836cbd066ca424fa4dc5dbb90057fb55a (main, PR #196), verified 2026-09-16. R1–R8 are closed within their original defect scopes. This issue tracks the follow-up below. R9 / #79 / #198 is explicitly deferred to a separate effort and is not a dependency of this workstream. Source merge, release artifacts and deployment acceptance must be reported separately.
Follow-up
The prevention defect is #185 (fixed in main by #194); this issue owns readiness for existing state, not reopening that source repair. IAM revocations are handled in #200. Feed the outcome into #202 and #203.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
Vonng commentedon Sep 15, 2026
A read-only inventory tool and bilingual operator procedure are prepared in pgsty/silo.pgsty.com#25. The tool lists all versions, performs an exact-version HEAD for each data version, preserves marker identities, and emits a bounded JSONL manifest. It performs no body reads or object writes.
Six contract tests passed with seeded API responses: non-current/null versions, pagination, unusual object keys, mixed encoding tokens, denied HEAD, changed identity/state and interrupted listings. A failed HEAD remains ambiguous; a partial listing cannot report a complete inventory. The output omits credentials and user metadata values, although object names/version IDs still require redaction before sharing.
The draft procedure explains trusted-source comparison, raw-byte checks, metadata/lock/SSE enrichment, conflict handling and operation-specific rollback. It also bounds historical tag and marker-purge checks. The existing published repair matrix remains the source-versus-release reference.
Keeping this issue open for actual stored-object fixtures and clone remediation with Object Lock, SSE, replication convergence and rollback evidence. The SDK response fixtures do not establish those results. No production installation has been inventoried or rewritten, and no general in-place historical-version repair is claimed.
Vonng commentedon Sep 16, 2026
The inventory tool has now been exercised against actual stored objects, beyond the previously reported SDK response fixtures.
A read-only account inventoried Server 20260903 storage, a stopped-storage clone upgraded to build
70c7ec4a9fbf5fbd64cf23dfd8fe835f31ae2b1d(runtime baseline40220bd836cb), and the upgraded clone after restart. The old image was pinned topgsty/silo@sha256:b616a0cf8cb281e7e6bb3c9b1fb53875b4016a2878223925541c18f82d6c5ca3; the new binary SHA-256 is773506d3cbe670d2b9db833477c8259ccb1a455b5b6ae13945874034622cc3f3.All three inventories agreed on 21 version/marker records: 6 confirmed headers, 2 ambiguous encodings, 12 unaffected headers and 1 delete marker. The fixtures covered a non-current affected version, a null version, Unicode/space/newline/literal-percent keys, genuine gzip bytes, mixed/repeated/case-variant encoding tokens, and an SSE-S3 object with verified GOVERNANCE retention and legal hold. The tool preserved exact-version identities and marked the ambiguous encodings for investigation. Original raw bytes, tags and the verified lock/encryption state survived the upgrade; the affected headers also remained. This confirms that installing the prevention fix does not repair historical metadata.
A narrow remediation rehearsal on the clone corrected one unlocked current gzip object, using an exact source version, source ETag condition and explicitly preserved metadata/tags. Raw gzip bytes and tags matched afterward. COPY created a new version and left the original version's bad header intact; deleting only that newly created unlocked version restored the previous current version. Restarting the clone reproduced the original inventory.
The bilingual procedure and validation scope are updated in website PR #25. The lab was a single-process/single-drive Linux ARM64 server with a static test KMS key. Multi-site convergence, SSE-C/external KMS, locked-version rewrites and a general in-place historical-version repair remain unvalidated. No production installation has been scanned or rewritten. Keeping this issue open for those scoped acceptance items; the existing published repair matrix continues to distinguish source repairs from shipped releases.
Vonng commentedon Sep 16, 2026
The previously missing multi-site clone-remediation check now has bounded runtime evidence. This extends the earlier single-server rehearsal using the same Server 20260903 binary and candidate
70c7ec4a9fbf(runtime baseline40220bd836cb); it does not change the Server implementation.Scope: three sites, two Server processes and four drives per site, inside an isolated Linux ARM64 container. A complete stopped old-version backup was restored into the candidate. The bucket had explicitly configured SSE-S3 defaults at every site, using a static lab KMS key.
The completed run verified:
COMPLETED.Earlier failures remain part of the record. The initial fixture omitted destination encryption defaults and failed its SSE-S3 assertion. Another attempt received
SlowDownWriteimmediately after upgrade. A rollback begun immediately after a later restart encountered target write failures; peer listings still contained replacement versions after the 180-second observation window. That attempt did not test the longer MRF recovery path and does not establish permanent replication failure.The accepted run therefore required signed write/read canaries on all six processes before both mutation phases, in addition to startup/health checks. It also checked exact-version lists and replication completion, rather than accepting current-object reads alone. The English runbook and matching Chinese page in website PR #25 now document these requirements and uncertain-write handling.
Keeping #201 open for website integration and review of the documented limits. SSE-C, external KMS, locked-version rewriting and every delayed-event ordering remain outside this bounded result. No production objects were inventoried or changed, and no universal historical-version repair is approved. Existing published context remains in the component and repair matrix.
Vonng commentedon Sep 16, 2026
The inventory tool and both operator guides are now integrated through website PR #25 and #30, on website main at
6e9d3d31887d018f5c18f7d91a00832879ac69c3. I checked that the guide pages, tool and contract tests are byte-for-byte identical to the reviewed head. The tool SHA-256 remainsccc9d035809b2b41157b4a3f1d35a21108ae4b3af2836e99416a1d2eec1efef2.Canonical links: English procedure, Chinese procedure and read-only tool. Cloudflare Pages reports SUCCESS for the main commit. A direct download request from this environment returned HTTP 403, so I cannot independently attest to the served file bytes; the immutable source is the verified reference.
The bounded clone-COPY, offline-peer convergence, restart and rollback evidence remains as recorded above. Documentation/tool integration is complete; review of the documented limits remains open. This does not establish general in-place historical repair or authorize a production inventory or rewrite.
Vonng commentedon Sep 16, 2026
Closing as completed for read-only inventory tooling and the bounded operator remediation procedure.
The inventory tool and bilingual guides are integrated through website #25 and #30. The recorded validation includes affected/unaffected and ambiguous headers, non-current versions, and preservation of bytes, tags and the verified lock state across upgrade and restart.
The bounded clone rehearsals also cover exact-source-version replacement COPY, an offline peer returning, convergence across three sites/six processes, cold restart, and rollback by deleting only the newly created unlocked replacement versions. COPY creates a new version: the original affected versions and their headers remain. This is not an in-place historical repair.
References:
ccc9d035809b2b41157b4a3f1d35a21108ae4b3af2836e99416a1d2eec1efef2.The rehearsal evidence remains tied to Server 20260903 and candidate
70c7ec4a9fbf(runtime baseline40220bd836cb). Earlier startup write failures and uncertain replication/rollback observations remain in the record. Signed per-process write/read canaries, exact-version listings and replication completion are required before accepting a mutation or rollback.Final-artifact and stack validation remains in #203, including the documented startup/replication observations and public guide/tool delivery verification. Direct public readback still returned HTTP 403 from the review environment; the source links above identify the verified bytes. SSE-C, external KMS, rewriting locked versions, every delayed-event ordering and general in-place repair are outside this completed scope and require separately scoped validation.
Closure means the tool and bounded operating procedure are ready. It does not mean historical production data has been repaired or that all encrypted/locked objects can be repaired automatically. No production inventory or rewrite is claimed by this closure.