capcli dump-blobs-snapshots can silently replace correct historical blob segments with empty ones
and exit 0. Found while repairing a gnosis archive node; it destroyed ~1,580 chunks of previously
sound segments, including ranges that had been downloaded and hash-verified from the CDN.
Three independent defects combine.
1. Completeness checking is disabled
cmd/capcli/cli.go passes nil for blobCountFn:
return freezeblocks.DumpBlobsSidecar(ctx, blobStorage, db, from, to, salt, dirs,
estimate.CompressSnapshot.Workers(), nil, log.LvlInfo, log.Root())In DumpBlobSidecarsRange that sets sanityCheckBlobCount := blobCountFn != nil to false, which
disables both guards. The remaining path is:
if commitmentsCount == 0 {
sn.AddWord(nil)
continue
}commitmentsCount comes from storage.KzgCommitmentsCount(ctx, blockRoot). A block with
commitments whose count row is absent from the blob store therefore reads as "no blobs" and is
written as an empty entry. The dump reports success.
cl/antiquary/antiquary.go:503 has always passed a real function, so the node path is unaffected —
only the manual command.
2. The start slot is pinned to the Deneb fork, and segments are rewritten unconditionally
from := ((beaconConfig.DenebForkEpoch * beaconConfig.SlotsPerEpoch) / snaptype.CaplinMergeLimit) * snaptype.CaplinMergeLimit
DumpBlobsSidecar iterates from → to with no existing-file check, and DumpBlobSidecarsRange
creates a fresh compressor for each range. So --to=30060000 on a node whose frontier was
28,530,000 did not dump 153 chunks; it rewrote ~1,580, back to 14,230,000.
That interacts badly with (1): blobs already frozen below the frontier are pruned from the hot
store, so every one of those ranges took the commitmentsCount == 0 branch.
3. It is version-blind
It writes current-version files regardless of what the range is pinned at. The gnosis manifest is a
mix of v1.0 and v1.1; the dump wrote v1.1 segments and indexes over v1.0 ranges. After
cleanup this left orphan v1.1-*-blocksidecars.idx files whose v1.0 counterparts had been
restored from the CDN.
Reproduction
On an archive node with a pruned blob store:
capcli dump-blobs-snapshots --chain=gnosis --datadir=<dir> --to=<frontier>
capcli check-blobs-snapshots-count --chain=gnosis --datadir=<dir> --from=<deneb-fork>
→ slot 14237725: blob count mismatch, have 0, want 1
The dump exits 0; the checker finds the damage afterwards.
Fixes
Two are implemented on lystopad/capcli-blob-fetch-36 and can be upstreamed as-is:
4ba6a0bec2 — build the same blobCountFn the antiquary uses, so the dump fails on a gap
instead of freezing one.c9ff7a7ea5 — add --from, validated to be a CaplinMergeLimit multiple and at or above the
fork boundary, so a range can be repaired without rebuilding history.
Still open:
- Refuse to write a version the manifest pins lower. With (1) and (2) fixed the destructive path
is much narrower, but a run that legitimately covers a v1.0 range will still emit v1.1.
- Skip ranges whose segment already exists unless explicitly asked to overwrite. Cheap, and it
removes the "rebuild everything by default" behaviour entirely.
Recovery, for anyone who hits this
erigon snapshots reset --datadir=<dir> --local=false drops the preverified lock so OtterSync
reconciles against the remote manifest and re-downloads damaged published ranges hash-verified.
--local defaults to true and deletes the entire chaindata directory — on a snapshotter that
forces a full resync, so pass --local=false and --dry-run first. Segments above the published
ceiling are locally produced and must be re-dumped.
Note that blob segments are blobsidecars.seg while their index is blocksidecars.idx
(db/snaptype/type.go:157), so any glob-based cleanup on one name silently misses the other.
capcli dump-blobs-snapshotscan silently replace correct historical blob segments with empty onesand exit 0. Found while repairing a gnosis archive node; it destroyed ~1,580 chunks of previously
sound segments, including ranges that had been downloaded and hash-verified from the CDN.
Three independent defects combine.
1. Completeness checking is disabled
cmd/capcli/cli.gopassesnilforblobCountFn:In
DumpBlobSidecarsRangethat setssanityCheckBlobCount := blobCountFn != nilto false, whichdisables both guards. The remaining path is:
commitmentsCountcomes fromstorage.KzgCommitmentsCount(ctx, blockRoot). A block withcommitments whose count row is absent from the blob store therefore reads as "no blobs" and is
written as an empty entry. The dump reports success.
cl/antiquary/antiquary.go:503has always passed a real function, so the node path is unaffected —only the manual command.
2. The start slot is pinned to the Deneb fork, and segments are rewritten unconditionally
DumpBlobsSidecariteratesfrom → towith no existing-file check, andDumpBlobSidecarsRangecreates a fresh compressor for each range. So
--to=30060000on a node whose frontier was28,530,000 did not dump 153 chunks; it rewrote ~1,580, back to 14,230,000.
That interacts badly with (1): blobs already frozen below the frontier are pruned from the hot
store, so every one of those ranges took the
commitmentsCount == 0branch.3. It is version-blind
It writes current-version files regardless of what the range is pinned at. The gnosis manifest is a
mix of
v1.0andv1.1; the dump wrotev1.1segments and indexes overv1.0ranges. Aftercleanup this left orphan
v1.1-*-blocksidecars.idxfiles whosev1.0counterparts had beenrestored from the CDN.
Reproduction
On an archive node with a pruned blob store:
The dump exits 0; the checker finds the damage afterwards.
Fixes
Two are implemented on
lystopad/capcli-blob-fetch-36and can be upstreamed as-is:4ba6a0bec2— build the sameblobCountFnthe antiquary uses, so the dump fails on a gapinstead of freezing one.
c9ff7a7ea5— add--from, validated to be aCaplinMergeLimitmultiple and at or above thefork boundary, so a range can be repaired without rebuilding history.
Still open:
is much narrower, but a run that legitimately covers a
v1.0range will still emitv1.1.removes the "rebuild everything by default" behaviour entirely.
Recovery, for anyone who hits this
erigon snapshots reset --datadir=<dir> --local=falsedrops the preverified lock so OtterSyncreconciles against the remote manifest and re-downloads damaged published ranges hash-verified.
--localdefaults totrueand deletes the entire chaindata directory — on a snapshotter thatforces a full resync, so pass
--local=falseand--dry-runfirst. Segments above the publishedceiling are locally produced and must be re-dumped.
Note that blob segments are
blobsidecars.segwhile their index isblocksidecars.idx(
db/snaptype/type.go:157), so any glob-based cleanup on one name silently misses the other.