Dolos for epoch-end ledger audits

[mpizenberg]

First, let me thank you for Dolos. I’m currently working on Tessera, a tool implementing CIP-179 on-chain surveys. We want to have reproducible, and independently verifiable survey results, so Dolos is one of the options I’ve explored. This is an experience report about what I did, and where I struggled.

Context

CIP-179 surveys can address different audiences, and weight each response by the responder's on-chain role: a DRep by its voting power, a stakeholder by its stake, etc. When a survey closes, my backend publishes a result artifact with a content hash.

Anyone can re-derive that artifact and compare hashes. I’m working on data sources for the verifier code, and one is Dolos, with Mithril.

What we need

For a survey ending in epoch E, I need the weights of DReps and stakeholders. That means:

Role	Value	Taken at
DRep	voting power	the DRep distribution taken at the end of E
DRep	registered or not	the end of E
Stakeholder	stake, behind a pool that has not retired	the mark snapshot taken at the end of E
Stakeholder	registered or not	the end of E

I also want the electorate totals: the DRep total and the active stake total, but this is less crucial.

I read the survey itself from the chain: label-17 transactions within a slots window, their CBOR, required signers, native scripts by hash, and block and epoch positions. Dolos serves all of that well through mini-Blockfrost and minikupo.

An audit of the survey result can happen any future time after E ends. So I need to read the ledger values after the fact, and that is where I struggled most.

What worked

My verifier's rebuild from Dolos gives the same artifact hash as its rebuild from Koios. I checked the ledger against the Haskell ledger state and against Koios, on preview:

Epoch	What	Result
1428	DRep voting power, distribution taken at the end of 1428	all 3023 equal Koios
1428	account mark stake behind its mark pool	all 372 accounts I checked equal Koios, including 7 that redelegated during the epoch
1395	DRep registration	all 8978 DReps agree
1395	DRep voting power	all but one agree (#1364)
1395	/epochs/1395/stakes	all 79734 rows equal the Haskell set, and their total equals Koios's active_stake
1395	stake registration	all 302383 credentials agree
1395	label-17 transactions	CBOR identical byte for byte

Also good:

• The Mithril bootstrap checks the certificate chain and a Merkle proof over the files.
• stop_epoch is documented and predictable.
• dolos serve on a stopped store never syncs past it.
• A copy resumed with bootstrap --continue gave the same dolos snapshot digest as a replay from genesis.

Workaround: stop the node at the boundary

Routes serve current values only, so I build one node per audited epoch and stop it right after E ends. I bootstrap the node from Mithril with chain.stop_epoch = E + 1. The boundary out of E runs when E + 1's first block arrives, Dolos applies that block and stops. The store then holds the ledger at the end of E, plus one block.

What was hard:

• One block too far. No supported way stops at the end of E with the boundary applied. That extra block can change a responder's registration, and a DRep unregistration in it zeroes the voting_power we need. It’s a problem for me.
• Picking --download-end. The replay skips the highest downloaded file, so --download-end N replays through N - 1. I compute one past the file holding E + 1's first slot.
• The bootstrap exits non-zero when stop_epoch halts it (forced stop epoch reached). A script can't tell that from a real failure, so mine reads the tip with dolos data summary instead.
• dolos init can't be fully scripted. I needed "keep everything" for history, and the bootstrap question comes after the config is saved, so the user has to run it manually and Ctrl-C there at the last question.
• The store is locked while served. dolos data summary fails if serve runs, and nothing reads the store alongside it.

Backup: dump-entity

Even on the stopped node, no route serves an account's mark stake and pool. So I fall back on dolos data dump-entity, which prints the stored entity. The verifier serves the Dolos for the chain reads, stops it, then dumps each responder:

Namespace	Key	Fields we read
accounts	credential CBOR: 8200581c / 8201581c + hash	mark stake (utxo + rewards - withdrawals), mark pool, registered_at, deregistered_at
dreps	CIP-129 payload: 22 / 23 + hash	voting_power, registration and unregistration slots
pools	pool id	retirement as of E

I judge registration by slot against E + 1's first slot, not from the routes' flags, because the stopped node has already applied one block of E + 1.

I know this is outside what dump-entity is meant for. The output is Rust Debug, so I parse fields by name, fail on any shape I don't know, and pin the Dolos release. It works, and it lets me audit today. History routes would let me drop it.

Possible bugs and smaller gaps

Please double check if I’m mistaken.

• Protocol-9 vote delegations (#1364, already filed): Dolos keeps some the Haskell ledger cleared, which overstates those DReps' power. It was the one DRep power mismatch at 1395.
• /accounts/{id}/registrations omits certificate 12, the one that registers a stake credential and delegates its vote together.
• A DRep's unregistered_at stays after it re-registers. I saw 52 such DReps at 1428. I compare the slots to decide registration, but I expected the field to clear, as an account's registered_at clears on deregistration.
• Native scripts: mini-Blockfrost's /scripts/{hash}/cbor is null for them, so I use minikupo. Minikupo indexes witness sets and outputs only, so it misses a script published only in auxiliary data, and it doesn't say when a script first appeared.

What would help

History routes would remove both workarounds. One synced node could then serve an audit of any epoch, with no stopped store per epoch and no dumps. In order of impact:

1. DRep voting power per epoch, from the distribution taken at each boundary, with registration at that boundary. Today /governance/dreps/{id} gives the current value only, and your source defers the history ("a per-epoch history, if APIs ever want one, is a new field at a higher index").
2. An account's mark stake and pool per epoch, served sooner. /accounts/{id}/history already logs active stake, but the snapshot taken at the end of E shows up there only as active stake for E + 2, once E + 3 closes. That is about three epochs after I need it. A route serving the snapshot as soon as it is stable would be perfect.
3. The DRep total per epoch, or a DRep list with amounts (#1121 is solving that?).
4. Stake registration at a past slot or epoch. /accounts/{id}/registrations would cover it once it includes certificate 12.

Smaller things that would help anyone scripting Dolos, as I do:

• a distinct exit status when stop_epoch halts a bootstrap;
• flags for every dolos init question, the history setting included;
• a documented way to stop exactly at the end of an epoch, with the boundary applied;

Setup

I ran Dolos 2.0.0-alpha.0 (a08c9d13), on preview, on a 12-core Mac with 36 GB of RAM. For E = 1428, the node's dolos.toml comes from dolos init with "keep everything" history. I merge this overlay over it with -c:

[chain]
stop_epoch = 1429

[serve.minibf]
listen_address = "[::]:3000"

[serve.minikupo]
listen_address = "[::]:1442"

dolos -c audit.toml bootstrap mithril --download-end 28581
dolos -c audit.toml serve                              # chain reads
dolos data dump-entity --namespace accounts --key 8200581c<hash>

Step	Wall time	Memory	Disk
Replay from genesis to the end of 1395	53 min, 6 of them download	7.0 GB resident	14 GB download, 14 GB store
Replay to 1429's first block	about 50 min	not measured	16 GB
Rebuild of the survey from the node	seconds, plus the dumps	not measured	none

My code is in packages/dolos, which setups the Dolos node and reads the dumps. The auditor's guide is docs/AUDIT.md. I haven't tried preprod or mainnet yet.