[pk910]

Summary

Triage of slow pages on dora.hoodi.ethpandaops.io and the /debug/cache 502 on light-hoodi.

Filtered list queries (GetWithdrawalsFiltered, GetDepositsFiltered, GetDepositTxsFiltered, GetMevBlocksFiltered)

• The WITH cte AS (...) SELECT count(*) FROM cte UNION ALL (SELECT * FROM cte ORDER BY .. LIMIT n) pattern references the CTE twice, so Postgres materializes the whole filtered set even for the 20-row page. On the 19M-row hoodi withdrawals table that was >90s per view and saturated the DB for everyone else.
• Page query now runs on its own with DESC NULLS LAST ordering (the schema's ASC NULLS FIRST indexes can only serve that form via a backward scan), count runs separately via countFilteredRows:
  • page not full → total known without counting
  • PostgreSQL: EXPLAIN (FORMAT JSON) first (simple protocol, so the planner sees real values); exact count if cost < 10k, also exact if estimate < 10k rows and cost < 100k (small estimates are unreliable), otherwise the planner estimate
  • SQLite: always exact
• Withdrawals slot filter rewritten from block_uid >> 16 to a block_uid range (index-served, estimable).
• Deposits "valid signature only" filter as EXISTS semi-join: the two-column join was estimated at 12 rows (real 648k) which forced a hash join for the page and a count of 30.

Deposit queue anchor: cached per queue snapshot root. It was re-resolved on every deposits page, validator deposits tab, builder page and deposit API call, each one a full count over deposits when no deposit was in the block cache (1–7 concurrent on hoodi from external API traffic).

Debug page: stored-object stats (slots aggregates + full pebble key scan, ~20s on light-hoodi) are computed in a background goroutine and served as a timestamped snapshot (10 min refresh). Default httpWriteTimeout is now page call timeout + 5s (35s default) so a timed-out page delivers the error page instead of a cut connection.

Migration 20261022000000_perf-indexes.sql: tx_pending_signatures(queuetime) (poller sorted 5.6M rows every 10s) and withdrawals(type, block_uid) replacing the single-column type index (rare-type filter walked 212k rows backward otherwise). Note: the withdrawals index builds at startup inside the goose transaction; a few minutes on mainnet-scale tables.

Measured on hoodi (in-pod EXPLAIN ANALYZE, read-only)

Query	Before	After
withdrawals page, no filter	>90s (timeout)	2 ms
deposits included tab page (join)	1–5 s	3 ms
deposits valid-only page	1.7 s	5 ms
deposit queue anchor	1–3 s per call	0.05 ms, once per epoch
deposit_txs / mev_blocks page	150–340 ms	<0.1 ms
withdrawals count	90 s	0.3 ms (estimate 19.30M vs exact 19.30M)

Single-table count estimates are within 1–3 % of exact.

Test plan

• go build ./..., go vet ./..., go test ./...
• golangci-lint --new-from-rev=origin/master: 0 issues
• All new query shapes verified end-to-end through pgx against the hoodi DB (exact, estimate and page-not-full paths; uint64/uint8/[]byte/string args through the simple-protocol EXPLAIN)
• Deploy to hoodi and re-measure the page sweep

[redpandabot]

Summary

The PR splits filtered-list counts from page queries, caches the deposit queue anchor per snapshot root, and moves debug stored-object stats to a background snapshot; the new commits since the last review are only merges from master that touch no PR-scope files. The previously reported out-of-range pagination bug is still present: an empty page past the end of the result set reports the offset as the total row count.

Issues

• 🟡 db/query_helpers.go:123 — out-of-range page returns offset as the total count — see the thread on that line

---

_{Reviewed @ 975d0a5e
"Documentation is a love letter you write to your future self." — Damian Conway}