bug(test-benchmark): give every stateful benchmark the same startup blocks

[benaadams]

Description

Warm-up is currently an uncontrolled variable across the stateful benchmark
suite, and for one family it lands directly in the scored payload.

Counting the setup payloads each test delivers before its measured block, over
the 692 tests in the jochemnet/v1 + glamsterdam-devnet-8 stateful set:

setup payloads	tests
0	72
1	560
2	24
3-8	2
9-105	34

The blocks that do exist carry one to three transactions (directly measured at
1.1 per block on the long chains). The 72 with none are
test_ether_transfers_onchain_receivers, whose senders and receivers all exist
in the pre-run bundle, so it allocates nothing from pre and its setup step is
a lone forkchoiceUpdated. Under a runner that recreates the client container
per test — which is what the stateful context does — the measured payload for
that family is the first block the process has ever seen.

That matters only for the two clients with a managed runtime, but it matters a
lot. Per-block cost over a chain of near-empty blocks, median over the 16 tests
whose setup chain is at least 20 blocks long, all clients on identical hardware:

block	.NET client	JVM client	client C	client D	client E
1	170.2	80.7	31.5	2.8	20.0
3-10	21.4	89.2	10.7	18.0	21.8
11-20	22.1	78.5	12.4	20.2	21.4
21-30	18.5	74.5	13.6	22.1	22.1
31-50	20.0	66.8	13.9	24.5	21.4
51-105	19.5	63.2	15.4	30.8	21.3

Against each client's own steady state, the .NET client is 8.7x on the first
block and back to 1.1x by the third; the JVM client is 1.28x on the first,
rises to 1.41x over blocks three to ten, and is still 1.06x at blocks 31-50.
The other three show no warm-up at all — their curves rise, which is state
growth, not compilation.

At the top end this is worth a lot. For the family with no startup block, the
fitted per-block fixed cost is 575 ms for the .NET client against 29-154 ms for
the others, while the marginal cost of 100 Mgas more transfers is within 20%
across the three fastest. The whole gap is the fixed term.

What this changes

Run the same preamble ahead of every stateful benchmark, through the existing
setup_blocks= field, so no scenario is measured under a different warm-up
regime from another.

The two constants are sized from the cost of what they buy:

• Transactions are cheap — microseconds each, against tens of milliseconds
  for a block. So the first block carries 256 transfers, comfortably over the
  invocation counts at which JIT runtimes leave their first tier (.NET promotes
  at 30 and does not start counting until a startup delay expires; the JVM's
  first profiled tier is at 200). That is 5.4 Mgas.
• Blocks are expensive, so there are only four: enough to leave the
  first-block regime with margin. Reaching a per-block promotion threshold
  would take thirty of them, and at tens of milliseconds each that is not worth
  the run time.

Cost, against a per-test cycle of about 32 s on the reference runner: roughly
0.1 s per test for the .NET client and 0.4 s for the JVM one, or 0.3% and 1.3%
of the run. Fixtures grow by 259 transactions per test.

Every account the preamble touches is created by it, so no state a benchmark
reads is warmed; the blocks are setup-phase, so none of it is scored and
expected_benchmark_gas_used is unaffected.

Two tests gained the pre fixture, which they did not previously request.

What this does not fix

The JVM client's curve is still falling at block 100, and no startup block a
benchmark can afford reaches that runtime's second-tier threshold of several
thousand invocations. An AOT image is the lever there. This change stops the
fixture handing it an interpreted first block on top of that.

It is also worth noting separately that one client in this suite opts out of
the container recreate entirely via rollback_strategy: rpc-debug-setHead, so
it keeps its process and caches across every test. That is a larger consistency
question than this PR, but it moves the same numbers.

Testing

just static passes: mypy over 4263 files, ruff check, ruff format --check,
codespell, ethereum-spec-lint, vulture, actionlint, uv lock --check.
All 540 stateful benchmark tests still collect, and the helper was checked
directly to produce four setup-phase blocks of [256, 1, 1, 1] transactions with
259 distinct fresh receivers and a single throwaway sender.

I could not fill the stateful fixtures locally — that needs the bloatnet
snapshot and a client — so the generated fixtures are unverified. Worth a fill
against the snapshot before merging.

Related Issues or PRs

N/A.

Checklist

• Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
• PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

[Copilot]

🔵 Needs a closer look

The common setup helper still needs stateful fixture-fill or equivalent integration-test validation.

Pull request overview

Standardizes warm-up behavior across stateful benchmarks with shared setup-phase blocks before measured execution.

Changes:

• Adds a reusable four-block preamble containing 259 fresh transfers.
• Applies it across Bloatnet, depth, and EIP-7928 benchmarks.
• Adds required pre fixtures.

File summaries

File	Change
tests/benchmark/stateful/eip7928_block_level_access_lists/helpers.py	Adds startup setup to BAL benchmarks.
tests/benchmark/stateful/bloatnet/test_transient_storage.py	Adds startup setup and pre.
tests/benchmark/stateful/bloatnet/test_transaction_types.py	Adds startup setup.
tests/benchmark/stateful/bloatnet/test_sstore.py	Adds startup setup.
tests/benchmark/stateful/bloatnet/test_sload.py	Adds startup setup and pre.
tests/benchmark/stateful/bloatnet/test_create.py	Adds startup setup.
tests/benchmark/stateful/bloatnet/test_call.py	Adds startup setup.
tests/benchmark/stateful/bloatnet/test_account_query.py	Adds startup setup.
tests/benchmark/stateful/bloatnet/depth_benchmarks/test_deep_branch.py	Adds startup setup.
tests/benchmark/helper/transactions.py	Defines shared startup-block generation; integration validation remains required.

Review details

Suppressed comments (1)

tests/benchmark/helper/transactions.py:143

• This helper is now on the common path for every stateful benchmark, but the PR only checks collection/static analysis and explicitly does not fill a fixture against the required snapshot/client. That leaves the critical integration contract—259 valid setup transactions, setup-phase partitioning, and unchanged measured gas/BAL/post-state—unverified; please run the stateful fill or add an equivalent automated fixture test before merging.

    with TestPhaseManager.setup():
        sender = pre.fund_eoa()
        return [
            Block(
                txs=[
                    Transaction(
                        to=pre.fund_eoa(amount=0),
                        value=1,
                        sender=sender,
                    )
                    # Only the first block carries the transactions; the rest
                    # are there for the per-block cost and stay cheap.
                    for _ in range(STARTUP_BLOCK_TX_COUNT if i == 0 else 1)
                ]
            )
            for i in range(STARTUP_BLOCK_COUNT)

• Files reviewed: 10/10 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[LukaszRozmej]

To be completely correct each pre-test warm-up should warm-up the code paths it later benchmarks, but using different data paths. But that is a lot of work. (Applies to stateless benchmarks too, but there at least the run is shared, so warm-up somewhat accumulates in the node).

[spencer-tb]

One structural suggestion, rather than threading setup_blocks=build_startup_blocks(pre) through every call site (and adding pre to tests that don't otherwise need it), we can make this a property of the directory by overriding the benchmark_test spec fixture in a new tests/benchmark/stateful/conftest.py:

"""Pytest configuration for stateful benchmark tests."""

from typing import Any

import pytest
from execution_testing import Alloc, BenchmarkTestFiller, Fork

from tests.benchmark.helper.transactions import build_startup_blocks


@pytest.fixture
def benchmark_test(
    benchmark_test: BenchmarkTestFiller,
    pre: Alloc,
    fork: Fork,
    gas_benchmark_value: int,
) -> BenchmarkTestFiller:
    """
    Prepend the startup blocks to every stateful benchmark.

    Every stateful benchmark is measured on a client process that was
    started for it, so the same preamble runs ahead of each one, whatever
    the scenario.  Overriding the spec fixture here makes that a property
    of the directory rather than of each call site, so a new test cannot
    forget it.  Blocks a test passes through ``setup_blocks`` itself run
    after the preamble.

    ``fork`` and ``gas_benchmark_value`` are requested only so they stay
    in the fixture closure: pytest builds the closure from the innermost
    definition of each fixture name, so the parametrized fixtures the
    spec fixture depends on would otherwise be invisible to the forks
    and benchmarking plugins, which then skip their parametrization.
    """
    del fork, gas_benchmark_value

    class StatefulBenchmark(benchmark_test):  # type: ignore[misc,valid-type]
        def __init__(self, *args: Any, **kwargs: Any) -> None:
            kwargs["setup_blocks"] = build_startup_blocks(pre) + list(
                kwargs.get("setup_blocks", [])
            )
            super().__init__(*args, **kwargs)

    return StatefulBenchmark

With that, all the per test edits in this PR go away and any new stateful test gets the change without having to know about it.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.44%. Comparing base (f7847b7) to head (8ecbeb3).
⚠️ Report is 5 commits behind head on forks/amsterdam.

Additional details and impacted files

@@               Coverage Diff                @@
##           forks/amsterdam    #3596   +/-   ##
================================================
  Coverage            94.44%   94.44%           
================================================
  Files                  624      624           
  Lines                36928    36928           
  Branches              3326     3326           
================================================
  Hits                 34875    34875           
  Misses                1450     1450           
  Partials               603      603           

Flag	Coverage Δ
unittests	94.44% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[benaadams]

This might be better ethpandaops/benchmarkoor-tests#40