Skip to content

Release 2630 distribution #3271

Closed
Closed

Description

@jpraynaud

Why

In order to deploy new features, improvements and bug fixes, we need to release a new distribution.

Note

The previous distribution has been created with #2967.

What

Create and release a new distribution 2630

Important

  • Delete all legacy crates publication tokens in the GitHub settings
  • Rotate ancillary signer service account keys inrelease-mainnet
  • Remove deprecated features of mithril-client
  • Clean infra upgrade branches if any
  • Next phase of 'Switching to Cardano database incremental certification'
  • Activate DMQ network in Release GitHub workflow
  • Add DMQ compatibility matrix in GitHub release notes
  • Remove CardanoImmutableFilesFull from AGGREGATOR_SIGNED_ENTITY_TYPES in:
    • pre-release-preview
    • release-preprod
    • release-mainnet
  • Activate CardanoBlocksTransactions signed entity type:
    • pre-release-preview
    • release-preprod

Distribution +1:

Note

  • Rotate ancillary signer service account keys inrelease-mainnet
  • Remove deprecated features of mithril-client
  • Clean infra upgrade branches if any
  • Upgrade to Cardano node 11.1.X?
    • pre-release-preview
    • release-preprod
    • release-mainnet
  • Activate CardanoBlocksTransactions signed entity type?:
    • release-mainnet
  • Dev blog post:
    • Distribution XXXX
    • Create a dev blog post explaining the transition to dual genesis
    • Update the Multiple Aggregators testing phase dev blog post
  • TBD

How

Phase 1: Prepare distribution

  • Refresh NPM token for publication
  • Run manual successful restoration of Cardano database + restart of the Cardano node (with ancillary)
  • Run a backward compatibility test in the CI on the unstable distribution (with the unstable built e2e test)
  • Run an all green aggregator stress test in the CI
  • Run an all green client multi-platform test in the CI
  • Prepare a PR to upgrade repository dependencies (with runbook)
  • Reset the CI caches just before merging the PR to upgrade repository dependencies
  • Merge the PR to upgrade repository dependencies
  • Check if Cardano minimum supported versions in networks.json must be updated following the runbook
  • Create a draft PR for rotating the documentation
  • Create a draft PR for updating networks.json file
  • Create a PR for rotating the CHANGELOG.md (set the distribution version)
  • Update the PR for rotating the CHANGELOG.md (if needed)
  • Merge the PR for rotating the CHANGELOG.md
  • Prepare announcements and highlights
  • Prepare dev blog posts

Phase 2: Rollout pre-release distribution

  • Remove CardanoImmutableFilesFull from AGGREGATOR_SIGNED_ENTITY_TYPES in:
    • pre-release-preview
  • Tag and deploy a 2630.0 pre-release distribution on the pre-release-preview network
  • Fill information in GitHub release:
    • Release notes
    • Highlights
  • Make sure pre-release-preview is working as expected
  • Create fix deployment jpraynaud/fix-warp-huge-backtraces branch
  • Activate CardanoBlocksTransactions signed entity type:
    • pre-release-preview
  • Re-deploy pre-release-preview with jpraynaud/fix-warp-huge-backtraces branch:
    • First with previous distribution image 2617.0-2478748
    • Then with new distribution 2630.0-23e124d
  • Make sure pre-release-preview is working as expected
  • Merge fix deployment jpraynaud/fix-warp-huge-backtraces branch on main
  • Make sure Grafana dashboards are fetching data for pre-release-preview
  • Make sure libraries version next have been published on npm registry
  • Run an all green client multi-platform test in the CI
  • Run a Cardano transaction prover benchmark:
    • Store results in the issue
    • Make sure that there is no performance regression since last release
  • Announce the pre-release on discord channel
  • Announce any breaking change in:
    • #cardano-scaling slack channel for pre-release
    • #blockfrost-scaling slack channel for pre-release
    • #ddz-iog-daedalus slack channel for pre-release
  • Wait for at least 2 epochs and check that:
    • Previous signers have registered in the current epoch
    • New signers have registered in the current epoch
    • Previous signers have signed in the 2 epochs
    • New signers have signed in the 2 epochs
    • Certificates are created for the CardanoBlocksTransactions

Phase 3: Rollout release distribution

  • Rotate KMS credentials:
    • Destroy the previous key for the ancillary signer service account of release-mainnet
    • Create a new key for the ancillary signer service account of release-mainnet
    • Configure release-mainnet environment with new service account credentials
    • Check that KMS credentials are working with the creation of the next Cardano database snapshot
  • Release the 2630.0 distribution
  • Fill information in GitHub release:
    • Cardano node compatibility
    • Networks compatibility
  • Make sure release-mainnet is working as expected
  • Make sure release-preprod is working as expected
  • Remove CardanoImmutableFilesFull from AGGREGATOR_SIGNED_ENTITY_TYPES in:
    • release-preprod
    • release-mainnet
  • Activate CardanoBlocksTransactions signed entity type:
    • release-preprod
  • Update CardanoBlocksTransactions configuration parameter AGGREGATOR_CARDANO_BLOCKS_TRANSACTIONS_SIGNING_CONFIG_SECURITY_PARAMETER to 5:
    • release-preprod
  • Re-deploy release-mainnet and release-preprod with jpraynaud/fix-warp-huge-backtraces branch
    • First with previous distribution image 2617.0-2478748
    • Then with new distribution 2630.0-23e124d
  • Make sure release-mainnet is working as expected
  • Make sure release-preprod is working as expected
  • Make sure libraries have been published on crates.io
  • Make sure libraries version latest have been published on npm registry
  • Make sure Grafana dashboards are fetching data for release-mainnet
  • Make sure Grafana dashboards are fetching data for release-preprod
  • Merge the PR for rotating the documentation
  • Merge the PR for updating networks.json file
  • Merge the PR for rotating the CHANGELOG.md
  • Announce the release on discord channel (with breaking-change, upcoming features activation, ...)
  • Announce any breaking/new feature change in:
    • #cardano-scaling slack channel for release
    • #blockfrost-scaling slack channel for release
    • #ddz-iog-daedalus slack channel for release
  • Update Hydra tutorial if client breaking change for database restoration
  • Dev blog post:
    • Distribution 2630
    • Update the Switching to Cardano database incremental certification dev blog post
  • Update roadmap with new distribution
  • Update the signer version in the next Cardano node release

Activity

  1. self-assigned this
    on May 18, 2026
  2. changed the title [-]Release `2623` distribution[/-] [+]Release `2624` distribution[/+] on Jun 3, 2026
  3. jpraynaud commented on Jun 11, 2026

    @jpraynaud
    MemberAuthor

    Manual deletion of old certificates

    pre-release-preview

    # Make sure foreign keys are satisfied
    pragma foreign_key=true;
    
    # Get last genesis
    select * from certificate where parent_certificate_id is null order by rowid desc limit 1;
    
    # Remove signed entity before last genesis
    delete from signed_entity where epoch < 1129;
    
    # Remove certificates beofre last genesis
    delete from certificate where epoch < 1129;
  4. jpraynaud commented on Jun 11, 2026

    @jpraynaud
    MemberAuthor

    Cardano transactions prover benchmark

    Note

    run on GCP with e2-highmem-4 machine with 4 vCPUs and 64GB of RAM
    not during Cardano database snapshotting

    • Benchmark results:
    total_requests concurrency transactions/request requests/s
    500 50 1 117.92
    500 50 10 45.52
    500 50 20 37.54
    500 50 30 34.10
    500 50 40 31.02
    500 50 50 26.00

    vs previous results

    total_requests concurrency transactions/request requests/s
    500 50 1 108.55
    500 50 10 66.64
    500 50 20 50.81
    500 50 30 43.29
    500 50 40 36.71
    500 50 50 34.18
    • Command run:
    # Aggregator endpoint
    export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator
    
    # Transactions file to prove
    export TRANSACTIONS_FILE=transactions-preview.txt
    
    # Transactions proved per request range definition
    export TRANSACTIONS_PER_REQUEST_MIN=0
    export TRANSACTIONS_PER_REQUEST_MAX=50
    export TRANSACTIONS_PER_REQUEST_STEP=10
    
    # Apache benchmark total request sent per benchmark
    export AB_TOTAL_REQUESTS=500
    
    # Apache benchmark concurrency level range definition
    export AB_CONCURRENCY_MIN=50
    export AB_CONCURRENCY_MAX=50
    export AB_CONCURRENCY_STEP=50
    
    # Run the benchmark:
    ./benchmark-aggregator-prover.sh
    • Command result:
    MITHRIL AGGREGATOR PROVER ROUTE BENCHMARK
    
    >> Aggregator endpoint: https://aggregator.pre-release-preview.api.mithril.network/aggregator
    >> Aggregator route: /proof/cardano-transaction
    >> Transactions file: transactions-preview.txt
    >> Transactions available: [100]
    >> Transactions per request range: [0 10 20 30 40 50]
    >> AB concurrency range: [50]
    >> AB total requests per run: [500]
    >> AB total runs: 6
    >> Output file: benchmark.csv
    
    >> [#1/6] Running stress test with 500 requests with 1 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (117.92 requests/s)
    
    >> [#2/6] Running stress test with 500 requests with 10 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (45.52 requests/s)
    
    >> [#3/6] Running stress test with 500 requests with 20 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (37.54 requests/s)
    
    >> [#4/6] Running stress test with 500 requests with 30 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (34.10 requests/s)
    
    >> [#5/6] Running stress test with 500 requests with 40 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (31.02 requests/s)
    
    >> [#6/6] Running stress test with 500 requests with 50 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (26.00 requests/s)
    
    >> Benchmark completed:
    
    total_requests,concurrency,transactions/request,requests/s
    500,50,1,117.92
    500,50,10,45.52
    500,50,20,37.54
    500,50,30,34.10
    500,50,40,31.02
    500,50,50,26.00
  5. changed the title [-]Release `2624` distribution[/-] [+]Release `2625` distribution[/+] on Jun 15, 2026
  6. changed the title [-]Release `2625` distribution[/-] [+]Release `2626` distribution[/+] on Jun 18, 2026
  7. changed the title [-]Release `2626` distribution[/-] [+]Release `2627` distribution[/+] on Jul 6, 2026
  8. changed the title [-]Release `2627` distribution[/-] [+]Release `2628` distribution[/+] on Jul 8, 2026
  9. changed the title [-]Release `2628` distribution[/-] [+]Release `2630` distribution[/+] on Jul 27, 2026
  10. jpraynaud commented on Jul 30, 2026

    @jpraynaud
    MemberAuthor

    Cardano transactions prover benchmark

    Note

    run on GCP with e2-highmem-4 machine with 4 vCPUs and 32GB of RAM
    not during Cardano database snapshotting.

    • Benchmark results:
    total_requests concurrency transactions/request requests/s
    500 50 1 117.92
    500 50 10 45.52
    500 50 20 37.54
    500 50 30 34.10
    500 50 40 31.02
    500 50 50 26.00

    vs previous results

    total_requests concurrency transactions/request requests/s
    500 50 1 108.55
    500 50 10 66.64
    500 50 20 50.81
    500 50 30 43.29
    500 50 40 36.71
    500 50 50 34.18
    • Command run:
    # Aggregator endpoint
    export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator
    
    # Transactions file to prove
    export TRANSACTIONS_FILE=transactions-preview.txt
    
    # Transactions proved per request range definition
    export TRANSACTIONS_PER_REQUEST_MIN=0
    export TRANSACTIONS_PER_REQUEST_MAX=50
    export TRANSACTIONS_PER_REQUEST_STEP=10
    
    # Apache benchmark total request sent per benchmark
    export AB_TOTAL_REQUESTS=500
    
    # Apache benchmark concurrency level range definition
    export AB_CONCURRENCY_MIN=50
    export AB_CONCURRENCY_MAX=50
    export AB_CONCURRENCY_STEP=50
    
    # Run the benchmark:
    ./benchmark-aggregator-prover.sh
    • Command result:
    MITHRIL AGGREGATOR PROVER ROUTE BENCHMARK
    
    >> Aggregator endpoint: https://aggregator.pre-release-preview.api.mithril.network/aggregator
    >> Aggregator route: /proof/cardano-transaction
    >> Transactions file: transactions-preview.txt
    >> Transactions available: [100]
    >> Transactions per request range: [0 10 20 30 40 50]
    >> AB concurrency range: [50]
    >> AB total requests per run: [500]
    >> AB total runs: 6
    >> Output file: benchmark.csv
    
    >> [#1/6] Running stress test with 500 requests with 1 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (117.19 requests/s)
    
    >> [#2/6] Running stress test with 500 requests with 10 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (41.91 requests/s)
    
    >> [#3/6] Running stress test with 500 requests with 20 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (35.97 requests/s)
    
    >> [#4/6] Running stress test with 500 requests with 30 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (31.77 requests/s)
    
    >> [#5/6] Running stress test with 500 requests with 40 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (28.11 requests/s)
    
    >> [#6/6] Running stress test with 500 requests with 50 transactions per request and 50 concurrency
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests
    >>>> Success (28.90 requests/s)
    
    >> Benchmark completed:
    
    total_requests,concurrency,transactions/request,requests/s
    500,50,1,117.19
    500,50,10,41.91
    500,50,20,35.97
    500,50,30,31.77
    500,50,40,28.11
    500,50,50,28.90
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions