ci, db/state: matrix-test serial vs parallel commitment across the test workflows

[mh0lt]

Summary

Parallel execution (dbg.Exec3Parallel) is now the default, so the serial/parallel exec-mode CI matrix added in #21017 no longer needs to toggle it — both legs already run parallel exec. This PR repurposes that matrix axis to exercise serial vs parallel commitment instead, so divergence between statecfg.ExperimentalParallelCommitment true and false is caught on the PR rather than after a release.

Modeled directly on #21017 (the exec-mode matrix PR).

Mechanism

• New env toggle ERIGON_COMMITMENT_PARALLEL. db/state/statecfg/state_schema.go now initializes ExperimentalParallelCommitment from dbg.EnvBool("COMMITMENT_PARALLEL", false); envLookup in common/dbg/dbg_env.go auto-prepends the ERIGON_ prefix, exactly like EXEC3_PARALLEL.
• The existing CLI flag still composes:
  • the main erigon binary only ever flips the flag on (node/eth/backend.go), so the env value survives when the flag isn't passed;
  • the integration command's BoolVar now defaults to the env-derived value instead of clobbering it back to false at flag registration.
• Across the 11 test workflows: the exec_mode matrix axis is renamed commitment_mode, the ERIGON_EXEC3_PARALLEL assignment is dropped (exec is parallel in both legs by default), and ERIGON_COMMITMENT_PARALLEL is driven from the axis. For the two workflow_dispatch perf tests the manual exec_mode input becomes commitment_mode. Docker ENV injection for the hive/kurtosis/txpool runtime images is updated to the new var. Per-mode Go build-cache lineages and artifact/enclave/testbed names carry over unchanged in shape.

Why

ExperimentalParallelCommitment was previously reachable only via a CLI flag, so no CI job exercised the parallel-commitment trie path. Reusing the (now redundant) exec-mode axis gives serial-vs-parallel commitment coverage at no extra matrix cost — same runner count, the axis just tests a different dimension.

Note on coverage trade-off

This drops the dedicated serial-execution CI leg. That is deliberate: parallel execution is the default and is what ships; the serial-exec matrix leg was guarding a non-default path, whereas serial-vs-parallel commitment is the currently-diverging surface worth gating. Serial execution remains runnable locally via ERIGON_EXEC3_PARALLEL=false.

Workflows touched: test-all-erigon, test-all-erigon-race, test-bench, test-hive, test-hive-eest, test-kurtosis-assertoor, qa-stage-exec, qa-txpool-performance-test, qa-rpc-integration-tests-latest, qa-rpc-performance-tests, qa-rpc-performance-comparison-tests.

Latest main sync

The main sync includes #22895, which split the legacy Constantinople race shard into Byzantium and remaining-fork partitions. Both partitions now have sequential and parallel commitment variants. A structured audit confirms that all 14 logical legacy blocktest shards, race and non-race, have matching mode pairs and differ only in the commitment toggle.

Validation

• make lint (two clean runs)
• make erigon integration
• make test-short
• YAML parsing, Bash syntax, and the structured legacy-shard pair audit

TDD was not used for the main-sync conflict resolution and shard-matrix update because these are mechanical CI configuration changes.

[taratorio]

@mh0lt need to also update test-eest-spec.yml shards

[mh0lt]

Good catch — done in f86ceab. The EEST spec shards toggled execution via the -sequential/-parallel suffix (manifest exec3-parallel → ERIGON_EXEC3_PARALLEL), so I repurposed them the same way as the other workflows:

• tools/eest-spec-shards.yml: per-entry field exec3-parallel → commitment-parallel.
• tools/run-eest-spec-test.sh: reads commitment-parallel and exports ERIGON_COMMITMENT_PARALLEL instead of ERIGON_EXEC3_PARALLEL; execution is now parallel in every shard (default).

Shard names are unchanged, so the ci-gate check names and make eest-spec-* targets stay stable — the -sequential/-parallel suffix now selects serial vs parallel commitment. Also rebased onto your main-merge.

[Copilot]

Pull request overview

This PR repurposes the existing CI “serial vs parallel” matrix axis from execution mode (now parallel by default) to state commitment mode, so CI exercises both serial and parallel commitment paths via a new environment toggle.

Changes:

• Add ERIGON_COMMITMENT_PARALLEL env plumbing by initializing statecfg.ExperimentalParallelCommitment from dbg.EnvBool("COMMITMENT_PARALLEL", false).
• Preserve env-derived commitment defaults through CLI flag registration (notably in the integration command).
• Update CI workflows and EEST tooling to drive serial vs parallel commitment via the new matrix axis and env var.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
db/state/statecfg/state_schema.go	Initializes ExperimentalParallelCommitment from env (COMMITMENT_PARALLEL via dbg.EnvBool).
cmd/integration/commands/flags.go	Uses env-derived defaults when registering experimental commitment flags so env isn’t clobbered.
tools/run-eest-spec-test.sh	Switches EEST shard runner to export ERIGON_COMMITMENT_PARALLEL from the shard manifest.
tools/eest-spec-shards.yml	Renames shard-manifest key to commitment-parallel to control commitment mode per shard.
.github/workflows/test-all-erigon.yml	Renames matrix axis to commitment_mode and drives ERIGON_COMMITMENT_PARALLEL for Go tests.
.github/workflows/test-all-erigon-race.yml	Same as above for race-detector test groups with mode-separated build cache keys.
.github/workflows/test-bench.yml	Runs benchmarks in both commitment modes via ERIGON_COMMITMENT_PARALLEL.
.github/workflows/test-hive.yml	Updates Hive matrix to commitment_mode and injects ERIGON_COMMITMENT_PARALLEL into the Erigon client Dockerfile.
.github/workflows/test-hive-eest.yml	Same commitment-mode matrix + Docker ENV injection for Hive EEST runs.
.github/workflows/test-kurtosis-assertoor.yml	Updates Kurtosis matrix axis and bakes ERIGON_COMMITMENT_PARALLEL into the runtime image per leg.
.github/workflows/qa-stage-exec.yml	Switches QA stage-exec matrix to commitment mode and disambiguates artifacts/datadirs accordingly.
.github/workflows/qa-txpool-performance-test.yml	Switches txpool perf matrix to commitment mode and bakes the env var into the runtime image.
.github/workflows/qa-rpc-integration-tests-latest.yml	Switches RPC integration tests to commitment-mode matrix and artifacts naming.
.github/workflows/qa-rpc-performance-tests.yml	Renames workflow_dispatch input and runtime env handling to ERIGON_COMMITMENT_PARALLEL.
.github/workflows/qa-rpc-performance-comparison-tests.yml	Same input/env handling update for the comparison workflow.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[taratorio]

@awskii fyi to check the eest-spec-tests failures

[mh0lt]

Addressed the two shard-config comments in c923faae76: dropped commitment-parallel: true from the suffix-less blocktests-devnet and blocktests-devnet-race-amsterdam shards. Those were mechanical renames of the old exec3-parallel: true (which forced parallel execution — now the default), so as suffix-less shards they now run the default serial commitment, matching the manifest's suffix-selects-mode rule. commitment-parallel is now set iff the shard name ends in -parallel (13 parallel / 13 sequential / 5 suffix-less serial; verified). Parallel-commitment coverage comes from the -parallel shards.

[awskii]

The remaining red legs here (eest blocktests/enginextests parallel, race-tests core-rpc/execution-other parallel) all fail on one class: after an unwind or on a fresh tiny chain, the restored commitment state is a sole-account root with no root branch record, and processMounted either errors (unfold root: empty branch data read during unfold) or silently drops the untouched root leaf (wrong trie root, e.g. TestCallTraceUnwind and TestBlockchainHeaderchainReorgConsistency both produce root f8d8c1e8…).

#22257 fixes this. Verified locally on this branch with it trial-merged: both repro tests pass, and the full enginextests-stable-parallel + blocktests-stable-parallel shards run clean (63920 tests, 0 failed).

So the order is: land #22257, then merge main into this branch again — the parallel legs should go green with no further changes here.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

.github/workflows/qa-rpc-performance-comparison-tests.yml:210

• This step now drives commitment mode, but still uses the EXEC_MODE variable name and messages that refer to a generic "mode". Renaming to COMMITMENT_MODE (and updating the echo text) will prevent confusion with execution-mode toggles elsewhere in CI.

        env:
          EXEC_MODE: ${{ github.event.inputs.commitment_mode }}
        run: |
          set +e # Disable exit on error
          echo "Starting Erigon..."

.github/workflows/qa-rpc-performance-comparison-tests.yml:12

• The PR description says all RPC perf workflows were repurposed from exec-mode to commitment-mode, but qa-rpc-performance-tests.yml still uses the exec_mode input and toggles ERIGON_EXEC3_PARALLEL. This leaves that workflow testing a different dimension than the rest and likely misses the intended serial-vs-parallel commitment coverage.

This issue also appears on line 206 of the same file.

      commitment_mode:
        description: 'Erigon commitment mode ("default" leaves ERIGON_COMMITMENT_PARALLEL unset)'

[yperbasis]

[P2] Give commitment modes distinct QA result identities

The RPC integration upload at

erigon/.github/workflows/qa-rpc-integration-tests-latest.yml

Line 280 in 1845b65

--test_name rpc-integration-tests-latest${{ matrix.commitment_mode == 'parallel' && '-parallel' || '' }} \

and the RPC performance upload at

erigon/.github/workflows/qa-rpc-performance-comparison-tests.yml

Line 393 in 1845b65

--test_name rpc-performance-test-latest${{ (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (matrix.client == 'erigon' && github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}-$method \

reuse result names whose serial/parallel suffix previously meant execution mode. The QA uploader stores no separate mode field, so historical queries and dashboards will combine incompatible configurations under the same series.

Please use explicit commitment-serial and commitment-parallel identities for persisted QA results. The same naming pass should distinguish the two commitment legs uploaded by qa-stage-exec and qa-txpool-performance-test, which currently use one identical test name for both rows.

[mh0lt]

Pushed 6461c82 (merged current main + review items):

@yperbasis [P2] Distinct QA result identities. All four persisted --test_name uploads now carry an explicit -commitment-<mode> identity so the two legs no longer collide with each other or with the historical exec-mode series:

• qa-rpc-integration-tests-latest: rpc-integration-tests-latest-commitment-${{ matrix.commitment_mode }}
• qa-rpc-performance-comparison-tests: -commitment-parallel / -commitment-serial (was -parallel/-serial)
• qa-stage-exec: TEST_NAME is now ${{ matrix.test_name }}-commitment-${{ matrix.commitment_mode }} (the two rows previously shared one name)
• qa-txpool-performance-test: txpool-performance-commitment-${{ matrix.commitment_mode }} (same)

@taratorio #22822 new shards. Merging main pulled in the new legacy blocktest shards; they came in on the old exec3-parallel key. I applied this PR's rename across the manifest so every -parallel/-race legacy shard is now commitment-parallel: true and every -sequential one is serial commitment. run-eest-spec-test.sh reads commitment-parallel and also keeps main's new expected-tests field. The unsuffixed devnet shards (blocktests-devnet, blocktests-devnet-race-amsterdam) stay serial, consistent with the earlier Copilot fix.

Copilot. Renamed the comparison workflow's EXEC_MODE env var → COMMITMENT_MODE to match what it drives. The other note (that qa-rpc-performance-tests.yml still uses exec_mode) is intentional — that workflow is rpcdaemon-only, where the parallel trie is never enabled, so per your earlier [P2] I reverted it to its pre-existing exec_mode rather than add an ineffective commitment axis.

[yperbasis]

CI is red

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (6)

tools/run-eest-spec-test.sh:57

• The comment implies that any shard without a "-parallel" suffix runs with ERIGON_COMMITMENT_PARALLEL=false, but commitment mode is actually driven by the manifest’s commitment-parallel key (and some shards opt in without the suffix). This can mislead future shard additions/edits.

#   *-parallel                                  any of the above with "-parallel"
#                                              appended runs with
#                                              ERIGON_COMMITMENT_PARALLEL=true. Every
#                                              other shard runs with
#                                              ERIGON_COMMITMENT_PARALLEL=false so

tools/eest-spec-shards.yml:25

• The header says the -sequential/-parallel suffix selects serial vs parallel commitment, but there are shards (for example *-race entries) that set commitment-parallel: true without a -parallel suffix. Clarifying that the manifest key is authoritative would avoid confusion.

#   commitment-parallel   (optional, default false) — pins ERIGON_COMMITMENT_PARALLEL.
#                                      Execution is parallel in every shard
#                                      (dbg.Exec3Parallel defaults true); the
#                                      -sequential/-parallel suffix selects
#                                      serial vs parallel commitment.

.github/workflows/test-kurtosis-assertoor.yml:343

• The comment says each suite runs twice, but the matrix includes glamsterdam only in commitment_mode: parallel. Either add the missing serial entry or update the comment to reflect the exception.

        # Each suite runs twice — once with serial commitment
        # (ERIGON_COMMITMENT_PARALLEL=false) and once with parallel — so
        # divergence is caught on the PR. Execution is parallel in both legs
        # (default). Matrix entries spawn separate hosted runners and run
        # concurrently — wall-clock unchanged, runner-minutes doubled.

.github/workflows/test-hive-eest.yml:33

• The comment says each shard runs twice, but the matrix includes glamsterdam-devnet only in commitment_mode: parallel. Either add the missing serial entry or adjust the comment so it matches the matrix.

        # Each shard runs twice — once with serial commitment
        # (ERIGON_COMMITMENT_PARALLEL=false) and once with parallel — so
        # divergence between the two paths is caught on the PR. Execution is
        # parallel in both legs (default). Matrix entries spawn separate runners
        # and run concurrently — wall-clock unchanged, runner-minutes doubled.

.github/workflows/qa-rpc-performance-comparison-tests.yml:489

• The Erigon log artifact suffix still uses -parallel/-serial, which is ambiguous now that the axis is commitment mode (execution is always parallel). Aligning the suffix with the -commitment-* naming used elsewhere would make logs easier to interpret and avoid confusion with historical exec-mode naming.

          name: erigon-logs-${{ env.CHAIN }}${{ (github.event.inputs.commitment_mode == 'parallel' && '-parallel') || (github.event.inputs.commitment_mode == 'serial' && '-serial') || '' }}

.github/workflows/qa-rpc-performance-comparison-tests.yml:12

• PR description says the exec_mode matrix axis was repurposed to commitment_mode across 11 test workflows, including qa-rpc-performance-tests.yml, but that workflow still uses exec_mode / ERIGON_EXEC3_PARALLEL (see .github/workflows/qa-rpc-performance-tests.yml:14 and :225+). Either update that workflow too, or adjust the PR description/scope so coverage expectations match reality.

      commitment_mode:
        description: 'Erigon commitment mode ("default" leaves ERIGON_COMMITMENT_PARALLEL unset)'

[yperbasis]

Blocked by Issue #22881

[taratorio]

@mh0lt fyi, once #22885 merges to main you will need to git pull and adapt the new shards it adds to the new commitment-serial/parallel variants

will be addressed by #22885

[taratorio]

Follow-up summary for the last three non-merge implementation commits:

• 9c4d5b2 pairs the legacy consensus, Constantinople, and Cancun race partitions into serial and parallel commitment shards while preserving their worker counts, failure budgets, fixture counts, filters, and race coverage.
• 163d91e restores serial commitment coverage alongside parallel coverage for Glamsterdam Hive, Kurtosis, and EEST devnet tests. It makes commitment-parallel explicit for every EEST shard, forwards that input as ERIGON_COMMITMENT_PARALLEL, and wires SharedDomains benchmarks to the DB so the parallel benchmark leg selects the real parallel trie.
• a060423 pairs the stable, devnet, and legacy statetest suites into serial and parallel commitment shards. The statetest harness now calls EnableParaTrieDB before pre-state writes, and a regression test verifies that both requested trie variants are activated.

Validation performed locally:

• make lint passed repeatedly.
• make erigon integration passed.
• The affected SharedDomains benchmarks passed with serial and parallel commitment selected.
• The real evm statetest CLI test passed in both commitment modes, and the full cmd/evm package passed.
• EEST shard coverage, manifest validation, workflow syntax, and shell checks passed.

The complete stable, devnet, and legacy EEST fixture corpora were not run locally; CI will exercise those matrix legs.

[mh0lt]

Merged current main and resolved the conflicts (pushed ac91e1383f; build + make lint clean, YAML/bash validated). @taratorio flagging the adaptation for your review since it's in your area.

Adapted main's #22885 new devnet shards to the serial/parallel commitment convention:

• test-hive-eest.yml — main's 5 new devnet consume-enginex shards (paris+shanghai-devnet, cancun-devnet, prague-devnet, osaka-devnet, glamsterdam-devnet) are now each a commitment_mode: serial + parallel pair. Every shard in the matrix runs twice (20 entries, verified one serial + one parallel each, no dupes).
• eest-spec-shards.yml — paired main's new enginextests-devnet into enginextests-devnet-{sequential,parallel} (commitment-parallel false/true). Took main's reduced budgets on blocktests-legacy-constantinople-sequential and -cancun-sequential (24/26 → 0) and dropped the now-stale PoW comments, keeping the branch's commitment-parallel: false keys.
• run-eest-spec-test.sh — doc updated to the commitment convention; routing confirmed (the -sequential/-parallel suffix strips to enginextests-devnet, which the existing case arm handles).
• Both SKILL.md — target lists updated to the commitment-mode convention and the now-shipping enginextests-devnet variant.

One semantic call to sanity-check: the merge-base had glamsterdam-devnet as sim: consume-engine (BAL-EIP sim-limit); main #22885 changed it to sim: consume-enginex (fork_(Amsterdam|BPO2ToAmsterdam), still --experimental.bal), while this branch had kept the consume-engine form and paired it. I took main's consume-enginex definition (authoritative) and paired it serial/parallel, dropping the branch's superseded consume-engine BAL variant. If you intended to keep the dedicated consume-engine BAL leg in addition, let me know and I'll add it back under a distinct shard name.

Validated: no conflict markers, all touched YAML parses, run-eest-spec-test.sh -n OK, Go packages build, every manifest shard has a boolean commitment-parallel, and the new shards propagate through the yq load-matrix. The coverage guard I couldn't run locally (needs the fixture cache) — but I didn't touch any race-shard run regex or fork partitioning, so CI's guard leg should be unaffected.