Skip to content

op-acceptance-tests,op-devstack,op-service: check the Isthmus withdrawals root across persistence runs - #23438

Closed
einar-oplabs wants to merge 1 commit into
einar/23397-sequencer-persistence-livenessfrom
einar/23398-withdrawals-root-persistence
Closed

einar-oplabs wants to merge 1 commit into
einar/23397-sequencer-persistence-livenessfrom
einar/23398-withdrawals-root-persistence

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Since Isthmus withdrawalsRoot is the message passer's storage root, computed through overlay state providers that reth 2.6 rewrites, and no test checked it with persistence engaged.

Initiates withdrawals in every block for twice reth's in-memory window under thresholds 7 and 50, and requires every block's withdrawalsRoot to equal the message passer storage root proven by the sequencer, an op-reth follower, an op-geth follower and a proofs-history follower: once with sequencing stopped while the newest blocks are still in reth's memory, and again once every op-reth node reports (via reth_subscribePersistedBlock) that it persisted the run.

Stacked on #23436. Closes #23398. Local verification, findings and notes: detail comment.

Breakdown

Added Removed Net
Implementation 401 13 +388
Tests 659 0 +659
Docs 0 0 0

AI Reviews

  • <ISODATETIME> <COMMITID> Fable 5.1 xhigh
  • <ISODATETIME> <COMMITID> GPT-6.1-Sol xhigh
  • <ISODATETIME> <COMMITID> Opus 5.5 xhigh

🤖 Generated with Claude Code

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 29, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from a307188 to ae5d1e9 Compare November 9, 2026 21:31
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 530218f to 458fd57 Compare November 9, 2026 21:32
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from ae5d1e9 to a43a3c9 Compare November 10, 2026 13:38
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 458fd57 to 2fc40d6 Compare November 10, 2026 13:38
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Claude: Detail moved out of the PR description when the branch was squashed into 2fc40d6. It is the description as it stood before the squash, unedited, so it describes the earlier commits.


Since Isthmus a block's withdrawalsRoot is the L2ToL1MessagePasser storage root, computed through reth's overlay state providers, which the reth 2.6 bump (#23046) rewrites while it moves the in-memory window from about 12 blocks to about 55. docs/ai/reth-update-review.md (precondition 6) asks for storage roots to be tested with persistence engaged, and the only acceptance test (isthmus/withdrawal_root) does one withdrawal on a chain that never persists. This adds the test the review doc needs: withdrawals in every block for longer than a persistence cycle, with every node's header and eth_getProof checked at every block.

What the test covers

op-acceptance-tests/tests/op_reth_withdrawals_root_persistence, TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns:

  • Mixed runtime at a 1s block time: an op-reth sequencer without proofs history, an op-reth reference-follower without proofs history, an op-reth proofs-follower running proofs history, and an op-geth geth-follower. Every op-reth node gets --engine.persistence-threshold and --engine.memory-block-buffer-target=5 explicitly; the two op-reth nodes without proofs history get --rpc.eth-proof-window so they answer eth_getProof from reth's own historical state.
  • Workload: four funded accounts each initiate a withdrawal (a transfer to L2ToL1MessagePasser) in every block for 2 * (threshold + 5) blocks, 24 at threshold 7 and 110 at threshold 50. A round submits one withdrawal per account back to back and waits for the next block, not for its receipts, so the next round is in the pool while the block after that is built. The test starts the workload only once the chain is on the wall clock (CaughtUpToSchedule, bounded at 60 block times), since a chain catching up builds blocks back to back.
  • Assertions, all in the test body:
    • the run crossed a persistence boundary: the blocks from the first withdrawal to the last number more than threshold + 5, the most blocks reth holds unpersisted (the issue's "threshold plus buffer");
    • every persistence run persisted a block with a withdrawal: the longest gap without one is shorter than the persistence period (threshold + 1 - 5 blocks: 3 at threshold 7, 46 at threshold 50), and the workload reached nearly every block: at least four in five blocks of the run carry a withdrawal (every block did in every local run);
    • for every block of the run, newest first, the sequencer, the reference follower and the op-geth follower hold the same header, the same withdrawalsRoot, and each serves an eth_getProof for the message passer at the block hash that verifies against the header's state root and whose storage hash is that root (dsl.L2ELNode.VerifyWithdrawalsRootAt);
    • the proofs follower holds the reference follower's chain and serves message passer proofs that verify against the state root, equal the reference's, and equal the header's withdrawalsRoot (VerifyHistoricalProofsMatch from op-acceptance-tests,op-devstack,op-reth: add a windowed proof oracle to the L2 EL DSL #23418, bounded to the run).
  • The sweep runs twice over the same blocks: right after the run, while reth still holds the newest threshold + 5 blocks of it in memory and has persisted the rest, and again once every node, not only the sequencer, is threshold + 5 blocks past the last block of the run (persistence is per node), so every swept block has been served both from reth's in-memory overlay and from its database.

Matrix: threshold {7, 50} x masking {0, 30} as subtests (threshold_<n>-masking_<m>), CL {op-node, kona-node} through DEVSTACK_L2CL_KIND (CI runs both).

Local verification

Debug op-reth built from this branch, kona-node release binary built from develop today, -parallel 2:

cd op-acceptance-tests && cd .. && \
  DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> RUST_BINARY_PATH_KONA_NODE=<release kona-node> \
  go test -count=1 -timeout 40m -parallel 2 -run 'TestWithdrawalsRootMatches' -v ./op-acceptance-tests/tests/op_reth_withdrawals_root_persistence/

op-node (default):

--- PASS: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_7-masking_0 (63.68s)
--- SKIP: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_7-masking_30 (0.00s)
--- SKIP: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_50-masking_30 (0.00s)
--- PASS: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_50-masking_0 (194.35s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_withdrawals_root_persistence   195.864s

Threshold 7: 96 withdrawals in blocks 22–45, 24 of 24 blocks with a withdrawal, longest gap 0. Threshold 50: 440 withdrawals in blocks 22–131, 110 of 110 blocks with a withdrawal, longest gap 0. Both sweeps (right after the run, and once every node was threshold + 5 blocks past the run's last block) passed on all four nodes in both cells. A sweep itself takes about 0.1 s per 24 blocks and 1 s per 110 blocks; the phases are separated by the wait for the chain, not by the checks.

kona-node (DEVSTACK_L2CL_KIND=kona-node, same command):

--- PASS: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_7-masking_0 (69.89s)
--- SKIP: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_7-masking_30 (0.00s)
--- SKIP: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_50-masking_30 (0.00s)
--- PASS: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_50-masking_0 (199.67s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_withdrawals_root_persistence   200.716s

Threshold 7: blocks 26–49, 24 of 24 with a withdrawal. Threshold 50: blocks 26–135, 110 of 110 with a withdrawal. Longest gap 0 in both.

Against #23046's op-reth (reth 2.6.0, the debug binary built from the bump head 93d9ec9 for #23436; op-node), same command: both running cells pass, so this check holds on the bump as well:

--- SKIP: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_7-masking_30 (0.00s)
--- SKIP: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_50-masking_30 (0.00s)
--- PASS: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_7-masking_0 (64.15s)
--- PASS: TestWithdrawalsRootMatchesMessagePasserStorageRootAcrossPersistenceRuns/threshold_50-masking_0 (194.96s)
ok      github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/op_reth_withdrawals_root_persistence   196.034s

Threshold 7: blocks 22–45, 24 of 24 with a withdrawal. Threshold 50: blocks 22–131, 110 of 110 with a withdrawal. Longest gap 0 in both. The kona-node cells were not run against the bump.

Unit test of the run summary (go test -run TestSummarizeRun ./op-acceptance-tests/tests/op_reth_withdrawals_root_persistence/): ok. go vet, the custom golangci lint on op-devstack/dsl and the new package, and semgrep on both files: clean.

Skipped cells

  • masking_30 (both thresholds): state masking is compiled out of reth 2.5.2 on develop and rust!: update reth to v2.6.0 #23046 rejects a nonzero --engine.num-state-masking-blocks; the skip names ethereum-optimism/protocol-team#457.
  • The whole test skips on DEVSTACK_L2EL_KIND=op-geth: the withdrawals root is checked across op-reth's persistence runs.

Findings

  • No mismatch on develop: every block's withdrawalsRoot equals the message passer storage hash from eth_getProof on all four nodes, from reth's in-memory overlay and from its database, under both thresholds and both consensus clients.
  • Serving eth_getProof for persisted blocks through --rpc.eth-proof-window makes reth log, per request, Attempt to calculate state root for an old block might result in OOM and Changeset cache MISS in range, falling back to aggregate DB-based computation (WARN, on the sequencer and the reference follower). Expected for the historical-state path; not an error. On the bump binary only the changeset-cache warning remains; reth 2.6 no longer logs the old-block one.
  • The proofs-history store trails the head by a few blocks while the sweep runs; VerifyHistoricalProofsMatch's store wait absorbed it (29 "Proofs store behind target" polls across the op-node run, none timing out).

Seams added

  • dsl.L2ELNode.VerifyWithdrawalsRootAt(block) eth.BlockInfo (op-devstack/dsl/l2_el_proofs.go): waits for the header of block, fetches the L2ToL1MessagePasser proof at the block hash, requires it to verify against the header's state root and its storage hash to equal the header's withdrawalsRoot, and returns the header. Works on any node that answers eth_getProof for the block (reth's own state behind --rpc.eth-proof-window, op-geth's archive state, or a proofs-history store). The withdrawals-root rule it shares with VerifyHistoricalProofsMatch now lives in one helper.
  • dsl.L2ELNode.CaughtUpToSchedule(blockTime) eth.L2BlockRef (op-devstack/dsl/l2_el_schedule.go): exports op-acceptance-tests,op-devstack,op-reth: hold the sequencer to its block schedule across persistence runs #23436's bounded wait for a chain that started behind the wall clock to catch up with it, for tests whose workload paces itself by the chain's blocks.

Notes on the issue text

The issue lists three nodes (sequencer, op-reth follower without proofs history, op-geth follower); the test also runs an op-reth follower with proofs history, so the proofs-history store is covered against an independent op-reth reference across the same persistence runs, and it also checks eth_getProof on the sequencer itself, since the sequencer is the node that computed the roots while building.

Stacked on #23436, #23431, #23427, #23421 and #23418.

Closes #23398

@einar-oplabs
einar-oplabs requested a review from aakoshh November 10, 2026 14:07
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 2fc40d6 to f4c7010 Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from a43a3c9 to b7ab15e Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from f4c7010 to be963b4 Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch 2 times, most recently from 47f5a25 to d9a8102 Compare November 11, 2026 13:59
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from be963b4 to 5b3e133 Compare November 11, 2026 13:59
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from d9a8102 to 55f06d3 Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 5b3e133 to 63096fc Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from 55f06d3 to 924225b Compare November 11, 2026 15:28
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 63096fc to 5fe7487 Compare November 11, 2026 15:28
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from 924225b to a8c5f35 Compare November 11, 2026 16:28
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 5fe7487 to 4c4f1fb Compare November 11, 2026 16:28
@sebastianst

Copy link
Copy Markdown
Member

🤖 Fable says: A note from the op-geth decoupling work (#20257). The new test here starts an op-geth follower (ELKind: sysgo.MixedL2ELOpGeth).

op-geth is EOL as of Karst, and #21744 removes the in-process op-geth L2 EL from op-devstack/sysgo (l2_el_opgeth.go, MixedL2ELOpGeth, DEVSTACK_L2EL_KIND). It cannot exist after the go.mod flip to upstream go-ethereum (#20266): the same code would silently run upstream geth as the L2 EL. So please don't add new test or CI dependencies on op-geth; they would have to be removed again before the flip.

Can the follower be op-reth instead?

…wals root across persistence runs

Adds an acceptance test that initiates withdrawals from four accounts in
every block for twice as many blocks as reth holds in memory, under
thresholds 7 and 50, and requires every block of the run to carry as its
withdrawalsRoot the L2ToL1MessagePasser storage root that eth_getProof
serves on the op-reth sequencer and an op-geth follower, with a
proofs-history follower swept against an op-reth follower reading its
own historical state, and every node on the sequencer's chain. The first
sweep runs with sequencing stopped within reth's memory buffer of the
run's last block, once the run's first block is persisted; the second
runs once every op-reth node reports, through
reth_subscribePersistedBlock, that it persisted the whole run. The test
asserts that the run crossed a persistence boundary, that no gap between
blocks with a withdrawal spans a whole persistence run, and that at
least four in five blocks carry one. The op-geth follower runs a genesis
without Karst, which op-geth v1.101702.4-rc.2 refuses.

Adds the DSL check that a block's withdrawalsRoot is the message
passer's storage root the node proves against the header's state root, a
tracker of reth's persisted block on top of a new client.SubscribeWS,
and L2ELNode.NonceAt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from a8c5f35 to 055c21b Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs force-pushed the einar/23398-withdrawals-root-persistence branch from 4c4f1fb to 19b45d1 Compare November 12, 2026 17:25
@einar-oplabs einar-oplabs changed the title op-acceptance-tests,op-devstack: check the Isthmus withdrawals root across persistence runs op-acceptance-tests,op-devstack,op-service: check the Isthmus withdrawals root across persistence runs Nov 12, 2026
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 13:18
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 13:18
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants