Repository navigation
op-acceptance-tests: cover proofs re-execution across a sequencer reorg - #23428
Closed
einar-oplabs wants to merge 8 commits into
Closed
einar-oplabs wants to merge 8 commits into
einar-oplabs wants to merge 8 commits into
Conversation
einar-oplabs
force-pushed
the
einar/23383-proofs-reorg-reexec
branch
from
October 14, 2026 17:32
b51d1de to
b591b62
Compare
This was referenced Oct 14, 2026
Contributor
|
This pr has been automatically marked as stale and will be closed in 5 days if no updates |
Add L2ELNode.VerifyHistoricalProofsMatch, which sweeps the blocks a proofs-history store serves and, at every sampled block, verifies eth_getProof for a configured account set against the header's state root and against an independent reference node, requiring the L2ToL1MessagePasser storage root to equal the header's withdrawalsRoot. VerifyProofsMatchAt is the single-block form; ProofsSyncStatus and WaitForProofsStoreBlock expose debug_proofsSyncStatus. Move NormalizeProofResponse and VerifyProof out of the op-reth proofs e2e utils into the DSL (as VerifyAccountProof) and switch that suite to the DSL methods. Add OpRethConfig.ProofsHistoryWindow with OpRethWithProofsHistoryWindow so a test can shrink a node's proofs window, and OpRethWithEthProofWindow so a node without proofs history can serve historical eth_getProof. Add the proofs_history_match acceptance test, which runs transfers and contract writes and sweeps a proofs follower against a sequencer without proofs history over the persistence-threshold x proofs-path matrix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 42fd696) (cherry picked from commit 021f7115a6dfd9ccb7b624bcf14a7c0df7e13492)
op-reth rejects a null storage key list where op-geth accepts it, so awaitProof always sends a list. Retries are logged at warn level so a failing proof lookup is diagnosable from the test log, and the acceptance test pins the sweep's last block to the last workload block so the whole workload is covered and the catch-up wait is exercised. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 9c4ce60) (cherry picked from commit ed29c772af8efc2f2f4a9675b57687d14b7591e5)
Re-read the proofs window at every swept block and skip blocks the pruner removed meanwhile when the caller did not pin FirstBlock, so a small proofs window does not race the sweep. Gate the withdrawalsRoot check on a non-empty header value, since pre-Isthmus headers carry the empty withdrawals hash. Reject a nonzero storage value claimed under an empty storage root. Require debug_proofsSyncStatus to be unknown on the reference rather than merely failing. Drop the storage-key padding the client already enforces. The acceptance test adds a withdrawal per round so the message passer's storage root changes inside the swept range, and waits for reth to pass its persistence threshold before the last round so the threshold-50 cells persist at least once. The proofs e2e suite drops the waits that VerifyProofsMatchAt performs itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 25e9f7e) (cherry picked from commit 36688d25fe3f86682b1243f3983e0f42daeaa543)
…o the mixed runtime SubProcess.Kill sends SIGKILL without an interrupt and waits a bounded time for the exit, so a test can end an op-reth process the way a crash or a power loss would, losing whatever it had not written to its datadir. OpReth.StopUngracefully exposes it, stack.UngracefulLifecycle lets the preset frontend forward it, and L2ELNode.StopUngracefully is the DSL method. Stop stays graceful. MixedSingleChainNodeSpec gains L2CLOpts, per-node op-node options passed to startL2CLNode, and L2CLVerifierSyncMode is the option a follower needs to EL-sync; the mixed runtime passed no CL options before, so every follower ran CL sync. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit b9386d0) (cherry picked from commit 13cc581abae9ed7047c503af0c162250c90853b3)
Two tests over threshold {7, 50} x proofs path {index, execute} x restart sync mode {CL, EL}:
a proofs-history follower is killed once its store has flushed blocks reth has not persisted,
restarted, and must catch up and serve proofs matching the sequencer's across the gap; in the
second test the sequencer's op-reth is killed too, so the lost heights are rebuilt with different
contents and the follower must serve the rebuilt chain's proofs rather than its store's. The
second test fails on develop with the ParentHashMismatch crash of
ethereum-optimism/protocol-team#464 in every cell and is skipped naming it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 0462ddc)
(cherry picked from commit 3053fcf35f4ae1819ac4f8e78db10518dc1769c1)
…wn tests Kill reaps a process that died on its own with a warning instead of reporting a kill, so a crash before the kill stays visible in the logs. The rebuild test kills both op-reths back to back before stopping the CLs, so the sequencer's datadir tip cannot catch up with the follower's store tip while the follower's CL shuts down, re-reads the store tip right before the kill, and requires the workload's funding block to survive the sequencer's crash. The NUMBER/TIMESTAMP recorder contract moves into the DSL as EOA.DeployBlockRecorder, shared by both proofs-history packages. Comments no longer claim SIGKILL models a power loss. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit c474193) (cherry picked from commit b96b619824ac56c77b80ad963dfa9d208d1f20d4)
…quencer Add a log recorder a test can register on an op-reth node: logpipe.LogRecord keeps every entry whose message, or raw text for an unstructured line, contains a substring, sysgo.OpReth feeds it from both pipes across restarts, and the preset frontend and dsl.L2ELNode expose it as RecordLogLines. dsl.EOA.PlanBlockRecorderDeployment returns the recorder deployment as a planned transaction so a test can keep its signed bytes. dsl.TestSequencer.SequenceBlockWithTxs closes out the job with Next after a successful publish, so consecutive blocks can be sequenced on one chain, and SequenceBlockWithTxsWhenDue builds a block no earlier than its timestamp, which op-node's gossip validation needs on publish and on receipt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit af71eab) (cherry picked from commit 25c2917ba75d2d605b262e9191cd256f815a6c8f)
A follower re-executing every block into its proofs store watches op-node's
sequencer stop at a fork point, the test sequencer build the blocks the reorg
replaces (a recorder deployment and calls into it, or transfers) and then one
conflicting block, and op-node resume on top of it. The follower must adopt the
replacement chain, keep executing and serve proofs from the fork point on that
match the sequencer's historical state, over reth threshold {7, 50} x masking
{0, 30} x depth {in memory shallow, in memory deep, on disk}, with reth's own
on-disk reorg line asserted per cell.
The dormant rust/op-reth/tests/proofs/reorg test, which CI never ran, is
superseded and removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit b591b62)
(cherry picked from commit aa393bd9f3396df8df088e07f6ba4bb382a2da2b)
einar-oplabs
force-pushed
the
einar/23383-proofs-reorg-reexec
branch
from
November 9, 2026 22:29
b591b62 to
7d1f04c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The proofs-history ExEx handles a reth reorg notification by unwinding its store to the fork point and indexing the replacement blocks from reth's trie data; a follower on the re-execution path (
--proofs-history.verification-interval=1) then executes every block that follows against the unwound state, including calls into code that only a replacement block created. The only coverage was an ExEx unit test with a hand-built replacement block; the real notification sequence of a sequencer reorg, with a fork point in reth's memory or below its persisted tip, had never been run against the proof oracle. #23046 changes what happens when a replaced block fails re-execution, so this is the acceptance coverage for it.What
op-acceptance-tests/tests/proofs_history_reorg(sequencer without proofs history as theeth_getProofreference and the node the test sequencer drives, follower re-executing every block, 1s L2 blocks, both op-reths with--engine.persistence-thresholdand--engine.memory-block-buffer-target=5passed explicitly):TestProofsHistoryFollowerMatchesTheReplacementChainAfterASequencerReorg: the batcher is stopped up front so the replaced blocks never reach L1; four EOAs are funded, traffic runs past the setup and op-node's sequencer stops, which makes its head the fork point; an in-memory cell then pads the chain with empty test-sequencer blocks until the replaced blocks will leave the fork point in reth's memory by the persistence model, so the fork point does not depend on where the stop landed. The test sequencer then builds thedepthblocks the reorg will replace on top of it, no faster than wall-clock (op-node rejects a gossiped payload more than 5s ahead), and the follower imports them by gossip and re-executes them; the test waits until its proofs store has flushed the last of them. The test sequencer then builds one conflicting block on the fork point, the batcher restarts and op-node resumes sequencing on top of it. The follower must adopt the replacement chain (ReorgExacton the first replaced height), survive (nopanickedline), catch up, serve the replacement block with its transactions, and after ten more blocks of trafficVerifyHistoricalProofsMatchsweeps every block from the fork point to the last traffic block for the four EOAs and, in the deployed-contract workload, the recorder's two slots; that workload also requires the recorder's slot 0 on the follower to hold a block number above the fork point. Finally the follower's op-reth log must show the ExEx reorg notification and the proofs engine unwinding stored blocks, and must show reth's "on-disk reorg detected" line exactly in the on-disk cells.deployed_contract(the first replaced block deploys theNUMBER/TIMESTAMPrecorder and the replaced blocks after it call it; the replacement block carries the same deployment transaction, so the contract exists at the same address, plus a call into it; the traffic after the reorg keeps calling it) andplain_transfers.blocksInRethMemorymodels reth's persistence schedule andmatrix_test.gopins it; the assertion on reth's own "on-disk reorg detected" line, present in exactly the on-disk cells, catches a wrong model instead of silently testing the other path. Note the issue's "threshold + 5" bound for the on-disk depth is conservative: reth holds at mostthresholdblocks (it persists once the head is more than the threshold above the persisted tip, down to the head minus the buffer target).Matrix, as table-driven subtests: threshold {7, 50} x masking {0, 30} x workload {deployed_contract, plain_transfers} x depth {in memory shallow, in memory deep, on disk}. The masking cells skip naming ethereum-optimism/protocol-team#457. Under
DEVSTACK_L2CL_KIND=kona-nodeevery cell skips: the test sequencer commits and publishes blocks through op-node'sopstack_commitBlockV1/opstack_publishBlockV1RPCs, which kona-node does not serve.op-devstackseams:logpipe.LogRecord(entries whose message, or raw text for a line that is not structured JSON such as a panic hook's output, contains a substring) andlogpipe.LogRecords(fan-out),stack.LogRecorder,sysgo.OpReth.RecordLogLines(substring)recording every stdout and stderr line at every level across restarts, forwarded by the presetl2ELFrontend, anddsl.L2ELNode.RecordLogLines(substring), which fails for a node whose log the devstack does not expose.logpipe.StructuredRustLogEntry.UnparsedLine()exposes the raw text of an unstructured line.dsl.EOA.PlanBlockRecorderDeployment(): the recorder deployment as a planned transaction, so a test can keep its signed bytes;DeployBlockRecorderuses it.dsl.TestSequencer.SequenceBlockWithTxscloses out the job withNextafter a successful publish (every step is already done, soNextonly resets), which lets consecutive blocks be sequenced on one chain; before, the secondNewfailed withblock already sealed. No existing caller sequenced two blocks on the same chain. A block that was not published still leaves the job sealed, as before, now with a warning.dsl.TestSequencer.SequenceBlockWithTxsWhenDue(t, network, parent, rawTxs): builds the block no earlier than its timestamp (fixed by its height) so op-node's gossip validation accepts it on publish and on receipt. The wait is a wall-clock wait inside the DSL, since the chain stands still while the test sequencer drives it.Deleted
rust/op-reth/tests/proofs/reorg/reorg_test.go: dormant (CI runsproofs/coreonly), it used sleeps and tested the index path; its reorg coverage is superseded here. Unit tests:TestLogRecordKeepsMatchingEntriesInOrder,TestLogRecordMatchesTheRawTextOfUnstructuredLines,TestLogRecordEntriesIsACopy,TestLogRecordsFanOutFromRegistration,TestBlocksInRethMemory,TestInMemoryDepthsFitRethMemoryWithinOnePersistenceCycle,TestOnDiskDepthsExceedRethMemory.Findings
skipping unsafe payload, since it is older than unsafe head/does not build onto the existing unsafe chain), so the replacement blocks below its head never get in through gossip. Once a gossiped block above its head arrives,checkForGapInUnsafeQueuehands it to reth withengine_newPayload; reth fetches the missing replacement blocks from its EL peer (Block added to fork chain) and makes the forkchoice target canonical once they connect. The follower switched 2s after op-node resumed in the in-memory cells and 4s in the on-disk ones; L1 derivation was not involved. The batcher still has to be off while the replaced blocks are built, or they become safe and the fork is impossible.Block left reth's chain during proofs executionline (and develop'sParent state not available in reth; skipping execute_blockskip) only fires when the ExEx is mid-execution of a block of the old chain at the instant reth reorgs. With a real reorg the ExEx has long finished the old chain by then (it executes a block in about a millisecond), so neither line appeared in any cell (parent_state_missing_skips=0 left_chain_handoffs=0), and neither can be asserted deterministically without a seam that delays ExEx execution. The test records both counts and logs them; the issue's expectation that a deeper reorg makes the line appear does not hold.reorg_depth=4 common_ancestor=20for a depth-2 reorg, since reth connects the fetched blocks and the block that followed before it switches.Local verification
Debug op-reth built from this branch,
DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> go test -count=1 -timeout 60m -parallel 2 -v -run TestProofsHistoryFollowerMatchesTheReplacementChainAfterASequencerReorg ./op-acceptance-tests/tests/proofs_history_reorg/:Every cell logged
reorg_notified=1 reorg_handled=1, the on-disk cellson_disk_reorg=1and the others0; no cell needed padding blocks in this run (the depth-6 cells at threshold 7 stopped on heights with seven blocks in memory). The same matrix passed once more before the review changes (401.965s).CI budget: the twelve cells sum to about 13 minutes back to back, so the package stays under 20 minutes with
-parallel 1.Not run: the kona-node variant (every cell skips under it, see above).
Skipped cells
masking_30cells: ethereum-optimism/protocol-team#457.DEVSTACK_L2CL_KIND=kona-node: the test sequencer's commit/publish RPCs are op-node only.Stacked on #23421 and #23418.
Closes #23383
🤖 Generated with Claude Code