Skip to content

op-acceptance-tests: cover proofs re-execution across a sequencer reorg - #23428

Closed
einar-oplabs wants to merge 8 commits into
einar/23380-proofs-unclean-shutdownfrom
einar/23383-proofs-reorg-reexec
Closed

einar-oplabs wants to merge 8 commits into
einar/23380-proofs-unclean-shutdownfrom
einar/23383-proofs-reorg-reexec

Conversation

@einar-oplabs

Copy link
Copy Markdown
Contributor

Why

The proofs-history ExEx handles a reth reorg notification by unwinding its store to the fork point and indexing the replacement blocks from reth's trie data; a follower on the re-execution path (--proofs-history.verification-interval=1) then executes every block that follows against the unwound state, including calls into code that only a replacement block created. The only coverage was an ExEx unit test with a hand-built replacement block; the real notification sequence of a sequencer reorg, with a fork point in reth's memory or below its persisted tip, had never been run against the proof oracle. #23046 changes what happens when a replaced block fails re-execution, so this is the acceptance coverage for it.

What

op-acceptance-tests/tests/proofs_history_reorg (sequencer without proofs history as the eth_getProof reference and the node the test sequencer drives, follower re-executing every block, 1s L2 blocks, both op-reths with --engine.persistence-threshold and --engine.memory-block-buffer-target=5 passed explicitly):

  • TestProofsHistoryFollowerMatchesTheReplacementChainAfterASequencerReorg: the batcher is stopped up front so the replaced blocks never reach L1; four EOAs are funded, traffic runs past the setup and op-node's sequencer stops, which makes its head the fork point; an in-memory cell then pads the chain with empty test-sequencer blocks until the replaced blocks will leave the fork point in reth's memory by the persistence model, so the fork point does not depend on where the stop landed. The test sequencer then builds the depth blocks the reorg will replace on top of it, no faster than wall-clock (op-node rejects a gossiped payload more than 5s ahead), and the follower imports them by gossip and re-executes them; the test waits until its proofs store has flushed the last of them. The test sequencer then builds one conflicting block on the fork point, the batcher restarts and op-node resumes sequencing on top of it. The follower must adopt the replacement chain (ReorgExact on the first replaced height), survive (no panicked line), catch up, serve the replacement block with its transactions, and after ten more blocks of traffic VerifyHistoricalProofsMatch sweeps every block from the fork point to the last traffic block for the four EOAs and, in the deployed-contract workload, the recorder's two slots; that workload also requires the recorder's slot 0 on the follower to hold a block number above the fork point. Finally the follower's op-reth log must show the ExEx reorg notification and the proofs engine unwinding stored blocks, and must show reth's "on-disk reorg detected" line exactly in the on-disk cells.
  • Workloads: deployed_contract (the first replaced block deploys the NUMBER/TIMESTAMP recorder and the replaced blocks after it call it; the replacement block carries the same deployment transaction, so the contract exists at the same address, plus a call into it; the traffic after the reorg keeps calling it) and plain_transfers.
  • Depths per threshold: 2 (fork point among the blocks reth never persisted), 6 (deeper than the memory block buffer target, fork point still in memory; at threshold 7 reth holds 5 to 7 blocks, so up to two padding blocks are needed), and 12 at threshold 7 / 55 at threshold 50 (below reth's persisted tip: on-disk reorg). blocksInRethMemory models reth's persistence schedule and matrix_test.go pins it; the assertion on reth's own "on-disk reorg detected" line, present in exactly the on-disk cells, catches a wrong model instead of silently testing the other path. Note the issue's "threshold + 5" bound for the on-disk depth is conservative: reth holds at most threshold blocks (it persists once the head is more than the threshold above the persisted tip, down to the head minus the buffer target).

Matrix, as table-driven subtests: threshold {7, 50} x masking {0, 30} x workload {deployed_contract, plain_transfers} x depth {in memory shallow, in memory deep, on disk}. The masking cells skip naming ethereum-optimism/protocol-team#457. Under DEVSTACK_L2CL_KIND=kona-node every cell skips: the test sequencer commits and publishes blocks through op-node's opstack_commitBlockV1/opstack_publishBlockV1 RPCs, which kona-node does not serve.

op-devstack seams:

  • Log recorder: logpipe.LogRecord (entries whose message, or raw text for a line that is not structured JSON such as a panic hook's output, contains a substring) and logpipe.LogRecords (fan-out), stack.LogRecorder, sysgo.OpReth.RecordLogLines(substring) recording every stdout and stderr line at every level across restarts, forwarded by the preset l2ELFrontend, and dsl.L2ELNode.RecordLogLines(substring), which fails for a node whose log the devstack does not expose. logpipe.StructuredRustLogEntry.UnparsedLine() exposes the raw text of an unstructured line.
  • dsl.EOA.PlanBlockRecorderDeployment(): the recorder deployment as a planned transaction, so a test can keep its signed bytes; DeployBlockRecorder uses it.
  • dsl.TestSequencer.SequenceBlockWithTxs closes out the job with Next after a successful publish (every step is already done, so Next only resets), which lets consecutive blocks be sequenced on one chain; before, the second New failed with block already sealed. No existing caller sequenced two blocks on the same chain. A block that was not published still leaves the job sealed, as before, now with a warning.
  • dsl.TestSequencer.SequenceBlockWithTxsWhenDue(t, network, parent, rawTxs): builds the block no earlier than its timestamp (fixed by its height) so op-node's gossip validation accepts it on publish and on receipt. The wait is a wall-clock wait inside the DSL, since the chain stands still while the test sequencer drives it.

Deleted rust/op-reth/tests/proofs/reorg/reorg_test.go: dormant (CI runs proofs/core only), it used sleeps and tested the index path; its reorg coverage is superseded here. Unit tests: TestLogRecordKeepsMatchingEntriesInOrder, TestLogRecordMatchesTheRawTextOfUnstructuredLines, TestLogRecordEntriesIsACopy, TestLogRecordsFanOutFromRegistration, TestBlocksInRethMemory, TestInMemoryDepthsFitRethMemoryWithinOnePersistenceCycle, TestOnDiskDepthsExceedRethMemory.

Findings

  • How a CL-synced follower adopts an unsafe reorg: op-node drops every gossiped block that does not extend its unsafe chain (skipping unsafe payload, since it is older than unsafe head / does not build onto the existing unsafe chain), so the replacement blocks below its head never get in through gossip. Once a gossiped block above its head arrives, checkForGapInUnsafeQueue hands it to reth with engine_newPayload; reth fetches the missing replacement blocks from its EL peer (Block added to fork chain) and makes the forkchoice target canonical once they connect. The follower switched 2s after op-node resumed in the in-memory cells and 4s in the on-disk ones; L1 derivation was not involved. The batcher still has to be off while the replaced blocks are built, or they become safe and the fork is impossible.
  • The bump's Block left reth's chain during proofs execution line (and develop's Parent state not available in reth; skipping execute_block skip) only fires when the ExEx is mid-execution of a block of the old chain at the instant reth reorgs. With a real reorg the ExEx has long finished the old chain by then (it executes a block in about a millisecond), so neither line appeared in any cell (parent_state_missing_skips=0 left_chain_handoffs=0), and neither can be asserted deterministically without a seam that delays ExEx execution. The test records both counts and logs them; the issue's expectation that a deeper reorg makes the line appear does not hold.
  • The reorg notification carries the whole replacement chain in one go: reorg_depth=4 common_ancestor=20 for a depth-2 reorg, since reth connects the fetched blocks and the block that followed before it switches.
  • On develop every cell matches the oracle, including the on-disk reorgs at both thresholds (reth rebuilds the replaced persisted blocks with overlay-computed trie updates for the notification, and the ExEx's fork-block check holds).

Local verification

Debug op-reth built from this branch, DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> go test -count=1 -timeout 60m -parallel 2 -v -run TestProofsHistoryFollowerMatchesTheReplacementChainAfterASequencerReorg ./op-acceptance-tests/tests/proofs_history_reorg/:

--- PASS: TestProofsHistoryFollowerMatchesTheReplacementChainAfterASequencerReorg
    threshold_7-masking_0-deployed_contract-depth_2_in_memory    (57.10s)  fork point 20, sweep 20..46, 6 accounts
    threshold_7-masking_0-plain_transfers-depth_2_in_memory      (55.88s)  fork point 19, sweep 19..45
    threshold_7-masking_0-deployed_contract-depth_6_in_memory    (48.69s)  fork point 19, sweep 19..41
    threshold_7-masking_0-plain_transfers-depth_6_in_memory      (48.91s)  fork point 19, sweep 19..41
    threshold_7-masking_0-deployed_contract-depth_12_on_disk     (61.13s)  fork point 19, sweep 19..53, on-disk reorg logged
    threshold_7-masking_0-plain_transfers-depth_12_on_disk       (65.72s)  fork point 20, sweep 20..56, on-disk reorg logged
    threshold_50-masking_0-deployed_contract-depth_2_in_memory   (57.10s)  fork point 20, sweep 20..46
    threshold_50-masking_0-plain_transfers-depth_2_in_memory     (56.94s)  fork point 20, sweep 20..46
    threshold_50-masking_0-deployed_contract-depth_6_in_memory   (48.77s)  fork point 19, sweep 19..41
    threshold_50-masking_0-plain_transfers-depth_6_in_memory     (48.97s)  fork point 19, sweep 19..41
    threshold_50-masking_0-deployed_contract-depth_55_on_disk   (108.42s)  fork point 19, sweep 19..98, on-disk reorg logged
    threshold_50-masking_0-plain_transfers-depth_55_on_disk     (108.06s)  fork point 19, sweep 19..98, on-disk reorg logged
    12 masking_30 cells skipped (protocol-team#457)
ok      op-acceptance-tests/tests/proofs_history_reorg  386.880s

Every cell logged reorg_notified=1 reorg_handled=1, the on-disk cells on_disk_reorg=1 and the others 0; no cell needed padding blocks in this run (the depth-6 cells at threshold 7 stopped on heights with seven blocks in memory). The same matrix passed once more before the review changes (401.965s).

go vet ./op-devstack/... ./op-acceptance-tests/tests/proofs_history_reorg/ ./op-service/logpipe/      # ok
go test -count=1 ./op-service/logpipe/ ./op-acceptance-tests/tests/proofs_history_reorg/ -run 'TestLogRecord|TestBlocksInRethMemory|TestInMemoryDepths|TestOnDiskDepths'   # ok
./linter/bin/op-golangci-lint run ./op-devstack/... ./op-service/logpipe/... ./op-acceptance-tests/tests/proofs_history_reorg/...   # ok
go test -count=1 ./op-devstack/sysgo/ -run 'TestProofsHistoryNodeArgs|TestOpRethWith|TestSubProcess|TestL2CLVerifierSyncMode'   # ok

CI budget: the twelve cells sum to about 13 minutes back to back, so the package stays under 20 minutes with -parallel 1.

Not run: the kona-node variant (every cell skips under it, see above).

Skipped cells

  • All masking_30 cells: ethereum-optimism/protocol-team#457.
  • Every cell under DEVSTACK_L2CL_KIND=kona-node: the test sequencer's commit/publish RPCs are op-node only.

Stacked on #23421 and #23418.

Closes #23383

🤖 Generated with Claude Code

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 29, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
einar-oplabs and others added 6 commits November 9, 2026 20:34
Add L2ELNode.VerifyHistoricalProofsMatch, which sweeps the blocks a
proofs-history store serves and, at every sampled block, verifies
eth_getProof for a configured account set against the header's state
root and against an independent reference node, requiring the
L2ToL1MessagePasser storage root to equal the header's withdrawalsRoot.
VerifyProofsMatchAt is the single-block form; ProofsSyncStatus and
WaitForProofsStoreBlock expose debug_proofsSyncStatus.

Move NormalizeProofResponse and VerifyProof out of the op-reth proofs
e2e utils into the DSL (as VerifyAccountProof) and switch that suite to
the DSL methods.

Add OpRethConfig.ProofsHistoryWindow with OpRethWithProofsHistoryWindow
so a test can shrink a node's proofs window, and OpRethWithEthProofWindow
so a node without proofs history can serve historical eth_getProof.

Add the proofs_history_match acceptance test, which runs transfers and
contract writes and sweeps a proofs follower against a sequencer without
proofs history over the persistence-threshold x proofs-path matrix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 42fd696)
(cherry picked from commit 021f7115a6dfd9ccb7b624bcf14a7c0df7e13492)
op-reth rejects a null storage key list where op-geth accepts it, so
awaitProof always sends a list. Retries are logged at warn level so a
failing proof lookup is diagnosable from the test log, and the
acceptance test pins the sweep's last block to the last workload block
so the whole workload is covered and the catch-up wait is exercised.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 9c4ce60)
(cherry picked from commit ed29c772af8efc2f2f4a9675b57687d14b7591e5)
Re-read the proofs window at every swept block and skip blocks the
pruner removed meanwhile when the caller did not pin FirstBlock, so a
small proofs window does not race the sweep. Gate the withdrawalsRoot
check on a non-empty header value, since pre-Isthmus headers carry the
empty withdrawals hash. Reject a nonzero storage value claimed under an
empty storage root. Require debug_proofsSyncStatus to be unknown on the
reference rather than merely failing. Drop the storage-key padding the
client already enforces.

The acceptance test adds a withdrawal per round so the message passer's
storage root changes inside the swept range, and waits for reth to pass
its persistence threshold before the last round so the threshold-50
cells persist at least once. The proofs e2e suite drops the waits that
VerifyProofsMatchAt performs itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 25e9f7e)
(cherry picked from commit 36688d25fe3f86682b1243f3983e0f42daeaa543)
…o the mixed runtime

SubProcess.Kill sends SIGKILL without an interrupt and waits a bounded time for the exit, so a
test can end an op-reth process the way a crash or a power loss would, losing whatever it had not
written to its datadir. OpReth.StopUngracefully exposes it, stack.UngracefulLifecycle lets the
preset frontend forward it, and L2ELNode.StopUngracefully is the DSL method. Stop stays graceful.

MixedSingleChainNodeSpec gains L2CLOpts, per-node op-node options passed to startL2CLNode, and
L2CLVerifierSyncMode is the option a follower needs to EL-sync; the mixed runtime passed no CL
options before, so every follower ran CL sync.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit b9386d0)
(cherry picked from commit 13cc581abae9ed7047c503af0c162250c90853b3)
Two tests over threshold {7, 50} x proofs path {index, execute} x restart sync mode {CL, EL}:
a proofs-history follower is killed once its store has flushed blocks reth has not persisted,
restarted, and must catch up and serve proofs matching the sequencer's across the gap; in the
second test the sequencer's op-reth is killed too, so the lost heights are rebuilt with different
contents and the follower must serve the rebuilt chain's proofs rather than its store's. The
second test fails on develop with the ParentHashMismatch crash of
ethereum-optimism/protocol-team#464 in every cell and is skipped naming it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 0462ddc)
(cherry picked from commit 3053fcf35f4ae1819ac4f8e78db10518dc1769c1)
…wn tests

Kill reaps a process that died on its own with a warning instead of reporting a kill, so a crash
before the kill stays visible in the logs. The rebuild test kills both op-reths back to back
before stopping the CLs, so the sequencer's datadir tip cannot catch up with the follower's store
tip while the follower's CL shuts down, re-reads the store tip right before the kill, and requires
the workload's funding block to survive the sequencer's crash. The NUMBER/TIMESTAMP recorder
contract moves into the DSL as EOA.DeployBlockRecorder, shared by both proofs-history packages.
Comments no longer claim SIGKILL models a power loss.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit c474193)
(cherry picked from commit b96b619824ac56c77b80ad963dfa9d208d1f20d4)
einar-oplabs and others added 2 commits November 9, 2026 20:39
…quencer

Add a log recorder a test can register on an op-reth node: logpipe.LogRecord
keeps every entry whose message, or raw text for an unstructured line, contains
a substring, sysgo.OpReth feeds it from both pipes across restarts, and the
preset frontend and dsl.L2ELNode expose it as RecordLogLines.

dsl.EOA.PlanBlockRecorderDeployment returns the recorder deployment as a planned
transaction so a test can keep its signed bytes.

dsl.TestSequencer.SequenceBlockWithTxs closes out the job with Next after a
successful publish, so consecutive blocks can be sequenced on one chain, and
SequenceBlockWithTxsWhenDue builds a block no earlier than its timestamp, which
op-node's gossip validation needs on publish and on receipt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit af71eab)
(cherry picked from commit 25c2917ba75d2d605b262e9191cd256f815a6c8f)
A follower re-executing every block into its proofs store watches op-node's
sequencer stop at a fork point, the test sequencer build the blocks the reorg
replaces (a recorder deployment and calls into it, or transfers) and then one
conflicting block, and op-node resume on top of it. The follower must adopt the
replacement chain, keep executing and serve proofs from the fork point on that
match the sequencer's historical state, over reth threshold {7, 50} x masking
{0, 30} x depth {in memory shallow, in memory deep, on disk}, with reth's own
on-disk reorg line asserted per cell.

The dormant rust/op-reth/tests/proofs/reorg test, which CI never ran, is
superseded and removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit b591b62)
(cherry picked from commit aa393bd9f3396df8df088e07f6ba4bb382a2da2b)
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: acceptance test for proofs re-execution during a sequencer reorg

3 participants