Skip to content

op-acceptance-tests,op-devstack,op-reth: add a windowed proof oracle to the L2 EL DSL - #23418

Closed
einar-oplabs wants to merge 1 commit into
developfrom
einar/23379-proofs-oracle
Closed

einar-oplabs wants to merge 1 commit into
developfrom
einar/23379-proofs-oracle

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Every proofs-store corruption so far shows up as a wrong eth_getProof for some historical block, so the proofs-history scenario tests need one shared oracle that catches it without knowing the cause.

L2ELNode.VerifyHistoricalProofsMatch sweeps the blocks a proofs-history store serves and checks eth_getProof against the header's state root and a reference node, including the message passer root against withdrawalsRoot. The op-reth proofs e2e helpers move into the DSL, and proofs_history_match sweeps a follower over the persistence-threshold x proofs-path matrix.

Base of the stack ending at #23448. Closes #23379. Local verification, findings and notes: detail comment.

Breakdown

Added Removed Net
Implementation 564 0 +564
Tests 542 254 +288
Docs 0 0 0

AI Reviews

  • <ISODATETIME> <COMMITID> Fable 5.1 xhigh
  • <ISODATETIME> <COMMITID> GPT-6.1-Sol xhigh
  • <ISODATETIME> <COMMITID> Opus 5.5 xhigh

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Oct 14, 2026
This was referenced Oct 14, 2026
@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 29, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23379-proofs-oracle branch 2 times, most recently from ea1e60a to 0d7e373 Compare November 10, 2026 13:38
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Claude: Detail moved out of the PR description when the branch was squashed into 0d7e373. It is the description as it stood before the squash, unedited, so it describes the earlier commits.


Why

Every proofs-store corruption so far (wipe handling, masked-trie overlays, the withdrawalsRoot bug, the crash gap) shows up as a wrong eth_getProof for some historical block. This adds one DSL sweep that catches all of them without knowing the cause, so the proofs-history scenario tests (#23380, #23383, #23389, #23390, #23395, #23396, #23398) can share a single oracle.

What

op-devstack/dsl/l2_el_proofs.go

  • L2ELNode.VerifyHistoricalProofsMatch(reference, opts...): reads the node's proofs window from debug_proofsSyncStatus, waits (bounded) for the store to reach the sweep's last block, then for every sampled block fetches eth_getProof by block hash for L2ToL1MessagePasser plus the configured accounts from the node under test and from reference, Merkle-verifies both against the header's state root, requires them equal after normalization, and requires the message passer's storage root to equal the header's withdrawalsRoot (post-Isthmus, i.e. whenever the header value is not the empty withdrawals hash). Headers are fetched from both nodes and must agree on the hash. The window is re-read at every block: a block the pruner removes during the sweep is skipped when the caller did not pin FirstBlock, and fails the sweep otherwise. The reference must answer debug_proofsSyncStatus with method-not-found, which only a node without a proofs-history store does.
  • HistoricalProofsOpts: Accounts []ProofAccount (address + storage keys), CatchUpTimeout (default 30s), FirstBlock/LastBlock (zero = window edge at sweep start), MaxSampledBlocks (zero = every block; edges always included).
  • VerifyProofsMatchAt(reference, block, accounts...): single-block form (listed accounts only, no independence check); ProofsSyncStatus() and WaitForProofsStoreBlock(target, timeout) expose the sync status RPC.
  • NormalizeProofResponse and VerifyAccountProof moved here from rust/op-reth/tests/proofs/utils/proof.go. The normalizer additionally turns an empty storage proof list into nil so geth and reth responses compare equal; the verifier additionally rejects a nonzero storage value claimed under an empty storage root. utils/proof.go and utils.WaitForProofsStoreBlock are deleted; proofs/core, proofs/prune and proofs/reorg call the DSL methods instead and drop the waits VerifyProofsMatchAt performs itself.

op-devstack/sysgo

op-acceptance-tests/tests/proofs_history_match

TestProofsHistoryFollowerServesMatchingHistoricalProofs: a sequencer started with OpRethWithoutProofsHistory() + OpRethWithEthProofWindow is the independent reference; a follower runs proofs history under the matrix cell's flags. Four rounds of: two EOAs transfer back and forth, one initiates a withdrawal (so the message passer's storage root changes inside the swept range), and one calls a contract that SSTOREs NUMBER and TIMESTAMP. Before the last round the test waits for the follower to pass persistence-threshold + 2 * memory-block-buffer-target blocks, so reth has persisted at least once and the sweep spans blocks on both sides of that. The follower is then swept from the window's first block to the last workload block for the message passer, both EOAs and the contract's two slots.

Matrix (table-driven subtests): persistence threshold {7, 50} x state masking {0, 30} x proofs path {index = verification-interval=0, execute = verification-interval=1}. The four masking_30 cells skip naming ethereum-optimism/protocol-team#457 (the flag does not exist on develop's reth 2.5.2 build and #23046 rejects a nonzero value).

Unit tests: VerifyAccountProof against a synthetic state trie (valid account with storage, without storage, absent account, zero slot under an empty root; tampered balance, tampered storage value, nonzero slot under an empty root, absent account claiming a balance, wrong root), NormalizeProofResponse (five cases), ProofsSyncStatus.serves (five), sampleBlockNumbers (seven), proofsHistoryNodeArgs and the two new options.

Local verification

go test -count=1 ./op-devstack/dsl/ -run 'TestVerifyAccountProof|TestNormalizeProofResponse|TestSampleBlockNumbers|TestProofsSyncStatusServes'   # ok
go test -count=1 ./op-devstack/sysgo/ -run 'TestProofsHistoryNodeArgs|TestOpRethWith'                                                         # ok

DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=.../target/debug/op-reth \
  go test -count=1 -timeout 40m -parallel 2 -v -run TestProofsHistoryFollowerServesMatchingHistoricalProofs \
  ./op-acceptance-tests/tests/proofs_history_match/
--- PASS: threshold_7-masking_0-index (58.61s)      sweep: 26 blocks x 4 accounts
--- PASS: threshold_7-masking_0-execute (56.88s)    sweep: 25 blocks x 4 accounts
--- PASS: threshold_50-masking_0-index (136.96s)    sweep: 66 blocks x 4 accounts
--- PASS: threshold_50-masking_0-execute (154.70s)  sweep: 65 blocks x 4 accounts
--- SKIP: threshold_{7,50}-masking_30-{index,execute}   (protocol-team#457)

An earlier run without LastBlock pinned showed the catch-up wait firing (Proofs store behind target ... window [0, 20]) while the sweep's last block was still buffered in the ExEx.

Proofs e2e suite after the migration (rust/op-reth/tests/proofs/core, both nodes op-reth, debug binary):

--- PASS: TestL2MultipleTransactionsInDifferentBlocks (37.23s)
--- PASS: TestL2MultipleTransactionsInSingleBlock (27.24s)
--- PASS: TestExecutePayloadSuccess (17.41s)
--- PASS: TestExecutePayloadWithInvalidParentHash (17.63s)
--- PASS: TestDebugExecutionWitness (26.82s)
--- PASS: TestResyncing (43.42s)
--- PASS: TestStorageProofUsingSimpleStorageContract (46.79s)
--- PASS: TestStorageProofUsingMultiStorageContract (67.78s)
--- PASS: TestTokenVaultStorageProofs (37.37s)
ok  	github.com/ethereum-optimism/optimism/rust/op-reth/tests/proofs/core	322.780s
EXIT=0

Not run: the kona-node CL variant (DEVSTACK_L2CL_KIND=kona-node; no kona-node binary built locally). The prune and reorg packages compile but stay dormant as before.

Corruption check (not committed)

A copy of the test started the follower with sysgo.OpRethWithProofsHistoryWindow(20), ran three times the workload so the pruner moved earliest to 25, and swept with opts.FirstBlock = 1:

Sweeping historical proofs  name=proofs-follower reference=sequencer window="[25, 45]" first=1 last=45 blocks=45 accounts=4
Error:      Should be true
Messages:   proofs store on proofs-follower does not serve block 1: window [25, 45]
--- FAIL: TestCorruptionCheckSmallerWindowBites (101.42s)

The sweep fails before any account is fetched, because the store no longer serves the block at all; a wrong proof for a served block fails one level deeper with block N account 0x..: eth_getProof from <node> does not verify against state root ... / ... differs from reference <node>, which the unit tests exercise with tampered proofs.

Findings

  • op-reth's eth_getProof rejects a null storage-key list (Invalid params: invalid type: null, expected a sequence) where op-geth accepts it. The old helpers always passed []common.Hash{}; the DSL now sends an empty list for a nil StorageKeys.
  • While that bug made every eth_getProof fail fast, three of four cells ended their 30s retry window with dial tcp 127.0.0.1:<proxy-port>: i/o timeout against the follower's user-RPC proxy, the symptom safeheaddb_elsync already skips on op-reth. It did not recur once requests stopped failing; noted in case it shows up again.
  • Deviation from the issue text: the issue assumes the devstack sequencer always runs proofs history and asks for a separate reference node. The test instead turns proofs history off on the sequencer (MixedSingleChainNodeSpec.OpRethOpts), which is the production shape and saves a third node. The verification criterion expects the smaller-window failure to name "the block and account"; the oracle fails earlier, at the window check, naming the block, the node and the window (see above).

Seams for the sibling PRs

  • L2ELNode.VerifyHistoricalProofsMatch / VerifyProofsMatchAt / WaitForProofsStoreBlock / ProofsSyncStatus.
  • sysgo.OpRethWithProofsHistoryWindow, sysgo.OpRethWithEthProofWindow (a reference node needs the latter together with OpRethWithoutProofsHistory).

Closes #23379

@einar-oplabs einar-oplabs changed the title op-devstack: add a windowed proof oracle to the L2 EL DSL op-acceptance-tests,op-devstack,op-reth: add a windowed proof oracle to the L2 EL DSL Nov 10, 2026
@einar-oplabs
einar-oplabs requested a review from aakoshh November 10, 2026 14:07
@einar-oplabs
einar-oplabs force-pushed the einar/23379-proofs-oracle branch from 0d7e373 to a74ef4c Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23379-proofs-oracle branch 3 times, most recently from 8d28fe0 to 6266926 Compare November 11, 2026 16:28
…to the L2 EL DSL

L2ELNode.VerifyHistoricalProofsMatch sweeps the blocks a proofs-history
store serves and, at each sampled block, verifies eth_getProof for a
configured account set against the header's state root and against a
reference node without proofs history, requiring the L2ToL1MessagePasser
storage root to equal a non-empty withdrawalsRoot. It re-reads the
proofs window at every block, retrying a failed read within the catch-up
budget, skips a block the store pruned before or while its proof was
fetched unless the caller pinned FirstBlock, and fails a sweep that
verified no block. Proof and header lookups retry only while the node
lags. VerifyProofsMatchAt is the single-block form; ProofsSyncStatus,
which takes the caller's context, and WaitForProofsStoreBlock expose
debug_proofsSyncStatus.

NormalizeProofResponse and VerifyProof move from the op-reth proofs e2e
utils into the DSL (as VerifyAccountProof), and that suite uses the DSL
methods. OpRethConfig gains an eth proof window option so a node without
proofs history can serve historical proofs, and
presets.NewProofsHistoryFollowerSystem builds a sequencer, a
proofs-history follower and a reference node.

The proofs_history_match acceptance test runs transfers, contract writes
and a withdrawal per round and sweeps a proofs follower against the
sequencer over the persistence-threshold x proofs-path matrix, with
reth's memory buffer target passed explicitly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23379-proofs-oracle branch from 6266926 to f2add41 Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: add a windowed proof oracle to the devstack DSL

3 participants