Skip to content

op-reth: test the proofs RPC window edges and old-block witness requests - #23434

Closed
einar-oplabs wants to merge 6 commits into
einar/23389-proofs-init-pathfrom
einar/23390-proofs-rpc-edges
Closed

einar-oplabs wants to merge 6 commits into
einar/23389-proofs-init-pathfrom
einar/23390-proofs-rpc-edges

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

The historical-proofs RPC handler (rust/op-reth/crates/rpc/src/eth/proofs.rs) and its state-provider factory (state.rs) had no tests, and the only e2e check near the window's earliest block was the dormant proofs/prune package. The reth 2.6 bump (#23046) rewrites the handler, so this pins what eth_getProof, debug_executePayload and debug_executionWitness answer at and around the window's edges before the rewrite lands.

What the tests cover

Rust: OpStateProviderFactory::state_provider (rust/op-reth/crates/rpc/src/state.rs, 16 cases)

A mock eth API (reth_rpc::EthApi over MockEthProvider) with a canonical chain of 21 blocks and an InMemoryProofsStorage seeded with blocks 10..=18, so the store trails the head the way it does between flushes:

  • served: the earliest block (by number and by hash), a block inside the window, the latest stored block (by number and by hash)
  • refused with StateForNumberNotFound(n): one below the earliest (by number and by hash of the canonical block), genesis, one above the store tip, the chain head the store has not flushed, the latest tag while the store trails
  • served once the store holds the head: the latest tag, no block id, the head by number
  • an unknown hash is ProviderError::Other("header not found") (the wrapping that ethereum-optimism/protocol-team#474 reports as surfacing to clients as an internal error)
  • an empty store refuses everything

Not covered in Rust: a hash that names a known non-canonical block. MockEthProvider resolves block_number(hash) and block_hash(number) from the same header map, so a non-canonical block cannot be modelled; the e2e reorg check below covers that branch.

The test build enables reth-optimism-trie/metrics so OpProofsStorage<S> is the production wrapper, as it is for the node and for CI's --workspace --all-features clippy and nextest runs; without it the alias is the bare S and clippy flags the conversion the node needs.

e2e, in rust/op-reth/tests/proofs/core (CI job op-reth-e2e-sysgo-tests, just test-e2e-sysgo with the default GO_PKG_NAME=proofs/core)

TestProofsFollowerServesExactlyTheWindowItReports, matrix threshold {7, 50} x proofs path {index, execute}, each cell a sequencer without proofs history (the independent eth_getProof reference, --rpc.eth-proof-window) and a follower with a 20-block window, --rpc.max-tracing-requests=2, 1s blocks. The chain is driven past the window until pruning ran, then sequencing is stopped so the window stands still, and the cell checks:

  • the earliest block is served by number and by hash, verifies against the header and matches the reference; the chain is driven a window past the workload's setup block first, so the earliest block holds the workload's accounts
  • one below the earliest is refused by number and by hash with an error naming the block (develop: no state found for block number N)
  • the latest block is served by number, by hash and by the latest tag and equals the reference at the head
  • eth_getProof latency at the earliest block versus the latest (medians of 10), logged for the bump comparison and bounded at 10x with a 250ms floor so only a lookup that re-walks the window can trip it
  • debug_executePayload builds on the earliest block
  • debug_executionWitness refuses the earliest block (it executes on the parent's state, which pruning removed) and serves the block after it
  • 8 concurrent debug_executionWitness requests, above the 2 tracing permits and above the handler's own 3, all succeed
  • after sequencing resumes, the first new height is refused until debug_proofsSyncStatus reports it and never answered with a proof of another block, then matches the reference; the subtest stops the chain again, so every subtest starts from a stopped chain and can run alone
  • a block the test sequencer builds on the head and then replaces: its hash is refused once the follower reorged, the replacement is served by hash and matches the reference

TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint, threshold {7, 50}, index path: the follower's reth prunes account and storage history 5 blocks behind its persisted tip (through a --config file, since reth's unwind-safe minimum of 10064 blocks has to be lowered in the file and --prune.* flags replace the file's prune section), with a proofs window that reaches below that checkpoint. Once eth_getStorageAt at the window's earliest block reports reth's own state as pruned, eth_getProof there must verify and match the reference and debug_executionWitness for the next block must execute: the store serves the window, not reth's history. This is the "block below reth's history prune checkpoint" cell from the issue; the issue's --full flag cannot be used because it pins the history distance to 10064 blocks.

The dormant proofs/prune package is folded into these checks (window read from debug_proofsSyncStatus and asserted equal to the configured window after pruning, pruned block refused, new earliest served) and deleted.

Matrix

test threshold proofs path state masking
window edges 7, 50 index, execute 0 only (develop compiles masking out; #23046 rejects a nonzero value; ethereum-optimism/protocol-team#457)
history-pruned follower 7, 50 index 0

L2 CL is op-node in every cell: the suite's preset runs op-node and the reorg check drives op-node's test sequencer.

Local verification

Rust (cd rust && cargo test -p reth-optimism-rpc --lib state::): 17 passed. cargo clippy -p reth-optimism-rpc --all-targets clean, just fmt-check clean.

e2e, run from rust/op-reth/tests against a debug op-reth built from this worktree (copied out of the shared target dir and selected with RUST_BINARY_PATH_OP_RETH, with the recipe's other env vars set by hand because just test-e2e-sysgo hard-codes ../../target/release/op-reth, which has no release build here):

DISABLE_OP_E2E_LEGACY=true DEVSTACK_ORCHESTRATOR=sysgo OP_RETH_ENABLE_PROOF_HISTORY=true RUST_BINARY_PATH_OP_RETH=<debug op-reth> \
  go test -count=1 -timeout 40m -parallel 2 -v -run 'TestProofsFollowerServesExactlyTheWindowItReports/<cell>$' ./proofs/core/
cell result setup block window after pruning pruned block refused with eth_getProof median latency earliest / latest
threshold_7-index PASS (85s) 18 [29, 49] no state found for block number 28 455µs / 406µs
threshold_7-execute PASS (85s) 18 [28, 48] no state found for block number 27 418µs / 346µs
threshold_50-index PASS (84s) 18 [27, 47] no state found for block number 26 351µs / 302µs
threshold_50-execute PASS (85s) 18 [28, 48] no state found for block number 27 417µs / 370µs

Every cell: debug_executionWitness for the earliest block is refused naming the block below it; the height imported right after sequencing resumed was refused with no state found for block number N while debug_proofsSyncStatus still reported the previous head, then matched the reference once flushed; the reorged-out hash was refused with header not found and the replacement served once the store had indexed blocks above it. The stale-proof probe right after the reorg (see Findings) reported stale=true in threshold_7-index and threshold_7-execute and stale=false in threshold_50-index and threshold_50-execute.

TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint (same env, -run 'TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint'):

cell result chain head at the stop reth history probe (block: state) checked block
threshold_7 PASS (72s) 65 1, 14, 27, 28 all pruned 28 served from the store, witness for 29 executes
threshold_50 PASS (112s) 105 1, 14, 27, 28 all pruned 28 served from the store, witness for 29 executes

Not run locally: op-geth validator (OP_DEVSTACK_PROOF_VALIDATOR_EL=op-geth skips both tests with a message) and a kona-node CL (the preset runs op-node).

Findings

  • Stale proofs right after a same-height reorg (bug candidate). Once the follower's reth has adopted a replacement block at a height the proofs store already holds, and until the ExEx has processed the reorg notification, eth_getProof by the replacement's hash passes the canonical-hash check and the window check and answers with the replaced block's state. debug_proofsSyncStatus cannot reveal the gap: it reports numbers, and the height is "served". Reproduced once in a local run of the reorg subtest (threshold 7, index path): 2ms after the follower reported the reorg at height 59, the window was still [39, 59] and the proof for the recorder at 59 failed with block 59 account 0x49C0...: eth_getProof from validator does not verify against state root 0xd97c.... The store's ProofWindowRange.latest carries the hash of the stored tip, so the factory could refuse requests while that hash is not the canonical hash of the height (or, for deeper reorgs, while the store's tip is on the replaced chain). The subtest now records whether it hit the gap and waits for the store to index blocks above the replacement before asserting; a fix should turn that probe into an assertion.
  • debug_executionWitness cannot serve the window's earliest block: it executes the block on its parent's state, which is the state pruning removed, so it fails with no state found for block number <earliest-1>. The oldest block whose witness a proofs node serves is earliest+1; debug_executePayload on the earliest block works. kona-host callers that walk down to debug_proofsSyncStatus.earliest should stop one block above it. Recorded, not a bug in the handler.
  • On develop, eth_getProof answers from the proofs store even where reth's own history is pruned: with account and storage history pruned before block 30, reth refuses eth_getStorageAt at 28 and below while the proofs follower serves verified proofs there and executes the witness for 29. The "broader limitation" paragraph of ethereum-optimism/protocol-team#474 (the path "fails on a history-pruned reth even when the proofs store has the block") does not reproduce on develop: state_at_block_id only builds a lazy historical provider, and the overlay reads accounts, storage and trie nodes from the store, reth only for bytecode and block hashes. The bump's rewrite can be checked against this test.
  • reth still answers historical state at the prune target block itself (before = 30 leaves 29 served, 28 refused): reconstructing state at the target needs only the changesets above it. The test checks the block below the target.
  • reth refuses a history prune distance under its unwind-safe minimum of 10064 blocks in both the CLI and the config file, and full is rejected the same way, so the issue's --full follower cannot be built on a short chain. A before = <block> mode in the config file with minimum_pruning_distance lowered in the same file is accepted; --prune.* flags cannot be used alongside, since they replace the file's prune section and with it the lowered minimum.
  • The pruner's own decisions are only logged at debug and the devstack prints op-reth's log at the test level, so the history-pruned follower runs with --log.stdout.filter=info,pruner=debug,engine::persistence=debug and the test keeps RecordLogLines records of the pruner's runs, which it prints when the precondition fails.
  • The error a client gets for a block outside the window on develop is no state found for block number N (an internal error, as Calls made to the Execution Manager do not return data #474 notes); the tests assert the error names the block, not its wording, so the bump can change the message.
  • The latest tag on a proofs follower resolves to the chain head and is refused while the store trails it (the Rust case latest_tag_while_the_store_trails); the e2e latest check runs with the chain stopped so the store has caught up.

Seams

  • utils.NewMixedOpProofPreset(t, opts...) takes MixedOpProofPresetConfig options: deployer options and per-node op-reth options for the sequencer and the validator, so a core test can run the reference-sequencer pattern of the acceptance tests.
  • utils.ValidatorELKind(t) for skipping when OP_DEVSTACK_PROOF_VALIDATOR_EL is not op-reth.
  • utils.NextPayloadAttributes(parent, rawTxs...) and utils.SignedTransfer(...) replace the attribute construction the two debug_executePayload tests duplicated.

CI follow-up

The stacked PR's run of op-reth-e2e-sysgo-tests (CircleCI job 5705613) failed two cells that passed on this PR's own run.

  • TestProofsFollowerServesExactlyTheWindowItReports/threshold_7-index/hash_of_a_reorged-out_block_is_refused_and_the_replacement_is_served failed with proofs store on validator does not serve block 57: window [61, 81]. The last two subtests resumed op-node's sequencer, which after a pause builds one block per second the chain stood still (it catches up to wall-clock time), so the batcher stop, the store's idle flush of the replaced block (8s on that box), the 2s polls and the workload transactions between the replaced block and the check all turned into blocks: the chain was 24 blocks past the replaced height when it was checked, 4 more than the window. Locally on the same code (-parallel 2 under six busy loops) the check of height 60 ran with window [59, 79], a margin of one block. The two subtests now build their blocks with the test sequencer and never resume op-node's sequencer: one block for the unflushed-height check; the replaced block, the replacement and two blocks above it for the reorg check, where the follower adopts the replacement chain through the batcher and L1 derivation (op-node drops gossip at its own unsafe height and gossip that does not extend its unsafe head). The chain advances exactly as far as the test says, so the window keeps every checked height however long the waits take.
  • TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint/threshold_50 failed in driveUntilBlock with must transact: upstream dep 0 (*plan.Lazy[*types.Receipt]) failed: operation failed permanently after 5 attempts: not found. The job's log shows the whole test process silent from 08:27:39 to 08:27:59 (no output from any test), the sequencer sealing nothing for 18s and then blocks 90 to 107 in one burst at 08:28:00; the EOA plan's default inclusion budget (5 exponential attempts, 15s) expired during the stall. Not a nonce or fee problem. The workload's transactions now carry a bounded 10-attempt inclusion budget (about 65s), so a stall of tens of seconds fails the run only if the sequencer does not catch up afterwards.

Local verification after the fix, the whole proofs/core package with the same env as above, 54 tests each:

run result reworked subtests, per cell
go test -count=1 -timeout 40m -v ./proofs/core/ PASS, 453s unflushed height 10-11s, reorg 22s
go test -count=1 -timeout 40m -parallel 4 -v ./proofs/core/ under six busy loops PASS, 513s unflushed height 10-11s, reorg 22s

In every cell of both runs the height just imported was refused while debug_proofsSyncStatus still reported the old head (the store flushes a lone block on its 10s idle timer), then matched the reference; the replacement was checked with the window's latest two blocks above it and its earliest 18 below (for example height 55 with window [37, 57]). The stale-proof probe of the Findings section reported refused=true in most cells with the derivation-driven reorg (the store had unwound the replaced block and not yet indexed the replacement) and stale=true once in the loaded run, so that finding still reproduces.

Before the fix, the same two cells passed locally under the same load (-parallel 2, six busy loops), which is why the CI run was needed to surface them.

Stacked on #23429, #23428, #23421 and #23418.

Closes #23390

🤖 Generated with Claude Code

CI follow-up: the history-pruned detection holds on both reth pins

waitForRethHistoryPrunedAt matched pruned in reth's eth_getStorageAt error, the message of ProviderError::StateAtBlockPruned (state at block #28 is pruned, reth 2.5.2). From reth 2.6.0 the read below the history prune checkpoint is refused with ProviderError::InsufficientChangesets (insufficient changesets to revert to block #28. Available changeset range: 29..=92), so on the bump the wait timed out before the proofs were verified. isRethHistoryPrunedError now accepts both messages (not the block they name: reth 2.5 names the block above the read, reth 2.6 the read's own); a purely structural check cannot tell the refusal from any other per-block failure. Verified with go test -run TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint ./proofs/core/: both thresholds pass on develop (op-reth 2.5.2). On the bump evaluation tree (einar/bump-eval-stacks, op-reth 2.6.0 + #23358) both cells now pass the wait and VerifyProofsMatchAt, then fail at the next assertion, which is a bump-only op-reth difference rather than a detection problem: debug_executionWitness for block 29 returns failed to apply blockhash contract call: database error: Database error: insufficient changesets to revert to block #28, i.e. the blockhash system call in the witness path reads the parent state from reth's pruned history instead of the proofs store.

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 31, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
einar-oplabs and others added 6 commits November 9, 2026 20:39
Rust: cover OpStateProviderFactory::state_provider with a mock eth API
over a seeded in-memory proofs store, at and around the window's edges.

e2e, in proofs/core: a window-edges matrix over reth's persistence
threshold and the proofs path that stops the chain and checks what
eth_getProof, debug_executePayload and debug_executionWitness answer at
the earliest block, below it, at the latest, for the latest tag, for a
height the store has not flushed, for a reorged-out hash, and under more
concurrent witness requests than tracing permits; plus a follower whose
reth prunes its own history below the proofs window. The dormant
proofs/prune package is folded into these checks and deleted.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 65d6a71)
(cherry picked from commit e2284bcfb095a8d9423a95e3bd1d5afe3ef8ba0a)
On a loaded CI box two cells of the proofs e2e suite failed that pass on an
idle one. The last two subtests of the window test resumed op-node's
sequencer, which after a pause builds one block per second the chain stood
still, so every wait between the replaced block and the check turned into
blocks and the 20-block window moved past the height under check (57 with
window [61, 81]). The subtests now build their blocks with the test
sequencer and never resume op-node's sequencer: one block for the unflushed
height, and the replaced block, the replacement and two blocks above it for
the reorg, which the follower adopts through the batcher and L1 derivation.
The chain advances exactly as far as the test says.

The history-pruned cell lost a workload transaction to the EOA plan's 15s
inclusion budget while the whole test process stalled for 20s. The
workload's transactions now carry a bounded 10-attempt budget of about a
minute, so a stall the sequencer catches up from no longer fails the run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit c2f177d)
(cherry picked from commit 9143461e43004cd9f584858496b25b7bdd1eb950)
`waitForRethHistoryPrunedAt` matched "pruned" in the eth_getStorageAt error,
the message of `ProviderError::StateAtBlockPruned` ("state at block #N is
pruned"). From reth 2.6 the read below the history prune checkpoint is
refused with `ProviderError::InsufficientChangesets` ("insufficient
changesets to revert to block #N. Available changeset range: M..=K"), so the
wait never saw reth's history go and the test timed out before verifying the
proofs.

A structural check (the read at the block fails while one above the
available range succeeds) cannot tell the refusal from any other per-block
failure, so the test keeps matching messages and accepts both. The block a
message names is not checked: reth 2.5 names the block above the read, reth
2.6 the read's own.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit de8f55d)
(cherry picked from commit 06fc35b24120bc6afaaa2a058d7fd1025ce93cfd)
…ed e2e test

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit ba59ee47267f735cb8761a121a9381e7d55856a4)
Drop the double parentheses the merge left around the state_provider arguments
and rustfmt the result.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…#683

debug_executionWitness above reth's history prune checkpoint fails on reth 2.6+:
the EIP-2935 blockhash call reads pruned history.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23390-proofs-rpc-edges branch from de8f55d to a1e8077 Compare November 9, 2026 22:29
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: tests for the proofs RPC window edges and old-block witness requests

3 participants