Repository navigation
op-reth: test the proofs RPC window edges and old-block witness requests - #23434
Closed
einar-oplabs wants to merge 6 commits into
Closed
einar-oplabs wants to merge 6 commits into
einar-oplabs wants to merge 6 commits into
Conversation
This was referenced Oct 15, 2026
Contributor
|
This pr has been automatically marked as stale and will be closed in 5 days if no updates |
Rust: cover OpStateProviderFactory::state_provider with a mock eth API over a seeded in-memory proofs store, at and around the window's edges. e2e, in proofs/core: a window-edges matrix over reth's persistence threshold and the proofs path that stops the chain and checks what eth_getProof, debug_executePayload and debug_executionWitness answer at the earliest block, below it, at the latest, for the latest tag, for a height the store has not flushed, for a reorged-out hash, and under more concurrent witness requests than tracing permits; plus a follower whose reth prunes its own history below the proofs window. The dormant proofs/prune package is folded into these checks and deleted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 65d6a71) (cherry picked from commit e2284bcfb095a8d9423a95e3bd1d5afe3ef8ba0a)
On a loaded CI box two cells of the proofs e2e suite failed that pass on an idle one. The last two subtests of the window test resumed op-node's sequencer, which after a pause builds one block per second the chain stood still, so every wait between the replaced block and the check turned into blocks and the 20-block window moved past the height under check (57 with window [61, 81]). The subtests now build their blocks with the test sequencer and never resume op-node's sequencer: one block for the unflushed height, and the replaced block, the replacement and two blocks above it for the reorg, which the follower adopts through the batcher and L1 derivation. The chain advances exactly as far as the test says. The history-pruned cell lost a workload transaction to the EOA plan's 15s inclusion budget while the whole test process stalled for 20s. The workload's transactions now carry a bounded 10-attempt budget of about a minute, so a stall the sequencer catches up from no longer fails the run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit c2f177d) (cherry picked from commit 9143461e43004cd9f584858496b25b7bdd1eb950)
`waitForRethHistoryPrunedAt` matched "pruned" in the eth_getStorageAt error,
the message of `ProviderError::StateAtBlockPruned` ("state at block #N is
pruned"). From reth 2.6 the read below the history prune checkpoint is
refused with `ProviderError::InsufficientChangesets` ("insufficient
changesets to revert to block #N. Available changeset range: M..=K"), so the
wait never saw reth's history go and the test timed out before verifying the
proofs.
A structural check (the read at the block fails while one above the
available range succeeds) cannot tell the refusal from any other per-block
failure, so the test keeps matching messages and accepts both. The block a
message names is not checked: reth 2.5 names the block above the read, reth
2.6 the read's own.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit de8f55d)
(cherry picked from commit 06fc35b24120bc6afaaa2a058d7fd1025ce93cfd)
…ed e2e test Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit ba59ee47267f735cb8761a121a9381e7d55856a4)
Drop the double parentheses the merge left around the state_provider arguments and rustfmt the result. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…#683 debug_executionWitness above reth's history prune checkpoint fails on reth 2.6+: the EIP-2935 blockhash call reads pruned history. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
einar-oplabs
force-pushed
the
einar/23390-proofs-rpc-edges
branch
from
November 9, 2026 22:29
de8f55d to
a1e8077
Compare
einar-oplabs
marked this pull request as ready for review
November 13, 2026 11:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The historical-proofs RPC handler (
rust/op-reth/crates/rpc/src/eth/proofs.rs) and its state-provider factory (state.rs) had no tests, and the only e2e check near the window's earliest block was the dormantproofs/prunepackage. The reth 2.6 bump (#23046) rewrites the handler, so this pins whateth_getProof,debug_executePayloadanddebug_executionWitnessanswer at and around the window's edges before the rewrite lands.What the tests cover
Rust:
OpStateProviderFactory::state_provider(rust/op-reth/crates/rpc/src/state.rs, 16 cases)A mock eth API (
reth_rpc::EthApioverMockEthProvider) with a canonical chain of 21 blocks and anInMemoryProofsStorageseeded with blocks 10..=18, so the store trails the head the way it does between flushes:StateForNumberNotFound(n): one below the earliest (by number and by hash of the canonical block), genesis, one above the store tip, the chain head the store has not flushed, thelatesttag while the store trailslatesttag, no block id, the head by numberProviderError::Other("header not found")(the wrapping that ethereum-optimism/protocol-team#474 reports as surfacing to clients as an internal error)Not covered in Rust: a hash that names a known non-canonical block.
MockEthProviderresolvesblock_number(hash)andblock_hash(number)from the same header map, so a non-canonical block cannot be modelled; the e2e reorg check below covers that branch.The test build enables
reth-optimism-trie/metricssoOpProofsStorage<S>is the production wrapper, as it is for the node and for CI's--workspace --all-featuresclippy and nextest runs; without it the alias is the bareSand clippy flags the conversion the node needs.e2e, in
rust/op-reth/tests/proofs/core(CI jobop-reth-e2e-sysgo-tests,just test-e2e-sysgowith the defaultGO_PKG_NAME=proofs/core)TestProofsFollowerServesExactlyTheWindowItReports, matrix threshold {7, 50} x proofs path {index, execute}, each cell a sequencer without proofs history (the independenteth_getProofreference,--rpc.eth-proof-window) and a follower with a 20-block window,--rpc.max-tracing-requests=2, 1s blocks. The chain is driven past the window until pruning ran, then sequencing is stopped so the window stands still, and the cell checks:no state found for block number N)latesttag and equals the reference at the headeth_getProoflatency at the earliest block versus the latest (medians of 10), logged for the bump comparison and bounded at 10x with a 250ms floor so only a lookup that re-walks the window can trip itdebug_executePayloadbuilds on the earliest blockdebug_executionWitnessrefuses the earliest block (it executes on the parent's state, which pruning removed) and serves the block after itdebug_executionWitnessrequests, above the 2 tracing permits and above the handler's own 3, all succeeddebug_proofsSyncStatusreports it and never answered with a proof of another block, then matches the reference; the subtest stops the chain again, so every subtest starts from a stopped chain and can run aloneTestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint, threshold {7, 50}, index path: the follower's reth prunes account and storage history 5 blocks behind its persisted tip (through a--configfile, since reth's unwind-safe minimum of 10064 blocks has to be lowered in the file and--prune.*flags replace the file's prune section), with a proofs window that reaches below that checkpoint. Onceeth_getStorageAtat the window's earliest block reports reth's own state as pruned,eth_getProofthere must verify and match the reference anddebug_executionWitnessfor the next block must execute: the store serves the window, not reth's history. This is the "block below reth's history prune checkpoint" cell from the issue; the issue's--fullflag cannot be used because it pins the history distance to 10064 blocks.The dormant
proofs/prunepackage is folded into these checks (window read fromdebug_proofsSyncStatusand asserted equal to the configured window after pruning, pruned block refused, new earliest served) and deleted.Matrix
L2 CL is op-node in every cell: the suite's preset runs op-node and the reorg check drives op-node's test sequencer.
Local verification
Rust (
cd rust && cargo test -p reth-optimism-rpc --lib state::): 17 passed.cargo clippy -p reth-optimism-rpc --all-targetsclean,just fmt-checkclean.e2e, run from
rust/op-reth/testsagainst a debug op-reth built from this worktree (copied out of the shared target dir and selected withRUST_BINARY_PATH_OP_RETH, with the recipe's other env vars set by hand becausejust test-e2e-sysgohard-codes../../target/release/op-reth, which has no release build here):no state found for block number 28no state found for block number 27no state found for block number 26no state found for block number 27Every cell:
debug_executionWitnessfor the earliest block is refused naming the block below it; the height imported right after sequencing resumed was refused withno state found for block number Nwhiledebug_proofsSyncStatusstill reported the previous head, then matched the reference once flushed; the reorged-out hash was refused withheader not foundand the replacement served once the store had indexed blocks above it. The stale-proof probe right after the reorg (see Findings) reportedstale=truein threshold_7-index and threshold_7-execute andstale=falsein threshold_50-index and threshold_50-execute.TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint(same env,-run 'TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint'):prunedprunedNot run locally: op-geth validator (
OP_DEVSTACK_PROOF_VALIDATOR_EL=op-gethskips both tests with a message) and a kona-node CL (the preset runs op-node).Findings
eth_getProofby the replacement's hash passes the canonical-hash check and the window check and answers with the replaced block's state.debug_proofsSyncStatuscannot reveal the gap: it reports numbers, and the height is "served". Reproduced once in a local run of the reorg subtest (threshold 7, index path): 2ms after the follower reported the reorg at height 59, the window was still[39, 59]and the proof for the recorder at 59 failed withblock 59 account 0x49C0...: eth_getProof from validator does not verify against state root 0xd97c.... The store'sProofWindowRange.latestcarries the hash of the stored tip, so the factory could refuse requests while that hash is not the canonical hash of the height (or, for deeper reorgs, while the store's tip is on the replaced chain). The subtest now records whether it hit the gap and waits for the store to index blocks above the replacement before asserting; a fix should turn that probe into an assertion.debug_executionWitnesscannot serve the window's earliest block: it executes the block on its parent's state, which is the state pruning removed, so it fails withno state found for block number <earliest-1>. The oldest block whose witness a proofs node serves is earliest+1;debug_executePayloadon the earliest block works. kona-host callers that walk down todebug_proofsSyncStatus.earliestshould stop one block above it. Recorded, not a bug in the handler.eth_getStorageAtat 28 and below while the proofs follower serves verified proofs there and executes the witness for 29. The "broader limitation" paragraph of ethereum-optimism/protocol-team#474 (the path "fails on a history-pruned reth even when the proofs store has the block") does not reproduce on develop:state_at_block_idonly builds a lazy historical provider, and the overlay reads accounts, storage and trie nodes from the store, reth only for bytecode and block hashes. The bump's rewrite can be checked against this test.before = 30leaves 29 served, 28 refused): reconstructing state at the target needs only the changesets above it. The test checks the block below the target.fullis rejected the same way, so the issue's--fullfollower cannot be built on a short chain. Abefore = <block>mode in the config file withminimum_pruning_distancelowered in the same file is accepted;--prune.*flags cannot be used alongside, since they replace the file's prune section and with it the lowered minimum.--log.stdout.filter=info,pruner=debug,engine::persistence=debugand the test keepsRecordLogLinesrecords of the pruner's runs, which it prints when the precondition fails.no state found for block number N(an internal error, as Calls made to the Execution Manager do not return data #474 notes); the tests assert the error names the block, not its wording, so the bump can change the message.latesttag on a proofs follower resolves to the chain head and is refused while the store trails it (the Rust caselatest_tag_while_the_store_trails); the e2elatestcheck runs with the chain stopped so the store has caught up.Seams
utils.NewMixedOpProofPreset(t, opts...)takesMixedOpProofPresetConfigoptions: deployer options and per-node op-reth options for the sequencer and the validator, so a core test can run the reference-sequencer pattern of the acceptance tests.utils.ValidatorELKind(t)for skipping whenOP_DEVSTACK_PROOF_VALIDATOR_ELis not op-reth.utils.NextPayloadAttributes(parent, rawTxs...)andutils.SignedTransfer(...)replace the attribute construction the twodebug_executePayloadtests duplicated.CI follow-up
The stacked PR's run of
op-reth-e2e-sysgo-tests(CircleCI job 5705613) failed two cells that passed on this PR's own run.TestProofsFollowerServesExactlyTheWindowItReports/threshold_7-index/hash_of_a_reorged-out_block_is_refused_and_the_replacement_is_servedfailed withproofs store on validator does not serve block 57: window [61, 81]. The last two subtests resumed op-node's sequencer, which after a pause builds one block per second the chain stood still (it catches up to wall-clock time), so the batcher stop, the store's idle flush of the replaced block (8s on that box), the 2s polls and the workload transactions between the replaced block and the check all turned into blocks: the chain was 24 blocks past the replaced height when it was checked, 4 more than the window. Locally on the same code (-parallel 2under six busy loops) the check of height 60 ran with window[59, 79], a margin of one block. The two subtests now build their blocks with the test sequencer and never resume op-node's sequencer: one block for the unflushed-height check; the replaced block, the replacement and two blocks above it for the reorg check, where the follower adopts the replacement chain through the batcher and L1 derivation (op-node drops gossip at its own unsafe height and gossip that does not extend its unsafe head). The chain advances exactly as far as the test says, so the window keeps every checked height however long the waits take.TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint/threshold_50failed indriveUntilBlockwithmust transact: upstream dep 0 (*plan.Lazy[*types.Receipt]) failed: operation failed permanently after 5 attempts: not found. The job's log shows the whole test process silent from 08:27:39 to 08:27:59 (no output from any test), the sequencer sealing nothing for 18s and then blocks 90 to 107 in one burst at 08:28:00; the EOA plan's default inclusion budget (5 exponential attempts, 15s) expired during the stall. Not a nonce or fee problem. The workload's transactions now carry a bounded 10-attempt inclusion budget (about 65s), so a stall of tens of seconds fails the run only if the sequencer does not catch up afterwards.Local verification after the fix, the whole
proofs/corepackage with the same env as above, 54 tests each:go test -count=1 -timeout 40m -v ./proofs/core/go test -count=1 -timeout 40m -parallel 4 -v ./proofs/core/under six busy loopsIn every cell of both runs the height just imported was refused while
debug_proofsSyncStatusstill reported the old head (the store flushes a lone block on its 10s idle timer), then matched the reference; the replacement was checked with the window's latest two blocks above it and its earliest 18 below (for example height 55 with window[37, 57]). The stale-proof probe of the Findings section reportedrefused=truein most cells with the derivation-driven reorg (the store had unwound the replaced block and not yet indexed the replacement) andstale=trueonce in the loaded run, so that finding still reproduces.Before the fix, the same two cells passed locally under the same load (
-parallel 2, six busy loops), which is why the CI run was needed to surface them.Stacked on #23429, #23428, #23421 and #23418.
Closes #23390
🤖 Generated with Claude Code
CI follow-up: the history-pruned detection holds on both reth pins
waitForRethHistoryPrunedAtmatchedprunedin reth'seth_getStorageAterror, the message ofProviderError::StateAtBlockPruned(state at block #28 is pruned, reth 2.5.2). From reth 2.6.0 the read below the history prune checkpoint is refused withProviderError::InsufficientChangesets(insufficient changesets to revert to block #28. Available changeset range: 29..=92), so on the bump the wait timed out before the proofs were verified.isRethHistoryPrunedErrornow accepts both messages (not the block they name: reth 2.5 names the block above the read, reth 2.6 the read's own); a purely structural check cannot tell the refusal from any other per-block failure. Verified withgo test -run TestProofsFollowerServesBlocksBelowRethHistoryPruneCheckpoint ./proofs/core/: both thresholds pass on develop (op-reth 2.5.2). On the bump evaluation tree (einar/bump-eval-stacks, op-reth 2.6.0 + #23358) both cells now pass the wait andVerifyProofsMatchAt, then fail at the next assertion, which is a bump-only op-reth difference rather than a detection problem:debug_executionWitnessfor block 29 returnsfailed to apply blockhash contract call: database error: Database error: insufficient changesets to revert to block #28, i.e. the blockhash system call in the witness path reads the parent state from reth's pruned history instead of the proofs store.