Skip to content

op-reth: live test of proofs execution across reth parent states - #23441

Closed
einar-oplabs wants to merge 3 commits into
einar/23393-proofs-cli-testsfrom
einar/23384-proofs-parent-states
Closed

einar-oplabs wants to merge 3 commits into
einar/23393-proofs-cli-testsfrom
einar/23384-proofs-parent-states

Conversation

@einar-oplabs

Copy link
Copy Markdown
Contributor

The proofs engine executes a block against reth's state for the block's parent, and reth moves that parent between memory and disk on its own schedule. Each parent state is unit-tested in isolation on #23046, but nothing ran the whole execute path across the transitions, which is where the next reth bump breaks it.

What the test does

rust/op-reth/crates/trie/tests/live/parent_states.rs (a submodule of tests/live.rs) drives a real BlockchainProvider whose CanonicalInMemoryState and database a SimulatedReth advances with reth's own persistence rule: once the head runs more than persistence_threshold blocks past the database tip, persist every block but the newest memory_block_buffer_target through reth's save_blocks writer in one transaction, then evict the persisted blocks from memory except the newest num_state_masking_blocks. The chain is block n deploying a contract, n+1 calling it (slot 0 := 1) and n+2 storing BLOCKHASH(n) (slot 1). The engine executes n+1 (call content) or n+2 (BLOCKHASH content) with the parent in one of these states:

parent state how it is reached expected
in memory reth's steady state stored
persisted and evicted reth runs ahead until its persistence cycle evicts the parent stored
persisted and masked masking keeps the persisted parent in memory ((50, 30) only; the deployment sits right above the first eviction boundary) stored
evicted while executing reth's persistence cycle runs after the engine snapshotted the parent state and before it executes stored
abandoned fork reorged out of memory before execution, never persisted EngineError::StateRootMismatch for the block, store stays at the parent
in memory, deployment evicted (BLOCKHASH only) the chain straddles the first eviction boundary, so the hash the block reads lies below reth's in-memory chain stored, slot 1 holds the deployment's hash
reorged below while executing (BLOCKHASH only) reth reorgs the deployment and the parent out after the snapshot stored, slot 1 holds the hash on the parent's ancestry

After a stored outcome the test checks the store's latest block, recomputes the state root at that block from the proofs store and compares it with the header, and reads the contract's slots back through OpProofsStateProviderRef. Before and after execution it compares where reth holds the parent (canonical, in memory, persisted) with the expectation of the parent state.

Mid-execution transitions go through StoreWithReadHook, a proofs store whose provider_ro() runs a one-shot action: the engine's execute task opens that read view right after it snapshots reth's parent state, on develop and on #23046 alike, so the action lands between the snapshot and execution. The engine also opens a read view to find its tip when its block buffer is empty, so the cases keep the preceding blocks buffered, and the test asserts right after execute_block returns that the hook fired.

Matrix

rstest tables over reth persistence threshold {7, 50} x masking {0, 30} x memory-block-buffer target {5} x block content {call, BLOCKHASH}, each #[case] naming the parent state and the outcome. reth rejects num_state_masking_blocks + memory_block_buffer_target >= persistence_threshold, so (7, 30) is not a valid cell and the matrix runs (7, 0), (50, 0) and (50, 30); the issue's matrix lists the invalid cell. 32 cases plus the six ignored ones:

  • execute_block_across_reth_parent_states: {in memory, persisted and evicted, evicted while executing, abandoned fork} x 3 persistence settings x 2 contents (24; the in-memory and the two fork rows never reach a persistence cycle, so there the settings only exercise the simulation)
  • execute_block_with_persisted_and_masked_parent: (50, 30) x 2 contents (2)
  • execute_block_hash_read_below_reth_memory: 3 persistence settings (3)
  • execute_block_hash_read_while_reth_reorgs_below_it: 3 persistence settings (3)
  • plus the six pre-existing live.rs tests

The issue's shared parameter matrix also lists the engine flush cadence (1, 2) and (5, 10). The test does not vary it: EngineHandle::flush only waits for a save already in flight and never starts one, so the engine is spawned with spawn_with_thresholds(n, n + 1) where n is the number of blocks the case stores; the save fires after the block under test (after the parent for the mismatch case) and flush waits for it. A cadence of 1 would also empty the buffer before the block under test and defeat the read hook.

Harness changes in tests/live.rs

  • TxSpec gains to: TxKind, data and gas_limit, in the shape rust!: update reth to v2.6.0 #23046 gives it, plus TxSpec::create and TxSpec::call.
  • initialize_storage holds the proofs-store initialization the scenario runner already did, so both tests share it.
  • rstest and reth-chain-state join the dev-dependencies.

Local verification

cd rust && cargo test -p reth-optimism-trie --test live
test result: ok. 32 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 12.03s

cd rust && cargo test -p reth-optimism-trie --test live -- --ignored abandoned_fork
test result: FAILED. 0 passed; 6 failed; 0 ignored; 0 measured; 32 filtered out; finished in 1.87s

cargo clippy -p reth-optimism-trie --all-targets --all-features is clean and the files are formatted with just fmt-fix.

Ignored cells

The six abandoned_fork cases (3 persistence settings x 2 contents) are #[ignore]d naming #23046. On develop the execute path resolves the parent with state_by_block_hash, which fails with StateForHashNotFound for a parent reth no longer holds, and the task treats that as recoverable: it returns Ok(()) and stores nothing. Each ignored case fails on develop with:

an abandoned parent must fail the state-root check: ()

#23046 replaces the historical state provider with a snapshot of reth's in-memory chain plus the Bytecodes table; a parent that is already off the canonical chain has no snapshot and the contract's code is not on disk, so execution fails the state-root check. The bump PR un-ignores the cases by deleting the #[ignore] line.

Verified on #23046's head

I ran the file on 93d9ec9d2f (the head of #23046) with the ignore removed. On reth 2.6 BlockchainProvider::state_by_block_hash serves an in-memory block through an OverlayStateProvider fed by an OverlayManager the engine tree maintains next to CanonicalInMemoryState, so SimulatedReth must register blocks there as well; with that adaptation, which belongs in the bump PR, all 39 tests pass with no ignores:

test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.67s

The adaptation (plus reth-storage-overlay.workspace = true under [dev-dependencies]):

+use reth_storage_overlay::OverlayManager;
 ...
     nonce: u64,
+    overlay_manager: OverlayManager,
     persistence: RethPersistence,
 ...
-        let factory = create_test_provider_factory_with_chain_spec(chain_spec.clone());
+        let overlay_manager = OverlayManager::default();
+        let factory = create_test_provider_factory_with_chain_spec(chain_spec.clone())
+            .with_overlay_manager(overlay_manager.clone());
 ...
     fn make_canonical(&mut self, update: NewCanonicalChain) {
         let tip = update.tip().clone_sealed_header();
+        if let NewCanonicalChain::Reorg { old, .. } = &update {
+            self.overlay_manager.remove_blocks(old.iter().map(|block| block.recovered_block().hash()));
+        }
         match &update {
             NewCanonicalChain::Commit { new } | NewCanonicalChain::Reorg { new, .. } => {
+                for block in new {
+                    self.overlay_manager.insert_block(block.clone());
+                }
                 self.canonical.extend(new.iter().cloned());
 ...
     fn evict_through(&mut self, number: u64) {
         assert!(number <= self.last_persisted.number, "only persisted blocks leave memory");
         self.in_memory.remove_persisted_blocks_until(self.last_persisted, number);
+        let evicted = self.canonical[self.last_evicted.number as usize..number as usize]
+            .iter()
+            .map(|block| block.recovered_block().hash())
+            .collect::<Vec<_>>();
+        self.overlay_manager.remove_blocks(evicted);
         self.last_evicted = self.num_hash(number);

On the bump reth's partial-persistence feature is always on, so SimulatedReth::persist can also pass the masked frontier (new_db_tip - num_state_masking_blocks) as save_blocks' new_partial_state_trie instead of the database tip, which models the held-back hashed state and trie of masked blocks.

Findings

  • No new bug. The abandoned-fork behavior on develop (skip and store nothing) is the known pre-bump behavior the issue describes.
  • Masking is compiled out on develop one level deeper than the CLI: reth's save_blocks panics with serializing FinishCheckpoint requires the partial-persistence feature as soon as it is asked to hold back the hashed state and trie of masked blocks, and op-reth enables that feature nowhere. The simulation therefore gives the writer both frontiers at the database tip and masking only decides which persisted blocks stay in memory. Enabling the feature through this crate's dev-dependencies would unify it into every workspace test build, so I left it off.
  • EngineHandle::flush only waits for a save that is already in flight; it never starts one. The existing scenario runner never asserted the store's tip, which is why this went unnoticed.
  • While the simulation still hit the panic above inside the read hook, the engine thread died and the test thread stayed blocked in EngineHandle::flush (and in one run in execute_block) instead of getting EngineError::EngineDied; sample showed no engine or persistence thread left. I did not chase the mechanism, but an engine panic leaving callers waiting is worth a look.
  • The issue's matrix cell (persistence threshold 7, masking 30) is a configuration reth rejects (num_state_masking_blocks + memory_block_buffer_target < persistence_threshold), so it has no case.

Seams for later PRs

  • SimulatedReth (reth's persistence rule over CanonicalInMemoryState and the test database, with save_blocks) and StoreWithReadHook (a one-shot action between the engine's parent-state snapshot and execution) live in tests/live/parent_states.rs and can be lifted into the crate's test utilities when op-reth: live test of index-versus-execute proofs store parity over rich block content #23388 needs them.
  • initialize_storage in tests/live.rs seeds a proofs store from the test database at a given block.

Stacked on #23437, #23434, #23429, #23428, #23421 and #23418.

Closes #23384

🤖 Generated with Claude Code

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 31, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
einar-oplabs and others added 3 commits November 9, 2026 22:06
Execute the call and BLOCKHASH blocks of a deploy, call, BLOCKHASH chain
through the proofs engine against a real BlockchainProvider while a
simulated reth moves the parent between memory, disk and an abandoned
fork with reth's own persistence rule (threshold 7 and 50, masking 0 and
30, memory-block-buffer target 5). A proofs store hook fires reth's
mid-execution transitions right after the engine snapshots the parent
state. The abandoned-fork cases assert the bump's state-root failure and
are ignored on develop, which skips such a parent with
StateForHashNotFound.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 22bbc1d)
(cherry picked from commit 852fb64a2d328e6a3c53af9eee89c4a59982a3e7)
Persist with reth's own save_blocks writer, keep live.rs's single-block
helper and share its storage initialization, state the read hook's
buffered-blocks precondition and check it right after execution, assert
block locations uniformly, add the BLOCKHASH case whose deployment sits
below reth's memory, name the persistence settings, use a fixed signer and
run the hook inline.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit cf9f0ca)
(cherry picked from commit 6db279118b71060163e09b3b2208f6976e625bb0)
Reth now serves in-memory state through its OverlayManager, so the simulated
reth registers the blocks it makes canonical, drops the ones it reorgs out or
evicts, and the abandoned_fork case holds without #23046.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23384-proofs-parent-states branch from cf9f0ca to 3754324 Compare November 9, 2026 22:30
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: live test of proofs execution across reth's in-memory, persisted and evicted parent states

3 participants