Skip to content

op-reth: test the proofs engine reorg and revert edges and check unwind targets - #23445

Closed
einar-oplabs wants to merge 4 commits into
einar/23388-proofs-index-execute-parityfrom
einar/23391-proofs-engine-edges
Closed

einar-oplabs wants to merge 4 commits into
einar/23388-proofs-index-execute-parityfrom
einar/23391-proofs-engine-edges

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Deterministic tests for the proofs engine's reorg, revert and unwind tasks at the boundaries the model-based test may not reach, with the two engine checks they found missing, and the ExEx executing a reorged chain that carries no trie data instead of failing the node.

The reorg task never checked the first new block's parent against the stored fork block, and nothing refused an unwind at or below the window's earliest block before the store saw it (MDBX refuses it; the in-memory store underflowed at block 0 and otherwise removed the earliest block). handle_chain_reorged returned an error for a new chain without trie data, which reth turns into a node exit.

Changes

  • EngineState::unwind (shared by the reorg and unwind tasks) now drains any in-flight save, then before touching anything refuses a target at or below earliest (EngineError::Storage(UnwindBeyondEarliest)) and a target whose parent differs from the stored block below it (ParentHashMismatch). The stores keep only the window's end hashes, so the engine buffer retains the hashes of the blocks it persisted (bounded to 1024) to verify forks below the disk tip within a process lifetime. A restarted engine cannot verify a fork below its disk tip; that cell is an #[ignore]d test (see below).
  • EngineHandle::reorg takes Vec<ReorgBlock>, each block either Index { block, hashed_state, trie_updates } or Execute(block); the task runs each through the same code as index_block and execute_block, so every block gets the gap and parent checks (an executed block the engine has to skip no longer lets the next indexed one land on a gap). The ExEx picks per block with the rule ChainCommitted uses (trie data present and verification not due → index, else execute), shared in precomputed_trie_data.
  • PersistenceState::unwind no longer waits for the in-flight save itself; EngineState::unwind drains it first, in one place. The two rejections log at warn.
  • InMemoryProofsStorage::unwind_history follows the trait contract: no-op above latest, UnwindBeyondEarliest at or below earliest, no underflow.
  • ParentHashMismatch's docs now describe the unwind case too.

Tests

Engine live tests in rust/op-reth/crates/trie/tests/live/reorg_revert_edges.rs, over backend {InMemory, Mdbx} × flush cadence {(1, 2), (5, 10)} unless noted, each on a seven-block transfer chain. The engine's tip is read through the parent check of a probe block; the persisted history is compared table by table with a fresh store indexed over reth's final canonical chain (BlockView / assert_same_view), and its proofs are verified against reth's state roots.

Case Test Cells
baseline reorg_replaces_the_stored_blocks_above_the_fork × path {Index at fork 6, Execute at fork 3, Mixed at fork 4}
(a) reorg_not_building_on_the_stored_fork_block_fails_and_changes_nothing × fork {6 (buffer or save), 2 (save), 0 (anchor)} × path
(a) restart restarted_engine_rejects_a_reorg_not_building_on_a_block_below_its_disk_tip #[ignore], see Findings
(b) unwind_or_reorg_at_or_below_the_earliest_block_fails_and_changes_nothing × {unwind at earliest, reorg below earliest, reorg at the anchor below a pruned window, unwind at the anchor as earliest}
(b) control unwind_to_the_block_above_the_earliest_keeps_only_the_earliest pruned window
(b) in flight unwind_at_a_block_the_in_flight_save_prunes_fails_after_the_save pruned window; pins that the checks run against the window as the in-flight save leaves it
(c) revert_of_buffered_blocks_unwinds_the_buffer_and_leaves_the_disk_untouched cadence (5, 10) only × revert from {6, 7}
(c) in flight revert_sent_during_an_in_flight_save_ends_with_the_disk_as_it_was
(d) reorg_sent_during_an_in_flight_save_ends_consistent_with_reth × path

A StoreWithWriteGate holds the engine's next write transaction so the save stays in flight until the case releases it. Whether the engine dequeues the revert or reorg before or after the held save completes is not observable from outside (both orders must end the same, and the tests assert that end state); the pruned-window in-flight case is the one where the order changes the outcome, and it pins it. SimulatedReth gained reorg_with(number, specs), executed_block, genesis, num_hash and spawn_engine_with_window; FlushCadence and Backend moved to live.rs for sharing.

Under cadence (1, 2) no block stays in the buffer outside a save in flight, so (c)'s "(1, 2)" cells are the in-flight revert test.

ExEx tests in rust/op-reth/crates/exex/src/lib.rs:

  • (e) chain_reorged_without_trie_data_executes_the_new_chain × backend: reth's canonical in-memory state holds the new chain, the notification carries no trie data, the store ends on the new tip and FinishedHeight acknowledges it.
  • (f) chain_reorged_with_different_fork_blocks_fails_and_changes_nothing: the error is returned, the store keeps its tip, no event is sent. handle_chain_reorged's docs state the expectation.
  • precomputed_trie_data_follows_the_verification_interval: a table over interval {0, 2} × block × trie data present/absent.

Unit tests: engine::buffer::state (hash_at through prune and unwind, bound), engine::state (unwind at the anchor, unwind onto a contradicted parent), and tests/lib.rs test_unwind_history_* over both backends.

Local verification

All from rust/ with the shared target dir, on the final commit:

  • cargo test -p reth-optimism-trie --test live — 120 passed, 8 ignored (6 pre-existing abandoned_fork cells from op-reth: live test of proofs execution across reth parent states #23441 and the 2 restart cells here), 40.0 s wall for the whole live target (21.9 s before this PR).
  • cargo test -p reth-optimism-trie --test live -- reorg_revert_edges — 64 passed, 2 ignored, 18.3 s.
  • cargo test -p reth-optimism-trie --lib — 248 passed (28 under engine::, including the 5 new buffer and state tests).
  • cargo test -p reth-optimism-trie --test lib — 94 passed (8 of them the new test_unwind_history_* cells over InMemory and Mdbx).
  • CARGO_INCREMENTAL=0 cargo test -p reth-optimism-exex --lib — 19 passed (12 existing, 2 new reorg tests, 5 precomputed_trie_data cases); cargo test -p reth-optimism-exex --doc --all-features — 1 passed (the crate doctest needs --all-features for OP_MAINNET, as just test-docs runs it; unchanged by this PR).
  • cargo clippy -p reth-optimism-trie -p reth-optimism-exex --all-targets — zero warnings; just fmt-check — clean.

The rust-code-reviewer agent ran on the diff; its findings (the reorg task's Index arm bypassing index_block's checks, the drain-before-check ordering untested, the in-flight test names, dead code in PersistenceState::unwind, naming and dedup) are addressed above.

Every matrix cell of the engine tests ran locally (both backends × both cadences); the only cells not run are the #[ignore]d restart cells, which fail by design today.

Findings

  • (a) and (b) fail on develop without the engine checks: with self.check_unwind_target(to)? removed from EngineState::unwind, all 28 cells of the two tests fail. (a) reorg_not_building_on_the_stored_fork_block_fails_and_changes_nothing: all 12 cells panic with the reorg does not build on the stored fork block: (), i.e. EngineHandle::reorg returned Ok(()) and inserted the chain on the wrong parent. (b) unwind_or_reorg_at_or_below_the_earliest_block_fails_and_changes_nothing: the 8 Mdbx cells get got Persistence(Storage(UnwindBeyondEarliest { .. })), the store refusing only after the engine drained the save and sent the unwind to the persistence service; with develop's InMemoryProofsStorage::unwind_history the 8 InMemory cells panic with the unwind would remove the window's earliest block: (), the unwind returning Ok(()) after removing the earliest block.
  • A restarted engine knows no hash below its store's latest block, so a reorg onto a wrong fork block there is accepted; restarted_engine_rejects_a_reorg_not_building_on_a_block_below_its_disk_tip is #[ignore]d naming this issue and op-reth: proofs ExEx test coverage to merge before the reth v2.6.0 bump #23385. Fixing it means keeping every block's hash in the store (a schema change), which is out of this issue's scope.
  • The in-memory store's unwind_history subtracted one from the target without checking earliest or latest, so an unwind to block 0 panicked the persistence thread and an unwind above latest moved latest to a block it never stored.

Stacked on #23442, #23441, #23437, #23434, #23429, #23428, #23421 and #23418.

Closes #23391

🤖 Generated with Claude Code

CI follow-up: the reorg edge cases hold on both reth pins

unwind_at_a_block_the_in_flight_save_prunes_fails_after_the_save and unwind_or_reorg_at_or_below_the_earliest_block_fails_and_changes_nothing::reorg_below_the_earliest_block assumed every save prunes the store to disk tip - window. The pruner of the reth 2.6 bump with #23358 never moves the earliest block past reth's persisted head, and the simulated reth persisted nothing, so nothing was pruned and the cases failed there (the save moved the earliest block up to the target, the window must have been pruned past the anchor). The scenario now persists reth through the block each expected save prunes to before indexing (commit_blocks_toward_a_save_at), so both pruners prune to the same height, and the reference store starts at reth's database tip instead of genesis. Verified with cargo test -p reth-optimism-trie --test live reorg_revert_edges: 64 passed, 2 ignored on develop (reth 2.5.2) and on the bump evaluation tree einar/bump-eval-stacks (reth 2.6.0 + #23358).

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 31, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23391-proofs-engine-edges branch from 5c55d97 to 078bab0 Compare November 9, 2026 22:30
einar-oplabs and others added 4 commits November 10, 2026 00:08
…nd targets

Drive the proofs engine's reorg, unwind and revert tasks at the boundaries the
model-based test may not reach, and add the two checks they found missing: an
unwind now refuses a target at or below the window's earliest block, and one
whose parent differs from the stored block below it, before touching anything.
The engine buffer keeps the hashes of the blocks it persisted so a fork below
the disk tip can be verified within a process lifetime.

`EngineHandle::reorg` takes `ReorgBlock`s, each indexed from reth's trie data or
executed through the EVM, and the ExEx picks per block with the rule its
ChainCommitted path uses instead of failing a reorg without trie data. The
in-memory store's `unwind_history` follows the trait contract: no-op above the
latest block, an error at or below the earliest, no underflow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 743644a)
(cherry picked from commit 23f2d2dda1f5f8dabec479c7ee6fd8190848100e)
Run every block of a reorg through the same code as index_block and
execute_block, so an indexed block cannot land on a gap left by an executed
block the engine had to skip, and cover a mixed reorg. Pin that the unwind
checks run against the window as an in-flight save leaves it with a pruned
window case, name the in-flight cases by the end state they assert, bound the
wait on a held write, drop the wait in PersistenceState::unwind that its only
caller now does first, log the two rejections, share the index helper and the
backend macro across the live tests, and add a table for the ExEx's trie-data
rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit c7bb2d6)
(cherry picked from commit afb5bb8730ad4a83299cc6c02018c62024973786)
…est block

The reorg and revert edge cases expect every save to prune the store to
`disk tip - window`. A pruner that holds the window back at reth's persisted
head (the reth 2.6 pin's, #23358) pruned nothing, because the simulated reth
never persisted: the cases that unwind at or below the earliest block found
the window unpruned, and the unwind at the block an in-flight save prunes
went through.

The scenario now persists reth through the block each expected save prunes
the window to before it indexes the blocks, so that both pruners prune to the
same height, and the reference store starts at reth's database tip, the
oldest state reth's database holds, instead of genesis.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 5c55d97)
(cherry picked from commit fdf9fad8bfca21fbbcdd30755aeb5d6e52343923)
The ExEx keeps alloy-eips as a regular dependency, the sorted trie types the
bump dropped leave the top-level import, and the bump's canonical-block test
helper takes a name that does not collide with the reorg tests' make_canonical.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23391-proofs-engine-edges branch from 078bab0 to 38e8eca Compare November 9, 2026 23:25
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: tests for proofs engine reorg and revert edge cases

3 participants