Skip to content

op-reth: test the proofs sync loop under pruning and the FinishedHeight contract - #23449

Closed
einar-oplabs wants to merge 3 commits into
einar/23391-proofs-engine-edgesfrom
einar/23382-proofs-sync-loop
Closed

einar-oplabs wants to merge 3 commits into
einar/23391-proofs-engine-edgesfrom
einar/23382-proofs-sync-loop

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Tests for the proofs catch-up path: the sync loop's retry under incoming events, catch-up while reth prunes bodies the ExEx acknowledged, and what FinishedHeight acknowledges. ethereum-optimism/protocol-team#477 and #478 had no coverage; #477 is fixed here because the fix is two lines in the runner.

Stacked on #23445, #23442, #23441, #23437, #23434, #23429, #23428, #23421 and #23418.

What the tests cover

  1. Sync retry under incoming events (trie/src/engine/runner.rs, sync_retry_is_not_postponed_by_incoming_actions). An engine over an in-memory store anchored at genesis with sync target 1, where reth reports block 1 as its head but does not hold it, fails three sync steps so the retry delay reaches 400 ms; the test then queues a SyncTo action before every step, 50 ms apart, and asserts a sync step runs within 4 s and not before the 400 ms delay has passed. The engine armed crossbeam_channel::after(backoff) on every loop iteration, so an action arriving before the timer re-armed it from scratch (geth: implement sync service spec #477). The engine now keeps sync_retry_at and arms crossbeam_channel::at(sync_retry_at); the instant only moves when a sync step fails, and a successful step still resets the delay to zero. The deadline is not cleared when an action moves the tip; nothing in the loop needs that. The test is not ignored.

  2. Catch-up under pruning (trie/tests/live/catch_up_under_pruning.rs). A store seeded at genesis, 70 blocks (more than REAL_TIME_BLOCKS_THRESHOLD) in SimulatedReth with one transfer each, engine.sync_to(70) the way the ExEx hands over a notification far behind reth's head, then five more blocks committed during the catch-up and handed to the engine along the index or execute path. reth's bodies pruner is modeled by SimulatedReth::prune_bodies_through, which deletes transaction static files exactly as reth's bodies prune segment does (prune_static_files -> delete_segment_below_block(Transactions, target + 1): whole files, never the newest), over a factory whose transaction files hold five blocks each. After a deletion reth refuses every block below its lowest remaining file with ProviderError::BlockExpired, which pruned_blocks_read_back_as_expired pins. The model is a worst case in two ways reth's pruner is gentler: it prunes right up to the acknowledged height (reth's bodies segment keeps MINIMUM_DISTANCE, 64 blocks, below the pruner's tip, and --minimal keeps 10,064) and after every acknowledgement (reth prunes after a save, gated by its block interval). The trie crate cannot see the ExEx, so the acknowledgement is modeled as a policy: NotificationTip (what the ExEx acknowledges today) or PersistedTip (the store's persisted tip, which any acknowledgement that waits for persistence stays at or below). The pruner follows the policy after every notification and while the test polls the store up to the tip. The case asserts that something was pruned, that the store's window reaches the tip and that the proofs at blocks 1, 70 and 75 hash to reth's state roots.

    • catch_up_completes_when_reth_prunes_bodies_behind_the_persisted_tip: all 8 cells pass.
    • catch_up_completes_when_reth_prunes_bodies_through_the_acknowledged_tip: #[ignore] naming [WIP] Fix: Batch submitter contract update #478; see the failure below.
    • pruned_blocks_read_back_as_expired: pins what reth serves for a pruned block.
  3. FinishedHeight contract (exex/src/lib.rs, finished_height_is_the_notification_tip). Through test_exex_context's events receiver, which the existing tests discarded, for a committed chain the engine indexed (5 blocks, saved), one scheduled for catch-up (reth's head set REAL_TIME_BLOCKS_THRESHOLD past the tip without holding the blocks) and one buffered and not flushed (3 blocks under the every-fifth-block cadence). The ExEx acknowledges the notification tip in all three cases and emits exactly one event; a comment at the assertion names [WIP] Fix: Batch submitter contract update #478 so the fix changes it deliberately. The store assertion after settling distinguishes the cases: tip for the indexed chain, genesis for the other two.

Matrix

Test reth threshold engine cadence (persistence, backpressure) proofs path cells
1 n/a n/a n/a 1
2 (persisted tip) 7, 50 (1, 2), (5, 10) index, execute 8
2 (acknowledged tip, ignored) 7, 50 (1, 2), (5, 10) index, execute 8
2 (BlockExpired pin) 7 n/a n/a 1
3 n/a (1, 2), (5, 10) index, execute 10

Test 3's buffered row only exists under the (5, 10) cadence: under (1, 2) every block starts a save, so nothing stays buffered. Test 2 runs on MDBX only: the cost of a cell is reth's persistence, where the proofs backend makes no difference, and the parity test holds the two backends to each other. The proofs path in test 2 applies to the blocks committed during the catch-up, since the catch-up itself always executes.

Local verification

cd rust
cargo test -p reth-optimism-trie --lib engine::runner
cargo test -p reth-optimism-trie --lib
cargo test -p reth-optimism-trie --test live
cargo test -p reth-optimism-exex --lib
cargo test -p reth-optimism-exex --doc --all-features
cargo clippy -p reth-optimism-trie -p reth-optimism-exex --all-targets

Results on this branch (shared target, macOS arm64):

  • cargo test -p reth-optimism-trie --lib engine::runner: 5 passed (the new test included) in 1.3 s.
  • cargo test -p reth-optimism-trie --lib: 249 passed.
  • cargo test -p reth-optimism-trie --test live: 129 passed, 16 ignored (the 8 new [WIP] Fix: Batch submitter contract update #478 cells plus the 8 pre-existing ones) in 58 s; the target took about 40 s before this PR, and the 8 new control cells account for the difference (reth's persistence every 3 blocks under threshold 7 dominates each cell, so the chain length, not the proofs side, is the cost).
  • cargo test -p reth-optimism-exex --lib: 29 passed (10 new cells) in 5.3 s, run with CARGO_INCREMENTAL=0 after a rustc incremental ICE in reth-optimism-node.
  • cargo test -p reth-optimism-exex --doc --all-features: 1 passed.
  • cargo clippy -p reth-optimism-trie -p reth-optimism-exex --all-targets: no warnings.
  • cargo test -p reth-optimism-trie --test live 'catch_up_under_pruning::catch_up_completes_when_reth_prunes_bodies_through_the_acknowledged_tip::persistence_1_RethPersistence__threshold_7_::cadence_1_FlushCadence__EVERY_BLOCK::path_1_ProofsPath__Index' -- --ignored --exact: fails as described under findings.

Findings

  • geth: implement sync service spec #477 reproduced and fixed. Before the fix, the runner test (then with a 1.6 s deadline) failed with: no sync step within 1.6s while actions kept arriving every 50ms. With the deadline kept on the engine it passes in under a second.
  • [WIP] Fix: Batch submitter contract update #478 reproduced. One cell of the ignored test (persistence_1_RethPersistence__threshold_7_::cadence_1_FlushCadence__EVERY_BLOCK::path_1_ProofsPath__Index), run with --ignored, fails with: the store stalled at block 0 below the tip 75 after reth pruned the bodies through block Some(64). The stall block depends on how far the sync loop gets before the pruner acts on the acknowledgement (another run stalled at block 18); the outcome does not.
  • The engine's advance_sync reads bodies through recovered_block, and reth (2.5.2, this pin) answers ProviderError::BlockExpired { requested, earliest_available } for a block below its lowest remaining transactions file, so the failure surfaces as a Sync step failed error log with that provider error and a sync loop that backs off forever. Nothing reports it to the ExEx.
  • Follow-up outside this diff: the comments at engine/tasks/execute_block.rs (the BlockBodyPruned guard) and tests/live.rs (test_execute_and_store_block_updates_body_pruned) say reth serves a body-pruned block with an empty body. With static-file bodies that is not what reth does, so the guard looks unreachable from the sync path; whether some other storage layout produces empty bodies is worth checking before removing it.
  • Follow-up: the ExEx test module now has its own copy of the live tests' FlushCadence; a thresholds struct in the trie crate (behind test-utils, or replacing the two positional u64s of EngineHandle::spawn_with_thresholds) would dedupe it.
  • Inherited from op-reth: live test of proofs execution across reth parent states #23441's observation: a panic on the engine thread leaves flush/execute_block callers blocked rather than returning EngineDied; untouched here, left to op-reth: liveness test of the proofs engine under a slow store #23392.

Seams for later PRs

  • SimulatedReth::with_body_files_of(chain_spec, key_pair, persistence, blocks_per_file) builds reth over a factory whose transaction static files hold blocks_per_file blocks.
  • SimulatedReth::prune_bodies_through(number) prunes bodies the way reth's bodies segment does with number as the target and returns the newest pruned block.
  • ExEx test helpers: FlushCadence, ProofsPath, engine_over(..., cadence) now takes the cadence, set_head_far_ahead, settle.

Notes on the issue text

The issue says to deliver "one ChainCommitted at the tip so the ExEx takes the sync_to path". A notification at reth's head takes the real-time path; it is a notification 64 or more blocks behind the head that takes sync_to. Both leave the engine with the same sync target, and test 3 drives the real ExEx along the sync_to path by moving reth's head ahead of the notification. Test 2 calls sync_to directly.

CI follow-up

finished_height_is_the_notification_tip::case_4_saved_every_block::path_1_ProofsPath__Execute failed on two stacked runs (CircleCI job 5710450) with handle chain commit: failed lock attempt. The cause is in the in-memory proofs store, not in the engine. Its cursors populated under RwLock::try_read, which fails while a writer holds the lock, so the engine thread executing block N+1 (which reads the store's hashed-account and trie cursors through OpProofsStateProviderRef) collided with the persistence thread saving block N under inner.write() in store_trie_updates_batch. Only the saved-every-block cell with the execute path has a save in flight while a block executes: under the every-fifth-block cadence nothing is saved before the whole chain is buffered, and the index path never reads the store. MDBX never fails this way: provider_ro opens a read transaction and a second writer waits in tx_mut. The try_read dates from when the store used a tokio RwLock that the synchronous cursor methods could not await; op-rs/op-reth#637 switched to parking_lot and ported it as it was. Through the trie cursor path the same failure also kills the engine thread, since reth's trie walker unwraps the cursor's result, and surfaces as Collector engine terminated unexpectedly.

Fix (trie/src/in_memory.rs, trie/src/error.rs): the three cursor sites take a blocking read(), the way every other read in the store does, and OpProofsStorageError::TryLockError is gone since nothing produces it any more (it had no consumer in the monorepo or in premium). No provider method nests the lock (each takes one guard and drops it) and the persistence thread never waits on the engine thread while holding the writer, so a blocking read cannot deadlock. The rstest table test a_read_started_during_a_write_completes_once_the_write_is_done covers the four cursor kinds: a reader thread started while the test holds the write guard completes once the guard drops. It is deterministic with the fix; without the fix it only fails when the reader's try_read runs before the guard drops, which a unit test cannot force without a hook, so the proof of the regression is the loop below.

Local reproduction before the fix, with cargo test -p reth-optimism-exex --lib finished_height -- --test-threads=8 in a loop, and ten instances of the test binary in parallel for the oversubscribed loop (10 cores):

  • 20 sequential runs: 1 failure, handle chain commit: failed lock attempt.
  • 100 oversubscribed runs: 16 failures, 14 with failed lock attempt and 2 with Collector engine terminated unexpectedly after live-trie-collector panicked in reth's walker.rs:272 on Custom(TryLockError).

After the fix:

  • 20 sequential runs: 20 passed, 0 failed.
  • 100 oversubscribed runs: 100 passed, 0 failed (every instance 10/10).
  • cargo test -p reth-optimism-trie --lib in_memory: 6 passed. cargo clippy -p reth-optimism-trie -p reth-optimism-exex --all-targets: no warnings.

Closes #23382.

🤖 Generated with Claude Code

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 31, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23382-proofs-sync-loop branch from 9ba9e6c to 42a7682 Compare November 9, 2026 22:30
einar-oplabs and others added 3 commits November 10, 2026 00:09
The engine armed a fresh backoff timer on every iteration of its event
loop, so once the retry delay exceeded the gap between incoming actions
the failed sync step was never retried. The retry instant now lives on
the engine and only moves when a sync step fails. A runner test grows
the delay to 400 ms, feeds sync-target updates every 50 ms and asserts
that the step runs again within four delays.

Fixes ethereum-optimism/protocol-team#477.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit f35c5ff)
(cherry picked from commit e19e4b68e90d3f63360ee8d75e43d704f82cf169)
…t contract

An ExEx unit test pins what FinishedHeight acknowledges for a committed
chain the engine saved, scheduled for catch-up, or holds buffered: the
notification tip in every case, which ethereum-optimism/protocol-team#478
tracks, so the fix has to change the assertion deliberately.

A live test seeds a store at genesis, builds 70 blocks in reth, hands the
engine the tip as a sync target the way the ExEx does, commits five more
blocks along the index or execute path while the engine catches up, and
has reth's bodies pruner follow the acknowledged height with no distance
below it. SimulatedReth gains static files of a configurable size for
transactions and a prune_bodies_through seam that deletes them as reth's
bodies segment does; a small test pins that reth then answers
BlockExpired for a pruned block. Pruning behind the store's persisted tip
lets every cell reach the tip; pruning through the acknowledged tip
stalls the catch-up, so those cells are ignored naming #478.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 2537180)
(cherry picked from commit e2b7a9577ed65d5ee579ed50a58f79f1654f8d6c)
The in-memory proofs store populated its cursors under `try_read`, which
fails while a writer holds the lock. The engine thread reads the store
through the hashed-account and trie cursors when it executes a block, and
the persistence thread writes it under `inner.write()` when it saves the
block before, so under the every-block cadence with the execute path the
two collided and `execute_block` failed with `failed lock attempt`; when
the trie cursor took the hit, reth's trie walker unwrapped the error and
the engine thread died. That made
`finished_height_is_the_notification_tip::case_4_saved_every_block::path_1_ProofsPath__Execute`
flaky. MDBX never fails a reader against a writer, so the cursors now take
a blocking `read()` like every other read in the store, and the
`TryLockError` variant nothing produces any more is gone. The `try_read`
dates from a tokio `RwLock` the synchronous cursor methods could not await
and survived the switch to parking_lot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 9ba9e6c)
(cherry picked from commit 56c9d40fd614f1485a75e5a15274e19b8fd29af0)
@einar-oplabs
einar-oplabs force-pushed the einar/23382-proofs-sync-loop branch from 42a7682 to 3e265c3 Compare November 9, 2026 23:25
einar-oplabs added a commit that referenced this pull request Nov 10, 2026
The in-memory store's cursors took the lock with try_read, so a reader
that landed while the engine's save thread held the write lock failed
with TryLockError, which reth's trie walker unwraps. Every other access
in the store blocks, and no guard outlives a method call, so blocking
reads cannot deadlock.

Mirrors the same change on the unmerged proofs stack (#23449); the
stack's copy becomes a no-op once this merges.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: tests for the proofs sync loop under pruning and the FinishedHeight contract

3 participants