Repository navigation
op-reth: test the proofs sync loop under pruning and the FinishedHeight contract - #23449
Closed
einar-oplabs wants to merge 3 commits into
Closed
einar-oplabs wants to merge 3 commits into
einar-oplabs wants to merge 3 commits into
Conversation
This was referenced Oct 15, 2026
einar-oplabs
force-pushed
the
einar/23382-proofs-sync-loop
branch
from
October 15, 2026 16:08
7c931b9 to
df1d7b4
Compare
einar-oplabs
force-pushed
the
einar/23382-proofs-sync-loop
branch
from
October 15, 2026 16:37
df1d7b4 to
86a72c1
Compare
einar-oplabs
force-pushed
the
einar/23382-proofs-sync-loop
branch
from
October 16, 2026 04:24
6d4965f to
9ba9e6c
Compare
Contributor
|
This pr has been automatically marked as stale and will be closed in 5 days if no updates |
einar-oplabs
force-pushed
the
einar/23382-proofs-sync-loop
branch
from
November 9, 2026 22:30
9ba9e6c to
42a7682
Compare
The engine armed a fresh backoff timer on every iteration of its event loop, so once the retry delay exceeded the gap between incoming actions the failed sync step was never retried. The retry instant now lives on the engine and only moves when a sync step fails. A runner test grows the delay to 400 ms, feeds sync-target updates every 50 ms and asserts that the step runs again within four delays. Fixes ethereum-optimism/protocol-team#477. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit f35c5ff) (cherry picked from commit e19e4b68e90d3f63360ee8d75e43d704f82cf169)
…t contract An ExEx unit test pins what FinishedHeight acknowledges for a committed chain the engine saved, scheduled for catch-up, or holds buffered: the notification tip in every case, which ethereum-optimism/protocol-team#478 tracks, so the fix has to change the assertion deliberately. A live test seeds a store at genesis, builds 70 blocks in reth, hands the engine the tip as a sync target the way the ExEx does, commits five more blocks along the index or execute path while the engine catches up, and has reth's bodies pruner follow the acknowledged height with no distance below it. SimulatedReth gains static files of a configurable size for transactions and a prune_bodies_through seam that deletes them as reth's bodies segment does; a small test pins that reth then answers BlockExpired for a pruned block. Pruning behind the store's persisted tip lets every cell reach the tip; pruning through the acknowledged tip stalls the catch-up, so those cells are ignored naming #478. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 2537180) (cherry picked from commit e2b7a9577ed65d5ee579ed50a58f79f1654f8d6c)
The in-memory proofs store populated its cursors under `try_read`, which fails while a writer holds the lock. The engine thread reads the store through the hashed-account and trie cursors when it executes a block, and the persistence thread writes it under `inner.write()` when it saves the block before, so under the every-block cadence with the execute path the two collided and `execute_block` failed with `failed lock attempt`; when the trie cursor took the hit, reth's trie walker unwrapped the error and the engine thread died. That made `finished_height_is_the_notification_tip::case_4_saved_every_block::path_1_ProofsPath__Execute` flaky. MDBX never fails a reader against a writer, so the cursors now take a blocking `read()` like every other read in the store, and the `TryLockError` variant nothing produces any more is gone. The `try_read` dates from a tokio `RwLock` the synchronous cursor methods could not await and survived the switch to parking_lot. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 9ba9e6c) (cherry picked from commit 56c9d40fd614f1485a75e5a15274e19b8fd29af0)
einar-oplabs
force-pushed
the
einar/23382-proofs-sync-loop
branch
from
November 9, 2026 23:25
42a7682 to
3e265c3
Compare
einar-oplabs
added a commit
that referenced
this pull request
Nov 10, 2026
The in-memory store's cursors took the lock with try_read, so a reader that landed while the engine's save thread held the write lock failed with TryLockError, which reth's trie walker unwraps. Every other access in the store blocks, and no guard outlives a method call, so blocking reads cannot deadlock. Mirrors the same change on the unmerged proofs stack (#23449); the stack's copy becomes a no-op once this merges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2 tasks
einar-oplabs
marked this pull request as ready for review
November 13, 2026 11:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tests for the proofs catch-up path: the sync loop's retry under incoming events, catch-up while reth prunes bodies the ExEx acknowledged, and what
FinishedHeightacknowledges. ethereum-optimism/protocol-team#477 and #478 had no coverage; #477 is fixed here because the fix is two lines in the runner.Stacked on #23445, #23442, #23441, #23437, #23434, #23429, #23428, #23421 and #23418.
What the tests cover
Sync retry under incoming events (
trie/src/engine/runner.rs,sync_retry_is_not_postponed_by_incoming_actions). An engine over an in-memory store anchored at genesis with sync target 1, where reth reports block 1 as its head but does not hold it, fails three sync steps so the retry delay reaches 400 ms; the test then queues aSyncToaction before every step, 50 ms apart, and asserts a sync step runs within 4 s and not before the 400 ms delay has passed. The engine armedcrossbeam_channel::after(backoff)on every loop iteration, so an action arriving before the timer re-armed it from scratch (geth: implement sync service spec #477). The engine now keepssync_retry_atand armscrossbeam_channel::at(sync_retry_at); the instant only moves when a sync step fails, and a successful step still resets the delay to zero. The deadline is not cleared when an action moves the tip; nothing in the loop needs that. The test is not ignored.Catch-up under pruning (
trie/tests/live/catch_up_under_pruning.rs). A store seeded at genesis, 70 blocks (more thanREAL_TIME_BLOCKS_THRESHOLD) inSimulatedRethwith one transfer each,engine.sync_to(70)the way the ExEx hands over a notification far behind reth's head, then five more blocks committed during the catch-up and handed to the engine along the index or execute path. reth's bodies pruner is modeled bySimulatedReth::prune_bodies_through, which deletes transaction static files exactly as reth's bodies prune segment does (prune_static_files->delete_segment_below_block(Transactions, target + 1): whole files, never the newest), over a factory whose transaction files hold five blocks each. After a deletion reth refuses every block below its lowest remaining file withProviderError::BlockExpired, whichpruned_blocks_read_back_as_expiredpins. The model is a worst case in two ways reth's pruner is gentler: it prunes right up to the acknowledged height (reth's bodies segment keepsMINIMUM_DISTANCE, 64 blocks, below the pruner's tip, and--minimalkeeps 10,064) and after every acknowledgement (reth prunes after a save, gated by its block interval). The trie crate cannot see the ExEx, so the acknowledgement is modeled as a policy:NotificationTip(what the ExEx acknowledges today) orPersistedTip(the store's persisted tip, which any acknowledgement that waits for persistence stays at or below). The pruner follows the policy after every notification and while the test polls the store up to the tip. The case asserts that something was pruned, that the store's window reaches the tip and that the proofs at blocks 1, 70 and 75 hash to reth's state roots.catch_up_completes_when_reth_prunes_bodies_behind_the_persisted_tip: all 8 cells pass.catch_up_completes_when_reth_prunes_bodies_through_the_acknowledged_tip:#[ignore]naming [WIP] Fix: Batch submitter contract update #478; see the failure below.pruned_blocks_read_back_as_expired: pins what reth serves for a pruned block.FinishedHeightcontract (exex/src/lib.rs,finished_height_is_the_notification_tip). Throughtest_exex_context's events receiver, which the existing tests discarded, for a committed chain the engine indexed (5 blocks, saved), one scheduled for catch-up (reth's head setREAL_TIME_BLOCKS_THRESHOLDpast the tip without holding the blocks) and one buffered and not flushed (3 blocks under the every-fifth-block cadence). The ExEx acknowledges the notification tip in all three cases and emits exactly one event; a comment at the assertion names [WIP] Fix: Batch submitter contract update #478 so the fix changes it deliberately. The store assertion after settling distinguishes the cases: tip for the indexed chain, genesis for the other two.Matrix
BlockExpiredpin)Test 3's buffered row only exists under the (5, 10) cadence: under (1, 2) every block starts a save, so nothing stays buffered. Test 2 runs on MDBX only: the cost of a cell is reth's persistence, where the proofs backend makes no difference, and the parity test holds the two backends to each other. The proofs path in test 2 applies to the blocks committed during the catch-up, since the catch-up itself always executes.
Local verification
Results on this branch (shared target, macOS arm64):
cargo test -p reth-optimism-trie --lib engine::runner: 5 passed (the new test included) in 1.3 s.cargo test -p reth-optimism-trie --lib: 249 passed.cargo test -p reth-optimism-trie --test live: 129 passed, 16 ignored (the 8 new [WIP] Fix: Batch submitter contract update #478 cells plus the 8 pre-existing ones) in 58 s; the target took about 40 s before this PR, and the 8 new control cells account for the difference (reth's persistence every 3 blocks under threshold 7 dominates each cell, so the chain length, not the proofs side, is the cost).cargo test -p reth-optimism-exex --lib: 29 passed (10 new cells) in 5.3 s, run withCARGO_INCREMENTAL=0after a rustc incremental ICE inreth-optimism-node.cargo test -p reth-optimism-exex --doc --all-features: 1 passed.cargo clippy -p reth-optimism-trie -p reth-optimism-exex --all-targets: no warnings.cargo test -p reth-optimism-trie --test live 'catch_up_under_pruning::catch_up_completes_when_reth_prunes_bodies_through_the_acknowledged_tip::persistence_1_RethPersistence__threshold_7_::cadence_1_FlushCadence__EVERY_BLOCK::path_1_ProofsPath__Index' -- --ignored --exact: fails as described under findings.Findings
no sync step within 1.6s while actions kept arriving every 50ms. With the deadline kept on the engine it passes in under a second.persistence_1_RethPersistence__threshold_7_::cadence_1_FlushCadence__EVERY_BLOCK::path_1_ProofsPath__Index), run with--ignored, fails with:the store stalled at block 0 below the tip 75 after reth pruned the bodies through block Some(64). The stall block depends on how far the sync loop gets before the pruner acts on the acknowledgement (another run stalled at block 18); the outcome does not.advance_syncreads bodies throughrecovered_block, and reth (2.5.2, this pin) answersProviderError::BlockExpired { requested, earliest_available }for a block below its lowest remaining transactions file, so the failure surfaces as aSync step failederror log with that provider error and a sync loop that backs off forever. Nothing reports it to the ExEx.engine/tasks/execute_block.rs(theBlockBodyPrunedguard) andtests/live.rs(test_execute_and_store_block_updates_body_pruned) say reth serves a body-pruned block with an empty body. With static-file bodies that is not what reth does, so the guard looks unreachable from the sync path; whether some other storage layout produces empty bodies is worth checking before removing it.FlushCadence; a thresholds struct in the trie crate (behindtest-utils, or replacing the two positionalu64s ofEngineHandle::spawn_with_thresholds) would dedupe it.flush/execute_blockcallers blocked rather than returningEngineDied; untouched here, left to op-reth: liveness test of the proofs engine under a slow store #23392.Seams for later PRs
SimulatedReth::with_body_files_of(chain_spec, key_pair, persistence, blocks_per_file)builds reth over a factory whose transaction static files holdblocks_per_fileblocks.SimulatedReth::prune_bodies_through(number)prunes bodies the way reth's bodies segment does withnumberas the target and returns the newest pruned block.FlushCadence,ProofsPath,engine_over(..., cadence)now takes the cadence,set_head_far_ahead,settle.Notes on the issue text
The issue says to deliver "one
ChainCommittedat the tip so the ExEx takes thesync_topath". A notification at reth's head takes the real-time path; it is a notification 64 or more blocks behind the head that takessync_to. Both leave the engine with the same sync target, and test 3 drives the real ExEx along thesync_topath by moving reth's head ahead of the notification. Test 2 callssync_todirectly.CI follow-up
finished_height_is_the_notification_tip::case_4_saved_every_block::path_1_ProofsPath__Executefailed on two stacked runs (CircleCI job 5710450) withhandle chain commit: failed lock attempt. The cause is in the in-memory proofs store, not in the engine. Its cursors populated underRwLock::try_read, which fails while a writer holds the lock, so the engine thread executing block N+1 (which reads the store's hashed-account and trie cursors throughOpProofsStateProviderRef) collided with the persistence thread saving block N underinner.write()instore_trie_updates_batch. Only the saved-every-block cell with the execute path has a save in flight while a block executes: under the every-fifth-block cadence nothing is saved before the whole chain is buffered, and the index path never reads the store. MDBX never fails this way:provider_roopens a read transaction and a second writer waits intx_mut. Thetry_readdates from when the store used a tokioRwLockthat the synchronous cursor methods could not await; op-rs/op-reth#637 switched to parking_lot and ported it as it was. Through the trie cursor path the same failure also kills the engine thread, since reth's trie walker unwraps the cursor's result, and surfaces asCollector engine terminated unexpectedly.Fix (
trie/src/in_memory.rs,trie/src/error.rs): the three cursor sites take a blockingread(), the way every other read in the store does, andOpProofsStorageError::TryLockErroris gone since nothing produces it any more (it had no consumer in the monorepo or in premium). No provider method nests the lock (each takes one guard and drops it) and the persistence thread never waits on the engine thread while holding the writer, so a blocking read cannot deadlock. Therstesttable testa_read_started_during_a_write_completes_once_the_write_is_donecovers the four cursor kinds: a reader thread started while the test holds the write guard completes once the guard drops. It is deterministic with the fix; without the fix it only fails when the reader'stry_readruns before the guard drops, which a unit test cannot force without a hook, so the proof of the regression is the loop below.Local reproduction before the fix, with
cargo test -p reth-optimism-exex --lib finished_height -- --test-threads=8in a loop, and ten instances of the test binary in parallel for the oversubscribed loop (10 cores):handle chain commit: failed lock attempt.failed lock attemptand 2 withCollector engine terminated unexpectedlyafterlive-trie-collectorpanicked in reth'swalker.rs:272onCustom(TryLockError).After the fix:
cargo test -p reth-optimism-trie --lib in_memory: 6 passed.cargo clippy -p reth-optimism-trie -p reth-optimism-exex --all-targets: no warnings.Closes #23382.
🤖 Generated with Claude Code