Repository navigation
op-reth: liveness test of the proofs engine under a slow store - #23455
Closed
einar-oplabs wants to merge 2 commits into
Closed
einar-oplabs wants to merge 2 commits into
einar-oplabs wants to merge 2 commits into
Conversation
einar-oplabs
force-pushed
the
einar/23381-proofs-engine-model
branch
from
October 15, 2026 16:08
4838120 to
f299c3a
Compare
einar-oplabs
force-pushed
the
einar/23392-proofs-slow-store
branch
from
October 15, 2026 16:08
e3107ed to
a45bb5e
Compare
einar-oplabs
force-pushed
the
einar/23381-proofs-engine-model
branch
from
October 15, 2026 16:37
f299c3a to
2383bbd
Compare
einar-oplabs
force-pushed
the
einar/23392-proofs-slow-store
branch
from
October 15, 2026 16:37
a45bb5e to
2206053
Compare
einar-oplabs
force-pushed
the
einar/23381-proofs-engine-model
branch
from
October 15, 2026 21:14
2383bbd to
53ccf4c
Compare
einar-oplabs
force-pushed
the
einar/23392-proofs-slow-store
branch
from
October 15, 2026 21:14
2206053 to
1a3eed1
Compare
einar-oplabs
force-pushed
the
einar/23381-proofs-engine-model
branch
from
October 16, 2026 04:24
e47a007 to
3f76589
Compare
einar-oplabs
force-pushed
the
einar/23392-proofs-slow-store
branch
from
October 16, 2026 04:29
1a3eed1 to
1ff0afe
Compare
1 task
Contributor
|
This pr has been automatically marked as stale and will be closed in 5 days if no updates |
einar-oplabs
force-pushed
the
einar/23381-proofs-engine-model
branch
from
November 9, 2026 22:30
3f76589 to
6e955eb
Compare
einar-oplabs
force-pushed
the
einar/23392-proofs-slow-store
branch
from
November 9, 2026 22:30
1ff0afe to
323d91a
Compare
einar-oplabs
force-pushed
the
einar/23381-proofs-engine-model
branch
from
November 9, 2026 23:25
6e955eb to
9956204
Compare
einar-oplabs
force-pushed
the
einar/23392-proofs-slow-store
branch
from
November 9, 2026 23:25
323d91a to
b691317
Compare
A new `engine_liveness` integration target drives the proofs engine over a store whose writes outlast its idle flush interval or its persistence timeout, under both flush cadences: every call returns, backpressure holds and releases, a failed save is retried, a reorg or revert under a save in flight completes or times out as the timeout says, and a dead persistence service or a panic on the engine thread reaches callers, gated ones included, as `EngineDied` instead of a hang. Engine: `EngineConfig` with `EngineHandle::spawn_with_config` exposes the idle flush interval and the persistence timeout; the engine stops once its persistence service is found gone, instead of buffering forever; and the handle waits on an exit signal the engine drops first as well as on its reply, since a caller whose action is already queued, as under backpressure, never saw the receiver drop. Tests: one `StoreWithHooks` replaces the three store wrappers of the model and live targets; `PendingCall` and the transfer-chain helpers move to shared modules. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 1ff0afe) (cherry picked from commit 976ea021ef273cd0ed55f5390911673d3da98da0)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
einar-oplabs
force-pushed
the
einar/23381-proofs-engine-model
branch
from
November 9, 2026 23:47
9956204 to
ebb44a7
Compare
einar-oplabs
force-pushed
the
einar/23392-proofs-slow-store
branch
from
November 9, 2026 23:47
b691317 to
5ad8024
Compare
einar-oplabs
marked this pull request as ready for review
November 13, 2026 11:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A liveness test of the proofs engine (
rust/op-reth/crates/trie/src/engine) over a persistence service that is slow, stalls, fails or dies, and the engine fixes it found. The engine waits for the save in flight before every unwind and gates actions under backpressure, so a slow store stalls the ExEx, and reth behind it, for as long as the engine waits; nothing covered how long that is, whether it releases, or whether a dead service surfaces at all.What the tests do
New integration target
rust/op-reth/crates/trie/tests/engine_liveness.rs(its own target so that the live target's time budget is untouched). ANodeholds theSimulatedRethof the live tests, an MDBX store behind the newStoreWithHookswrapper, and an engine spawned with the idle flush interval lowered to 100 ms and the persistence timeout to 400 ms (the defaults are 10 s and 60 s; the matrix needs a save slower than each, and cells at 60 s or more would not fit the budget). Every engine call runs on its own thread and fails the case when it does not return within 10 s (the saves in its way take at most a few seconds), so a hung engine fails the case instead of hanging it.Both slow-store tests, with every write transaction of the store opening
save_delaylate, first run through the same phases (Node::feed_under_backpressure_and_a_failed_save):index_block; every call returns.index_blockmust stay blocked for 300 ms, and return once the save is released.Then, with a save started and in flight (
Node::start_saveholds the next write until the store has it, so the save's delay runs from the unwind):engine_stays_live_over_a_slow_store(save 200 ms, longer than the idle flush interval): a reorg of reth's two newest blocks, or a revert of its newest one, returnsOkno earlier than the save could complete, and after settling the store serves reth's head.an_unwind_under_a_save_slower_than_the_persistence_timeout_times_out(save 800 ms, longer than the persistence timeout): the reorg or revert returnsPersistenceTimeoutafter at least one timeout, and the engine still answerstip(); nothing further is fed, since the engine then holds a chain the store no longer does (see findings).The death tests, over a store with no delay:
a_dead_persistence_service_reaches_callers_as_engine_died: the next write transaction panics on the persistence thread; within 10 s a block fed to the engine must come backEngineDied, which the ExEx turns into an error, instead ofOkinto a buffer that can never be flushed.a_dead_persistence_service_reaches_a_gated_caller_as_engine_died: anindex_blockgated by backpressure behind a held save, which is then released into a panic, must returnEngineDiedwithin 2 s rather than wait forever on an action the engine will never take.an_engine_panic_reaches_waiting_callers_as_engine_died: with a save held inside the store, the engine thread panics in the read viewexecute_blockopens; that call returnsEngineDied, and so must atip()issued right after, within 2 s, while the persistence thread is still inside the held save.Matrix
Save delay {200 ms, longer than the 100 ms idle flush interval; 800 ms, longer than the 400 ms persistence timeout} x engine (persistence, backpressure) thresholds {(1, 2), (5, 10)} x unwind {reorg, revert}, as rstest
#[values]axes; the three death tests run over the cadence axis. 14 cells. Store:MdbxProofsStorage; path: index (the engine's unwind and backpressure logic is shared by both paths, and the live and model targets cover execute); reth persistence threshold at its default of 7 (the engine's wait does not depend on it).Engine changes
EngineConfig(engine/mod.rs) carries the two thresholds plusidle_flush_intervalandpersistence_timeout, withEngineHandle::spawn_with_config;spawnandspawn_with_thresholdsdelegate to it,Engine::newandEngineState::newtake what they need from it, and the constants (DEFAULT_IDLE_FLUSH_INTERVAL,DEFAULT_PERSISTENCE_TIMEOUT, renamed to say so) keep their values as the defaults.advance_persistence(logged too) and went on accepting blocks into a buffer nothing would ever flush, while the ExEx kept acknowledging their heights.PersistenceStatenow records the disconnect wherever it is observed (the in-flight reply, a failed send, an unwind) andEngine::runstops once it is set. Found bya_dead_persistence_service_reaches_callers_as_engine_died:the engine kept taking blocks for 10s after the panic was armed in its persistence service.Engine's fields were droppedstatefirst, and dropping the state joins the persistence thread (ServiceGuard), so an engine thread that unwound inside a slow save kept the action channel open until the save completed; callers hung for that long (forever, for a stalled store) instead of gettingEngineDied. Found byan_engine_panic_reaches_waiting_callers_as_engine_died:tip after the engine died did not return within 2s. (b) Dropping the action channel's receiver does not reach a caller whose action is already queued: crossbeam's bounded channel keeps queued messages, and the reply sender inside each, alive until every handle is dropped (checked against crossbeam-channel 0.5.15; its unbounded flavor discards them, undocumented). The caller gated by backpressure is exactly such a caller, so a persistence thread dying under backpressure, the scenario of this issue, still hung the ExEx after the first fix. Found bya_dead_persistence_service_reaches_a_gated_caller_as_engine_died:index_block under backpressure did not return within 2s. The engine now owns aSender<()>declared first, so dropped first, when it exits by any path, andEngineHandlewaits on its reply withselect_biased!against that channel's receiver (EngineHandle::recv_reply), a reply already there winning over an exit. Both tests pass with it and fail without it.Test helpers
tests/live/store_hooks.rs:StoreWithHooks, one wrapper overOpProofsStorewith a hook before the next read, a hook before the next write and a delay on every write, plus the conveniencesfail_next_write,hold_next_write(returningHeldWrite, whoserelease_into_panickills the persistence thread mid-save),panic_in_next_readandpanic_in_next_write, andinner()for reads that must not consume an armed hook. It replacesStoreWithSaveFaults(tests/engine_model/faults.rs, deleted),StoreWithWriteGate(reorg_revert_edges.rs) andStoreWithReadHook(parent_states.rs), the follow-up op-reth: model-based test of the proofs engine state machine #23451 listed. It lives beside the other shared test modules rather than insrc/test_utils.rs: that module iscfg(test)-only, built on dev-dependencies, and nothing outside the crate's own test targets needs the wrapper.tests/live/waits.rs:PendingCall(an engine call on its own thread, withensure_blocked_forand a boundedwaitreturningCallError::{Hung, Panicked}), which the model harness'sHarness::callnow uses too, andwait_until.tests/live/chain.rs:serial_transfer_block,proof_targets_for_transfersandrecipient, moved out of the model harness so both targets share them;store_view::proof_window;SimulatedReth::spawn_engine_with_configandDEFAULT_PERSISTENCE_THRESHOLD.Findings
EngineState::unwindwaits for the save in flight up to the timeout, forgets it on expiry, sends the unwind and waits up to another timeout, then returnsPersistenceTimeout; the buffer is not unwound because the unwind failed first. The persistence thread later applies the forgotten save, the unwind and then the engine's next save of the same stale buffer, so the store ends on the old chain while reth is on the new one, and the next notification is refused withParentHashMismatch. The ExEx returns thePersistenceTimeout, reth's launch panics its critical ExEx task (ExEx proofs-history crashed: ...) and the node stops, so in production the restart reconciles it (op-reth: reconcile the proofs store with reth's chain at startup #23358); the test asserts the liveness part only (both calls return within two timeouts, the engine still answers). A follow-up should decide whether the engine stops on a persistence timeout or keeps waiting with a warning.Engine::process_next_eventcreates a freshafter(idle_flush_interval)on every pass, so a steady stream of actions faster than the interval postpones the idle flush indefinitely; in production the persistence threshold flushes long before that matters. The test polls the store, not the engine, while waiting for it.Local verification
Before the engine fixes the slow-store cells passed and the death cells failed as quoted above (the gated-caller test was added after the
rust-code-reviewerpass pointed out the queued-action gap; it failed on the field-order fix alone).Removing the backpressure gate (
incoming_rxalwaysself.incoming.clone()inEngine::process_next_event, not committed) fails every slow-store cell while the death tests stay green:Testing discussion
Behavior changed: the engine's loop ending on a dead persistence service, and callers waiting on the engine's exit as well as on their reply (with the engine's drop order making that exit precede the persistence join). Each is covered by a death test above that fails without its fix. The seam adds no behavior at the defaults (
EngineConfig::default()is the old constants); the existing unit, live and model targets pass unchanged.Stacked on #23451, #23449, #23445, #23442, #23441, #23437, #23434, #23429, #23428, #23421 and #23418.
Closes #23392
🤖 Generated with Claude Code