Skip to content

op-reth: liveness test of the proofs engine under a slow store - #23455

Closed
einar-oplabs wants to merge 2 commits into
einar/23381-proofs-engine-modelfrom
einar/23392-proofs-slow-store
Closed

einar-oplabs wants to merge 2 commits into
einar/23381-proofs-engine-modelfrom
einar/23392-proofs-slow-store

Conversation

@einar-oplabs

Copy link
Copy Markdown
Contributor

A liveness test of the proofs engine (rust/op-reth/crates/trie/src/engine) over a persistence service that is slow, stalls, fails or dies, and the engine fixes it found. The engine waits for the save in flight before every unwind and gates actions under backpressure, so a slow store stalls the ExEx, and reth behind it, for as long as the engine waits; nothing covered how long that is, whether it releases, or whether a dead service surfaces at all.

What the tests do

New integration target rust/op-reth/crates/trie/tests/engine_liveness.rs (its own target so that the live target's time budget is untouched). A Node holds the SimulatedReth of the live tests, an MDBX store behind the new StoreWithHooks wrapper, and an engine spawned with the idle flush interval lowered to 100 ms and the persistence timeout to 400 ms (the defaults are 10 s and 60 s; the matrix needs a save slower than each, and cells at 60 s or more would not fit the budget). Every engine call runs on its own thread and fails the case when it does not return within 10 s (the saves in its way take at most a few seconds), so a hung engine fails the case instead of hanging it.

Both slow-store tests, with every write transaction of the store opening save_delay late, first run through the same phases (Node::feed_under_backpressure_and_a_failed_save):

  1. Feed: 3x the backpressure threshold blocks fed back to back through index_block; every call returns.
  2. Backpressure holds: the engine's tip is at most the backpressure threshold above the store's latest block.
  3. Backpressure releases: with the next save held inside the store and the buffer filled to the threshold, one more index_block must stay blocked for 300 ms, and return once the save is released.
  4. Failed save: the next write transaction fails to open; the engine must take the next block, retry the save and catch the store up to reth's head (the idle flush does the last part, with the store slower than its interval), and the store then serves proofs at the head that hash to its state root.

Then, with a save started and in flight (Node::start_save holds the next write until the store has it, so the save's delay runs from the unwind):

  • engine_stays_live_over_a_slow_store (save 200 ms, longer than the idle flush interval): a reorg of reth's two newest blocks, or a revert of its newest one, returns Ok no earlier than the save could complete, and after settling the store serves reth's head.
  • an_unwind_under_a_save_slower_than_the_persistence_timeout_times_out (save 800 ms, longer than the persistence timeout): the reorg or revert returns PersistenceTimeout after at least one timeout, and the engine still answers tip(); nothing further is fed, since the engine then holds a chain the store no longer does (see findings).

The death tests, over a store with no delay:

  • a_dead_persistence_service_reaches_callers_as_engine_died: the next write transaction panics on the persistence thread; within 10 s a block fed to the engine must come back EngineDied, which the ExEx turns into an error, instead of Ok into a buffer that can never be flushed.
  • a_dead_persistence_service_reaches_a_gated_caller_as_engine_died: an index_block gated by backpressure behind a held save, which is then released into a panic, must return EngineDied within 2 s rather than wait forever on an action the engine will never take.
  • an_engine_panic_reaches_waiting_callers_as_engine_died: with a save held inside the store, the engine thread panics in the read view execute_block opens; that call returns EngineDied, and so must a tip() issued right after, within 2 s, while the persistence thread is still inside the held save.

Matrix

Save delay {200 ms, longer than the 100 ms idle flush interval; 800 ms, longer than the 400 ms persistence timeout} x engine (persistence, backpressure) thresholds {(1, 2), (5, 10)} x unwind {reorg, revert}, as rstest #[values] axes; the three death tests run over the cadence axis. 14 cells. Store: MdbxProofsStorage; path: index (the engine's unwind and backpressure logic is shared by both paths, and the live and model targets cover execute); reth persistence threshold at its default of 7 (the engine's wait does not depend on it).

Engine changes

  • Seam: EngineConfig (engine/mod.rs) carries the two thresholds plus idle_flush_interval and persistence_timeout, with EngineHandle::spawn_with_config; spawn and spawn_with_thresholds delegate to it, Engine::new and EngineState::new take what they need from it, and the constants (DEFAULT_IDLE_FLUSH_INTERVAL, DEFAULT_PERSISTENCE_TIMEOUT, renamed to say so) keep their values as the defaults.
  • Fix: a dead persistence service was silent. When the persistence thread died, the engine logged the disconnect, failed every later save at advance_persistence (logged too) and went on accepting blocks into a buffer nothing would ever flush, while the ExEx kept acknowledging their heights. PersistenceState now records the disconnect wherever it is observed (the in-flight reply, a failed send, an unwind) and Engine::run stops once it is set. Found by a_dead_persistence_service_reaches_callers_as_engine_died: the engine kept taking blocks for 10s after the panic was armed in its persistence service.
  • Fix: the engine's exit did not reach a caller already waiting. Two ways. (a) Engine's fields were dropped state first, and dropping the state joins the persistence thread (ServiceGuard), so an engine thread that unwound inside a slow save kept the action channel open until the save completed; callers hung for that long (forever, for a stalled store) instead of getting EngineDied. Found by an_engine_panic_reaches_waiting_callers_as_engine_died: tip after the engine died did not return within 2s. (b) Dropping the action channel's receiver does not reach a caller whose action is already queued: crossbeam's bounded channel keeps queued messages, and the reply sender inside each, alive until every handle is dropped (checked against crossbeam-channel 0.5.15; its unbounded flavor discards them, undocumented). The caller gated by backpressure is exactly such a caller, so a persistence thread dying under backpressure, the scenario of this issue, still hung the ExEx after the first fix. Found by a_dead_persistence_service_reaches_a_gated_caller_as_engine_died: index_block under backpressure did not return within 2s. The engine now owns a Sender<()> declared first, so dropped first, when it exits by any path, and EngineHandle waits on its reply with select_biased! against that channel's receiver (EngineHandle::recv_reply), a reply already there winning over an exit. Both tests pass with it and fail without it.

Test helpers

  • tests/live/store_hooks.rs: StoreWithHooks, one wrapper over OpProofsStore with a hook before the next read, a hook before the next write and a delay on every write, plus the conveniences fail_next_write, hold_next_write (returning HeldWrite, whose release_into_panic kills the persistence thread mid-save), panic_in_next_read and panic_in_next_write, and inner() for reads that must not consume an armed hook. It replaces StoreWithSaveFaults (tests/engine_model/faults.rs, deleted), StoreWithWriteGate (reorg_revert_edges.rs) and StoreWithReadHook (parent_states.rs), the follow-up op-reth: model-based test of the proofs engine state machine #23451 listed. It lives beside the other shared test modules rather than in src/test_utils.rs: that module is cfg(test)-only, built on dev-dependencies, and nothing outside the crate's own test targets needs the wrapper.
  • tests/live/waits.rs: PendingCall (an engine call on its own thread, with ensure_blocked_for and a bounded wait returning CallError::{Hung, Panicked}), which the model harness's Harness::call now uses too, and wait_until.
  • tests/live/chain.rs: serial_transfer_block, proof_targets_for_transfers and recipient, moved out of the model harness so both targets share them; store_view::proof_window; SimulatedReth::spawn_engine_with_config and DEFAULT_PERSISTENCE_THRESHOLD.

Findings

  • A save longer than the persistence timeout desynchronizes the engine from its store. On a reorg or revert, EngineState::unwind waits for the save in flight up to the timeout, forgets it on expiry, sends the unwind and waits up to another timeout, then returns PersistenceTimeout; the buffer is not unwound because the unwind failed first. The persistence thread later applies the forgotten save, the unwind and then the engine's next save of the same stale buffer, so the store ends on the old chain while reth is on the new one, and the next notification is refused with ParentHashMismatch. The ExEx returns the PersistenceTimeout, reth's launch panics its critical ExEx task (ExEx proofs-history crashed: ...) and the node stops, so in production the restart reconciles it (op-reth: reconcile the proofs store with reth's chain at startup #23358); the test asserts the liveness part only (both calls return within two timeouts, the engine still answers). A follow-up should decide whether the engine stops on a persistence timeout or keeps waiting with a warning.
  • Backpressure has no bound. A caller gated by backpressure waits for the save in flight with no timeout, so a stalled store stalls the ExEx indefinitely, by design; the test only shows that it releases once the store does.
  • The idle flush timer restarts on every event. Engine::process_next_event creates a fresh after(idle_flush_interval) on every pass, so a steady stream of actions faster than the interval postpones the idle flush indefinitely; in production the persistence threshold flushes long before that matters. The test polls the store, not the engine, while waiting for it.

Local verification

cd rust
cargo test -p reth-optimism-trie --test engine_liveness
# test result: ok. 14 passed; 0 failed; 0 ignored; finished in 14.97s  (three runs: 14.97 s, 15.01 s, 14.95 s)

cargo nextest run -p reth-optimism-trie --test engine_liveness
# Summary [  16.165s] 14 tests run: 14 passed, 0 skipped  (slowest cell 16.2 s)

cargo test -p reth-optimism-trie
# unittests 249 passed in 26.48s; engine_liveness 14 passed in 14.57s; engine_model 17 passed, 3 ignored in 13.22s;
# tests/lib 94 passed in 0.16s; live 129 passed, 16 ignored in 55.58s; doctests 1 passed, 1 ignored
# (2:19 wall including the rebuild of the test binaries; the test phases sum to about 1:50)

cargo clippy -p reth-optimism-trie -p reth-optimism-exex --all-targets --all-features -- -D warnings   # clean
cargo +nightly-2026-08-22 fmt -p reth-optimism-trie -- --check                                       # clean

Before the engine fixes the slow-store cells passed and the death cells failed as quoted above (the gated-caller test was added after the rust-code-reviewer pass pointed out the queued-action gap; it failed on the field-order fix alone).

Removing the backpressure gate (incoming_rx always self.incoming.clone() in Engine::process_next_event, not committed) fails every slow-store cell while the death tests stay green:

the engine holds 6 blocks above the store's latest block 0, more than its backpressure threshold 2
the engine holds 17 blocks above the store's latest block 13, more than its backpressure threshold 10
index_block under backpressure returned after 36.041µs while it had to stay blocked

Testing discussion

Behavior changed: the engine's loop ending on a dead persistence service, and callers waiting on the engine's exit as well as on their reply (with the engine's drop order making that exit precede the persistence join). Each is covered by a death test above that fails without its fix. The seam adds no behavior at the defaults (EngineConfig::default() is the old constants); the existing unit, live and model targets pass unchanged.

Stacked on #23451, #23449, #23445, #23442, #23441, #23437, #23434, #23429, #23428, #23421 and #23418.

Closes #23392

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Oct 15, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23381-proofs-engine-model branch from 4838120 to f299c3a Compare October 15, 2026 16:08
@einar-oplabs
einar-oplabs force-pushed the einar/23392-proofs-slow-store branch from e3107ed to a45bb5e Compare October 15, 2026 16:08
@einar-oplabs
einar-oplabs force-pushed the einar/23381-proofs-engine-model branch from f299c3a to 2383bbd Compare October 15, 2026 16:37
@einar-oplabs
einar-oplabs force-pushed the einar/23392-proofs-slow-store branch from a45bb5e to 2206053 Compare October 15, 2026 16:37
@einar-oplabs
einar-oplabs force-pushed the einar/23381-proofs-engine-model branch from 2383bbd to 53ccf4c Compare October 15, 2026 21:14
@einar-oplabs
einar-oplabs force-pushed the einar/23392-proofs-slow-store branch from 2206053 to 1a3eed1 Compare October 15, 2026 21:14
@einar-oplabs
einar-oplabs force-pushed the einar/23381-proofs-engine-model branch from e47a007 to 3f76589 Compare October 16, 2026 04:24
@einar-oplabs
einar-oplabs force-pushed the einar/23392-proofs-slow-store branch from 1a3eed1 to 1ff0afe Compare October 16, 2026 04:29
@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 31, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23381-proofs-engine-model branch from 3f76589 to 6e955eb Compare November 9, 2026 22:30
@einar-oplabs
einar-oplabs force-pushed the einar/23392-proofs-slow-store branch from 1ff0afe to 323d91a Compare November 9, 2026 22:30
@einar-oplabs
einar-oplabs force-pushed the einar/23381-proofs-engine-model branch from 6e955eb to 9956204 Compare November 9, 2026 23:25
@einar-oplabs
einar-oplabs force-pushed the einar/23392-proofs-slow-store branch from 323d91a to b691317 Compare November 9, 2026 23:25
einar-oplabs and others added 2 commits November 10, 2026 00:46
A new `engine_liveness` integration target drives the proofs engine over a
store whose writes outlast its idle flush interval or its persistence
timeout, under both flush cadences: every call returns, backpressure holds
and releases, a failed save is retried, a reorg or revert under a save in
flight completes or times out as the timeout says, and a dead persistence
service or a panic on the engine thread reaches callers, gated ones
included, as `EngineDied` instead of a hang.

Engine: `EngineConfig` with `EngineHandle::spawn_with_config` exposes the
idle flush interval and the persistence timeout; the engine stops once its
persistence service is found gone, instead of buffering forever; and the
handle waits on an exit signal the engine drops first as well as on its
reply, since a caller whose action is already queued, as under
backpressure, never saw the receiver drop.

Tests: one `StoreWithHooks` replaces the three store wrappers of the model
and live targets; `PendingCall` and the transfer-chain helpers move to
shared modules.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 1ff0afe)
(cherry picked from commit 976ea021ef273cd0ed55f5390911673d3da98da0)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23381-proofs-engine-model branch from 9956204 to ebb44a7 Compare November 9, 2026 23:47
@einar-oplabs
einar-oplabs force-pushed the einar/23392-proofs-slow-store branch from b691317 to 5ad8024 Compare November 9, 2026 23:47
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants