Skip to content

op-acceptance-tests,op-devstack,op-reth: hold the sequencer to its block schedule across persistence runs - #23436

Closed
einar-oplabs wants to merge 1 commit into
einar/23396-sequencer-el-killfrom
einar/23397-sequencer-persistence-liveness
Closed

einar-oplabs wants to merge 1 commit into
einar/23396-sequencer-el-killfrom
einar/23397-sequencer-persistence-liveness

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

reth 2.6 replaces the fork's fix for persistence stalling payload building and makes persistence batches about seven times larger, and nothing held the sequencer to its block schedule across them.

An acceptance test runs an op-reth sequencer under steady load for three persistence thresholds and fails on the first missed slot; a node-level test stalls a persistence batch inside its database write and requires the next payload job to finish within the block time. Adds a DSL block-schedule wait and a storage-filling contract.

Stacked on #23431. Closes #23397. Local verification, findings and notes: detail comment.

Breakdown

Added Removed Net
Implementation 348 0 +348
Tests 1065 0 +1065
Docs 0 0 0

AI Reviews

  • <ISODATETIME> <COMMITID> Fable 5.1 xhigh
  • <ISODATETIME> <COMMITID> GPT-6.1-Sol xhigh
  • <ISODATETIME> <COMMITID> Opus 5.5 xhigh

🤖 Generated with Claude Code

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 29, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from d952184 to 6377ed1 Compare November 9, 2026 21:31
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch 2 times, most recently from ae5d1e9 to a43a3c9 Compare November 10, 2026 13:38
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from 6377ed1 to 99f634e Compare November 10, 2026 13:38
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Claude: Detail moved out of the PR description when the branch was squashed into a43a3c9. It is the description as it stood before the squash, unedited, so it describes the earlier commits.


Nothing held an op-reth sequencer to its block schedule across reth's persistence runs: the fork fix reth#26708 existed because persistence stalled payload building, and the reth 2.6 bump (#23046) replaces that mechanism with upstream's overlay snapshots, makes persistence batches about seven times larger (threshold 7 to 50) and deletes the fork's regression test rust/op-reth/crates/node/tests/it/payload_handoff.rs. This adds an acceptance test that keeps a sequencer under load for at least three persistence runs and fails on the first missed slot, and a node-level test that stalls a persistence batch inside the database and requires the next payload job to finish within the block time, written against hooks that survive the bump.

What the tests cover

op-acceptance-tests/tests/op_reth_sequencer_persistence

TestOpRethSequencerKeepsScheduleAcrossPersistenceRuns: an op-reth sequencer alone (no proofs history, so reth's persistence is the only one on it) on --engine.persistence-threshold={7,50} with --engine.memory-block-buffer-target=5 and 1s blocks, with the batcher the mixed runtime attaches. A package-local workload keeps it under load from two accounts off the test goroutine, in rounds of 8 transactions submitted back to back with the last one awaited: either transfers between the accounts, or calls to a contract that creates 16 storage slots it has never written per call (dsl.EOA.DeployStorageFiller), so every persistence run writes a batch of new state. The test holds the sequencer to its schedule for three times the threshold in blocks through the new dsl.L2ELNode.AdvancedOnSchedule wait, stops the load (failing if it had failed first), and requires op-reth to have logged nothing at error level meanwhile (RecordLogs from #23427).

Three times the threshold puts at least three persistence runs inside the window wherever it starts: reth persists once more than the threshold of blocks sit in memory and keeps the last 5, so a run starts every threshold+1-5 blocks (at threshold 50 at blocks 51, 97, 143, ...; at threshold 7 every three blocks).

Matrix: threshold {7, 50} x load {transfers, storage_writes} as subtests, CL {op-node, kona-node} through DEVSTACK_L2CL_KIND (CI runs both). Skips on op-geth.

dsl.L2ELNode.AdvancedOnSchedule(blocks, blockTime)

Waits for the unsafe head to be on schedule (a sequencer that starts behind its genesis time builds back-to-back blocks until it has caught up, bounded by ScheduleOpts.CatchUpTimeout), then samples the head eight times per block time until it has advanced blocks blocks, and fails as soon as a slot is missed: every new block must extend the previous head, carry the previous head's timestamp plus the block time, and be seen canonical before its own timestamp plus the block time has passed on the wall clock, which is when the next block is due; a block not seen within a further block time of being due fails the wait before it arrives. The pure check blockOnSchedule has a table test. The wait logs the largest lag it saw.

rust/op-reth/crates/node/tests/it/payload_during_persistence.rs

payload_job_completes_while_persistence_batch_is_in_flight, an rstest table over threshold {7, 50} (reth 2.5's and 2.6's defaults). It launches the op-reth node on a WriteGatedDatabase, a Database wrapper injected through NodeBuilder::with_database that delegates everything and parks every write transaction at a gate right after opening it while the gate is closed. reth's persistence service opens its write transaction (database_provider_rw) before it writes anything in on_save_blocks, so closing the gate right before a batch is due keeps that batch in flight inside its transaction with nothing committed, and the test decides when it lands. The scenario, through the engine API like a consensus client (forkchoiceUpdated with attributes, engine_getPayloadV3, engine_newPayloadV3, forkchoiceUpdated):

  1. Build threshold blocks; nothing is on disk yet.
  2. Close the gate and build one more block, which with memory_block_buffer_target = 0 makes reth persist the whole chain as one batch; wait for the persistence service to be parked.
  3. Start a payload job and require it to resolve within 2s (OP Mainnet's block time) with the batch still parked, and the disk still at genesis.
  4. Start another payload job with the batch parked, then release the batch while the job runs and wait for it to reach the disk. Send a transfer to the pool, wait for the job's best payload to be rebuilt with it (so the rebuild read state after the batch landed), and require the job to resolve within 2s and its payload to be valid under newPayload, which checks the state root the builder computed across the batch's landing. Then wait for the in-memory state to drop the batch.
  5. Build one more block on the persisted base.

A guard opens the gate when dropped, so a failed assertion cannot leave the persistence service parked through shutdown. Every engine call and wait is bounded at 20s with a message naming the stalled step.

The deleted payload_handoff.rs tested the fork's mechanism (suppress_persistence_during_build, the deferred persisted handoff, the persistence-completion metric), none of which exists after the bump. The replacement uses only NodeBuilder::with_database, the Database trait, PayloadBuilderHandle::best_payload and the in-memory state's persisted marker, all unchanged on the #23046 pin (reth rev 8d359d0), so it is the regression test that carries over. Step 4 is the window the fork protected (a batch landing while a job holds the old overlay) and the one 2.6 changes (the job keeps its own snapshot and persistence reclaims the overlay at once), covered the same way on both.

Local verification

Run with this tree's debug op-reth (built by the previous agent of the stack from the same Rust tree) and the kona-node release copied out of the shared cargo target, contracts prebuilt:

RUST_BINARY_PATH_OP_RETH=<copy of rust/target/debug/op-reth> RUST_BINARY_PATH_KONA_NODE=<copy of rust/target/release/kona-node> \
  go test -count=1 -timeout 30m -parallel 2 -run 'TestOpRethSequencerKeepsScheduleAcrossPersistenceRuns' -v ./op-acceptance-tests/tests/op_reth_sequencer_persistence/

op-node (default), all four cells pass, 234s for the package at -parallel 2 (the run of the final code; an earlier run before the review changes passed with the same shape of numbers):

cell blocks held to schedule rounds included (transactions) largest lag seen time
threshold_7-transfers 6 to 27 17 (136) 74ms 34s
threshold_7-storage_writes 7 to 28 18 (144) 77ms 35s
threshold_50-transfers 6 to 156 130 (1040) 80ms 163s
threshold_50-storage_writes 7 to 157 132 (1056) 79ms 164s

About seven transactions per block; the threshold-50 storage cell created 16,896 storage slots over 150 blocks, about 5,000 per persistence run. No block was later than half a block time (the wait logs those at warning level) and no op-reth error log in any cell. The runs below were made while cargo builds for this PR ran on the same machine.

With DEVSTACK_L2CL_KIND=kona-node (kona-node release from this tree's rust/kona at the base commit), all four cells pass, 204s for the package (final code; an earlier run before the review changes passed the same way):

cell blocks held to schedule rounds included (transactions) largest lag seen time
threshold_7-transfers 6 to 27 17 (136) 127ms 38s
threshold_7-storage_writes 7 to 28 15 (120) 118ms 35s
threshold_50-transfers 6 to 156 124 (992) 175ms 167s
threshold_50-storage_writes 7 to 157 126 (1008) 128ms 164s

No cell needed the catch-up wait under either CL: the workload's setup covered the blocks the sequencer builds back to back after start.

Against #23046's op-reth (reth 2.6.0, built from the bump head 93d9ec9, op-node CL), all four cells pass, 202s for the package:

cell blocks held to schedule rounds included (transactions) largest lag seen time
threshold_7-transfers 10 to 31 18 (144) 76ms 38s
threshold_7-storage_writes 7 to 28 16 (128) 78ms 35s
threshold_50-transfers 10 to 160 129 (1032) 92ms 166s
threshold_50-storage_writes 7 to 157 130 (1040) 92ms 164s

So the verification criterion holds on both: no missed slot on develop and on #23046, under both thresholds and both loads.

Rust (cd rust && cargo test -p reth-optimism-node --test it payload_during_persistence):

test payload_during_persistence::payload_job_completes_while_persistence_batch_is_in_flight::case_1_reth_2_5_default ... ok
test payload_during_persistence::payload_job_completes_while_persistence_batch_is_in_flight::case_2_reth_2_6_default ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 20 filtered out; finished in 2.32s

Negative control (not committed): with config.engine.persistence_backpressure_threshold = Some(persistence_threshold + 1) the parked batch makes the engine backpressure at the batch tip, and both cases fail as they should with forkchoice update with attributes stalled after the 20s call bound, with the gate guard letting the node shut down (43s for both cases, no hang).

cargo clippy -p reth-optimism-node --all-targets -- -D warnings and the pinned nightly just fmt-fix are clean. (Locally, rustc 1.95 hit an incremental-compilation ICE, "unstable fingerprints for evaluate_obligation", twice while iterating on this test target; CARGO_INCREMENTAL=0 sidesteps it and CI builds are not incremental.)

Unit: go test -run TestBlockOnSchedule ./op-devstack/dsl/ passes.

Corrections to the issue

  • "Assert every block's timestamp delta equals the block time and that no slot is missed": on the OP Stack the timestamp delta alone cannot catch a stall. Both op-node and kona-node always stamp a block with its parent's timestamp plus the block time; a sequencer that stalls falls behind the wall clock and then catches up with back-to-back blocks, and the timestamps never show it. The wait checks the timestamp delta as asked, and anchors "missed slot" to the wall clock: a block is late when it is not canonical before its own timestamp plus the block time, that is before the next block is due.
  • "Using the hooks upstream exposes after the handoff removal": upstream exposes no persistence hook. TempDatabase's transaction hooks wrap read transactions only. What both pins expose is NodeBuilder::with_database with any Database implementation, and the persistence service's write transaction is the first thing a batch takes, so a gated database is the seam. Stated in the test's module comment.

Seams

  • dsl.L2ELNode.AdvancedOnSchedule / AdvancedOnScheduleFn with ScheduleOpts: holds a chain to its block schedule; the first DSL wait on block cadence.
  • dsl.EOA.DeployStorageFiller(slotsPerCall): a contract whose every call creates that many storage slots it has never written, for loads that must grow state.
  • The workload (two senders, rounds off the test goroutine) is package-local like the crash and upgrade packages' workloads; it is the fourth copy of the shape and the follow-up to fold them into the DSL now has four inputs.

go-code-reviewer and rust-code-reviewer were both run and every finding addressed: the workload goroutine is now joined in a t.Cleanup so a failed schedule wait stops the load before the nodes it drives; every transaction of a round is checked, not just the last; the block time comes from the chain config rather than the test's constant, and the wait rejects a block time that is not whole seconds; the ancestor fetch is bounded and retried; the overdue arithmetic shares one boundary definition with the per-block check and has its own table test; the catch-up wait logs once per head; a block later than half a block time is logged at warning level so a CI failure shows lag climbing before the failing block. On the Rust side the batch is parked inside the real write transaction rather than before it, a timed job now spans the batch landing (the window the fork protected), the gate uses Condvar::wait_while and closing it returns the reopen guard, the watchdog thread that could not fire under nextest's 120s timeout and would have killed the sibling case under cargo test is gone, and tautological assertions were dropped.

Stacked on #23431, #23427, #23421 and #23418.

Closes #23397

@einar-oplabs
einar-oplabs requested a review from aakoshh November 10, 2026 14:07
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from 99f634e to bcddde7 Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch 2 times, most recently from b7ab15e to 47f5a25 Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from bcddde7 to 50ea4c8 Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch 2 times, most recently from d9a8102 to 55f06d3 Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from 05e997a to 772f787 Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from 55f06d3 to 924225b Compare November 11, 2026 15:28
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from 772f787 to fc7871e Compare November 11, 2026 16:28
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from 924225b to a8c5f35 Compare November 11, 2026 16:28
…ock schedule across persistence runs

Adds an acceptance test that keeps an op-reth sequencer under a steady
transaction load for three times its persistence threshold in blocks,
under thresholds 7 and 50 and loads of transfers and fresh storage
writes, requires a minimum number of load rounds inside the measured
window, reports a stuck round, and fails on the first missed slot. The
DSL wait that holds a chain to its block schedule fails a block only
when a head sample that had not yet seen it was requested at or after
its slot end, credits a head lookup with at most one block time of its
own duration, lets the next successful sample decide after a failed
lookup, and fails once lookups keep failing for more than a block time.
Adds a storage-filling contract for loads that must grow state.

A node-level test stalls a persistence batch at a write gate before it
opens its database write and requires the next payload job, timed by the
engine calls alone and built after the overlay handoff, and every
forkchoice update sent while the batch is written and committed, to
finish within the block time, written against
NodeBuilder::with_database.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from fc7871e to d5f6acf Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs force-pushed the einar/23397-sequencer-persistence-liveness branch from a8c5f35 to 055c21b Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 13:18
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 13:18
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants