Repository navigation
op-acceptance-tests,op-devstack,op-reth: hold the sequencer to its block schedule across persistence runs - #23436
Conversation
|
This pr has been automatically marked as stale and will be closed in 5 days if no updates |
d952184 to
6377ed1
Compare
ae5d1e9 to
a43a3c9
Compare
6377ed1 to
99f634e
Compare
|
Claude: Detail moved out of the PR description when the branch was squashed into a43a3c9. It is the description as it stood before the squash, unedited, so it describes the earlier commits. Nothing held an op-reth sequencer to its block schedule across reth's persistence runs: the fork fix reth#26708 existed because persistence stalled payload building, and the reth 2.6 bump (#23046) replaces that mechanism with upstream's overlay snapshots, makes persistence batches about seven times larger (threshold 7 to 50) and deletes the fork's regression test What the tests cover
|
| cell | blocks held to schedule | rounds included (transactions) | largest lag seen | time |
|---|---|---|---|---|
| threshold_7-transfers | 6 to 27 | 17 (136) | 74ms | 34s |
| threshold_7-storage_writes | 7 to 28 | 18 (144) | 77ms | 35s |
| threshold_50-transfers | 6 to 156 | 130 (1040) | 80ms | 163s |
| threshold_50-storage_writes | 7 to 157 | 132 (1056) | 79ms | 164s |
About seven transactions per block; the threshold-50 storage cell created 16,896 storage slots over 150 blocks, about 5,000 per persistence run. No block was later than half a block time (the wait logs those at warning level) and no op-reth error log in any cell. The runs below were made while cargo builds for this PR ran on the same machine.
With DEVSTACK_L2CL_KIND=kona-node (kona-node release from this tree's rust/kona at the base commit), all four cells pass, 204s for the package (final code; an earlier run before the review changes passed the same way):
| cell | blocks held to schedule | rounds included (transactions) | largest lag seen | time |
|---|---|---|---|---|
| threshold_7-transfers | 6 to 27 | 17 (136) | 127ms | 38s |
| threshold_7-storage_writes | 7 to 28 | 15 (120) | 118ms | 35s |
| threshold_50-transfers | 6 to 156 | 124 (992) | 175ms | 167s |
| threshold_50-storage_writes | 7 to 157 | 126 (1008) | 128ms | 164s |
No cell needed the catch-up wait under either CL: the workload's setup covered the blocks the sequencer builds back to back after start.
Against #23046's op-reth (reth 2.6.0, built from the bump head 93d9ec9, op-node CL), all four cells pass, 202s for the package:
| cell | blocks held to schedule | rounds included (transactions) | largest lag seen | time |
|---|---|---|---|---|
| threshold_7-transfers | 10 to 31 | 18 (144) | 76ms | 38s |
| threshold_7-storage_writes | 7 to 28 | 16 (128) | 78ms | 35s |
| threshold_50-transfers | 10 to 160 | 129 (1032) | 92ms | 166s |
| threshold_50-storage_writes | 7 to 157 | 130 (1040) | 92ms | 164s |
So the verification criterion holds on both: no missed slot on develop and on #23046, under both thresholds and both loads.
Rust (cd rust && cargo test -p reth-optimism-node --test it payload_during_persistence):
test payload_during_persistence::payload_job_completes_while_persistence_batch_is_in_flight::case_1_reth_2_5_default ... ok
test payload_during_persistence::payload_job_completes_while_persistence_batch_is_in_flight::case_2_reth_2_6_default ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 20 filtered out; finished in 2.32s
Negative control (not committed): with config.engine.persistence_backpressure_threshold = Some(persistence_threshold + 1) the parked batch makes the engine backpressure at the batch tip, and both cases fail as they should with forkchoice update with attributes stalled after the 20s call bound, with the gate guard letting the node shut down (43s for both cases, no hang).
cargo clippy -p reth-optimism-node --all-targets -- -D warnings and the pinned nightly just fmt-fix are clean. (Locally, rustc 1.95 hit an incremental-compilation ICE, "unstable fingerprints for evaluate_obligation", twice while iterating on this test target; CARGO_INCREMENTAL=0 sidesteps it and CI builds are not incremental.)
Unit: go test -run TestBlockOnSchedule ./op-devstack/dsl/ passes.
Corrections to the issue
- "Assert every block's timestamp delta equals the block time and that no slot is missed": on the OP Stack the timestamp delta alone cannot catch a stall. Both op-node and kona-node always stamp a block with its parent's timestamp plus the block time; a sequencer that stalls falls behind the wall clock and then catches up with back-to-back blocks, and the timestamps never show it. The wait checks the timestamp delta as asked, and anchors "missed slot" to the wall clock: a block is late when it is not canonical before its own timestamp plus the block time, that is before the next block is due.
- "Using the hooks upstream exposes after the handoff removal": upstream exposes no persistence hook.
TempDatabase's transaction hooks wrap read transactions only. What both pins expose isNodeBuilder::with_databasewith anyDatabaseimplementation, and the persistence service's write transaction is the first thing a batch takes, so a gated database is the seam. Stated in the test's module comment.
Seams
dsl.L2ELNode.AdvancedOnSchedule/AdvancedOnScheduleFnwithScheduleOpts: holds a chain to its block schedule; the first DSL wait on block cadence.dsl.EOA.DeployStorageFiller(slotsPerCall): a contract whose every call creates that many storage slots it has never written, for loads that must grow state.- The workload (two senders, rounds off the test goroutine) is package-local like the crash and upgrade packages' workloads; it is the fourth copy of the shape and the follow-up to fold them into the DSL now has four inputs.
go-code-reviewer and rust-code-reviewer were both run and every finding addressed: the workload goroutine is now joined in a t.Cleanup so a failed schedule wait stops the load before the nodes it drives; every transaction of a round is checked, not just the last; the block time comes from the chain config rather than the test's constant, and the wait rejects a block time that is not whole seconds; the ancestor fetch is bounded and retried; the overdue arithmetic shares one boundary definition with the per-block check and has its own table test; the catch-up wait logs once per head; a block later than half a block time is logged at warning level so a CI failure shows lag climbing before the failing block. On the Rust side the batch is parked inside the real write transaction rather than before it, a timed job now spans the batch landing (the window the fork protected), the gate uses Condvar::wait_while and closing it returns the reopen guard, the watchdog thread that could not fire under nextest's 120s timeout and would have killed the sibling case under cargo test is gone, and tautological assertions were dropped.
Stacked on #23431, #23427, #23421 and #23418.
Closes #23397
99f634e to
bcddde7
Compare
b7ab15e to
47f5a25
Compare
bcddde7 to
50ea4c8
Compare
d9a8102 to
55f06d3
Compare
05e997a to
772f787
Compare
55f06d3 to
924225b
Compare
772f787 to
fc7871e
Compare
924225b to
a8c5f35
Compare
…ock schedule across persistence runs Adds an acceptance test that keeps an op-reth sequencer under a steady transaction load for three times its persistence threshold in blocks, under thresholds 7 and 50 and loads of transfers and fresh storage writes, requires a minimum number of load rounds inside the measured window, reports a stuck round, and fails on the first missed slot. The DSL wait that holds a chain to its block schedule fails a block only when a head sample that had not yet seen it was requested at or after its slot end, credits a head lookup with at most one block time of its own duration, lets the next successful sample decide after a failed lookup, and fails once lookups keep failing for more than a block time. Adds a storage-filling contract for loads that must grow state. A node-level test stalls a persistence batch at a write gate before it opens its database write and requires the next payload job, timed by the engine calls alone and built after the overlay handoff, and every forkchoice update sent while the batch is written and committed, to finish within the block time, written against NodeBuilder::with_database. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fc7871e to
d5f6acf
Compare
a8c5f35 to
055c21b
Compare
reth 2.6 replaces the fork's fix for persistence stalling payload building and makes persistence batches about seven times larger, and nothing held the sequencer to its block schedule across them.
An acceptance test runs an op-reth sequencer under steady load for three persistence thresholds and fails on the first missed slot; a node-level test stalls a persistence batch inside its database write and requires the next payload job to finish within the block time. Adds a DSL block-schedule wait and a storage-filling contract.
Stacked on #23431. Closes #23397. Local verification, findings and notes: detail comment.
Breakdown
AI Reviews
<ISODATETIME><COMMITID>Fable 5.1 xhigh<ISODATETIME><COMMITID>GPT-6.1-Sol xhigh<ISODATETIME><COMMITID>Opus 5.5 xhigh🤖 Generated with Claude Code