Repository navigation
op-acceptance-tests,op-devstack: cover a sequencer op-reth crash with unpersisted blocks - #23431
einar-oplabs wants to merge 1 commit into
Conversation
|
This pr has been automatically marked as stale and will be closed in 5 days if no updates |
75fcc02 to
0e56ee4
Compare
d952184 to
6377ed1
Compare
0e56ee4 to
82fc120
Compare
6377ed1 to
99f634e
Compare
|
Claude: Detail moved out of the PR description when the branch was squashed into 99f634e. It is the description as it stood before the squash, unedited, so it describes the earlier commits. Nothing killed a sequencer's execution layer: every restart test stops it cleanly, so the blocks reth holds only in memory, up to the persistence threshold plus the memory buffer, are always written out first. The reth 2.6 bump (#23046) raises that threshold from 7 to 50, so an op-reth sequencer crash loses about 55 blocks instead of about 12, and every follower and the batcher must absorb an unsafe reorg that deep. What the tests coverMixed runtime: an op-reth sequencer without proofs history, an op-reth follower running proofs history, an op-geth follower (archive, the independent Both tests kill the same way: the sequencer's consensus client is stopped, the head it leaves behind is read and the followers are waited on to hold it, then op-reth gets SIGKILL. op-reth is restarted on its datadir and the tip it persisted is read before anything feeds it; the tip must trail the head by at most threshold + buffer, lie at or above the workload's setup, and lie below the block that held the workload's last transaction (the one the kill's transaction-pool loss makes impossible to rebuild identically, which anchors the reorg checks; an empty lost block can legitimately come back byte-identical). Then op-reth is re-peered, the consensus client restarted and re-peered.
Matrix: threshold {7, 50} as subtests, CL {op-node, kona-node} through Local verificationRun with this tree's debug op-reth and the kona-node release copied out of the shared cargo target, contracts prebuilt: op-node (default), all four cells pass, 230s for the package at
The last cell's sequencer had caught up about 70 blocks of genesis-to-start time with back-to-back blocks, so the kill landed at 162, and reth had deferred the persistence due at 143 to 151 (down to 146) while an earlier save was in flight; the bound absorbs both. The proofs sweep compared 30 to 70 blocks per cell, four accounts each, with no mismatch. With
The batcher-running rows are from the run before the skip was added; the two cells failed the same way on a second run (threshold 7: Corrections to the issue
Seams
Follow-ups
Findings
Stacked on #23427, #23421 and #23418. Closes #23396 |
99f634e to
bcddde7
Compare
09e9bee to
44d938b
Compare
bcddde7 to
50ea4c8
Compare
44d938b to
604eed8
Compare
50ea4c8 to
05e997a
Compare
604eed8 to
fffdfb3
Compare
772f787 to
fc7871e
Compare
fffdfb3 to
faa379b
Compare
|
🤖 Fable says: A note from the op-geth decoupling work (#20257). The new test here starts an op-geth follower ( op-geth is EOL as of Karst, and #21744 removes the in-process op-geth L2 EL from op-devstack/sysgo ( Can the follower be op-reth instead? |
… unpersisted blocks op_reth_sequencer_crash halts an op-reth sequencer once its followers hold its head, resuming for one more workload round when the halt lands too far past the last workload block, then SIGKILLs it under persistence thresholds 7 and 50 with the persistence backpressure threshold pinned. The kill height follows from reth's save schedule and where setup ended. It reads the tip the datadir kept before the consensus client is back and checks that the last workload block was within reth's memory buffer, that the gap stays within the memory buffer plus the backpressure threshold, that the sequencer rebuilds from that tip, that the followers forget its peer entry and settle on the resulting chain with the persisted tip canonical everywhere, that the batcher keeps submitting until the chain past the crash is safe, and that the proofs follower indexes the lost branch and its proofs match the op-geth follower's across the crash. Post-crash traffic comes from freshly funded senders. With the batcher running, the batcher posts through the halted head before op-reth is killed, and every lost block comes back from L1. With the batcher stopped, every lost block is rebuilt and the followers reorg once the batcher posts the rebuilt chain. The batcher-running test skips under kona-node naming ethereum-optimism/protocol-team#483, and the op-geth follower runs a genesis without Karst, which op-geth v1.101702.4-rc.2 refuses. sysgo.OpRethWithPersistenceSchedule sets reth's memory buffer target, persistence threshold and backpressure threshold, and the batcher DSL gains FlushUntilSafe, which keeps flushing until a node's safe head reaches a block, since the batcher drops a flush signal when one is already pending. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fc7871e to
d5f6acf
Compare
faa379b to
3b2c3bc
Compare
No test killed a sequencer's execution layer, and the reth 2.6 threshold bump grows the blocks a crash loses from about 12 to about 55.
op_reth_sequencer_crashSIGKILLs an op-reth sequencer under thresholds 7 and 50 and checks the persisted gap, the rebuild, that op-reth and op-geth followers and the batcher settle on the result, and that proofs match across the crash, with the batcher running (posting through the halted head before the kill) and stopped. The batcher DSL gainsFlushUntilSafe, which re-flushes until a safe head is reached because the batcher drops a flush signal when one is pending.Stacked on #23427. Closes #23396. Local verification, findings and notes: detail comment.
Breakdown
AI Reviews
<ISODATETIME><COMMITID>Fable 5.1 xhigh<ISODATETIME><COMMITID>GPT-6.1-Sol xhigh<ISODATETIME><COMMITID>Opus 5.5 xhigh🤖 Generated with Claude Code