Skip to content

op-acceptance-tests,op-devstack: cover a sequencer op-reth crash with unpersisted blocks - #23431

Closed
einar-oplabs wants to merge 1 commit into
einar/23395-reth-upgrade-rollbackfrom
einar/23396-sequencer-el-kill
Closed

einar-oplabs wants to merge 1 commit into
einar/23395-reth-upgrade-rollbackfrom
einar/23396-sequencer-el-kill

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

No test killed a sequencer's execution layer, and the reth 2.6 threshold bump grows the blocks a crash loses from about 12 to about 55.

op_reth_sequencer_crash SIGKILLs an op-reth sequencer under thresholds 7 and 50 and checks the persisted gap, the rebuild, that op-reth and op-geth followers and the batcher settle on the result, and that proofs match across the crash, with the batcher running (posting through the halted head before the kill) and stopped. The batcher DSL gains FlushUntilSafe, which re-flushes until a safe head is reached because the batcher drops a flush signal when one is pending.

Stacked on #23427. Closes #23396. Local verification, findings and notes: detail comment.

Breakdown

Added Removed Net
Implementation 72 0 +72
Tests 510 0 +510
Docs 0 0 0

AI Reviews

  • <ISODATETIME> <COMMITID> Fable 5.1 xhigh
  • <ISODATETIME> <COMMITID> GPT-6.1-Sol xhigh
  • <ISODATETIME> <COMMITID> Opus 5.5 xhigh

🤖 Generated with Claude Code

@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 30, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from 75fcc02 to 0e56ee4 Compare November 9, 2026 21:31
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from d952184 to 6377ed1 Compare November 9, 2026 21:31
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from 0e56ee4 to 82fc120 Compare November 10, 2026 13:38
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from 6377ed1 to 99f634e Compare November 10, 2026 13:38
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Claude: Detail moved out of the PR description when the branch was squashed into 99f634e. It is the description as it stood before the squash, unedited, so it describes the earlier commits.


Nothing killed a sequencer's execution layer: every restart test stops it cleanly, so the blocks reth holds only in memory, up to the persistence threshold plus the memory buffer, are always written out first. The reth 2.6 bump (#23046) raises that threshold from 7 to 50, so an op-reth sequencer crash loses about 55 blocks instead of about 12, and every follower and the batcher must absorb an unsafe reorg that deep. op-acceptance-tests/tests/op_reth_sequencer_crash now kills the sequencer's op-reth with unpersisted blocks under both thresholds and checks what the system does with the gap.

What the tests cover

Mixed runtime: an op-reth sequencer without proofs history, an op-reth follower running proofs history, an op-geth follower (archive, the independent eth_getProof reference), the batcher. Both op-reth nodes run --engine.persistence-threshold={7,50} with --engine.memory-block-buffer-target=5. The workload moves balances and writes storage in nearly every block, so a rebuilt block never repeats a lost one.

Both tests kill the same way: the sequencer's consensus client is stopped, the head it leaves behind is read and the followers are waited on to hold it, then op-reth gets SIGKILL. op-reth is restarted on its datadir and the tip it persisted is read before anything feeds it; the tip must trail the head by at most threshold + buffer, lie at or above the workload's setup, and lie below the block that held the workload's last transaction (the one the kill's transaction-pool loss makes impossible to rebuild identically, which anchors the reorg checks; an empty lost block can legitimately come back byte-identical). Then op-reth is re-peered, the consensus client restarted and re-peered.

  • TestSequencerOpRethCrashKeepsPostedBlocksWithBatcherRunning: the batcher runs throughout. Between the datadir reopening and the consensus client restart the batcher is flushed, so what it had loaded lands on L1 before the sequencer is back (as it would behind any real restart; the batcher only checks its open channel's timeout when its rollup node answers, so without the flush nothing would be posted), and the followers' safe head is waited on for the block before the crash head, which the batcher is known to have loaded because its poll interval is set below the block time. Asserts: the restarted sequencer derives the posted blocks back from L1 and sequences past the crash head; both followers catch up; the chain past the crash head becomes safe on every node; on every node the lost blocks still held form a run from the tip through a height between what was seen posted and the crash head (the flushed data may still be landing when the floor is read), and everything above is replaced; the proofs follower's proofs match op-geth's from five blocks below the tip to the last block driven.
  • TestSequencerOpRethCrashRebuildsLostBlocksWithBatcherStopped: the batcher is stopped before the workload, so none of the lost blocks is on L1. Asserts: the sequencer replaces the block that held the workload's last transaction (ReorgTriggered, i.e. it resumed from the tip, not from the stale head); once the batcher is restarted, both followers reorg to that block within a bound; the chain past the crash head becomes safe; no lost block above the empty ones that can come back identical survives on any node; the proofs sweep as above.

Matrix: threshold {7, 50} as subtests, CL {op-node, kona-node} through DEVSTACK_L2CL_KIND (CI runs both), batcher {running, stopped} as the two tests. The kill heights land shortly before reth's next persistence under each threshold, where the datadir trails the head the most: at threshold 7 reth persists every three blocks from block 8, at threshold 50 at blocks 51 and 97, so the test kills around block 60 and 90 respectively.

Local verification

Run with this tree's debug op-reth and the kona-node release copied out of the shared cargo target, contracts prebuilt:

RUST_BINARY_PATH_OP_RETH=<copy of rust/target/debug/op-reth> RUST_BINARY_PATH_KONA_NODE=<copy of rust/target/release/kona-node> \
  go test -count=1 -timeout 30m -parallel 2 -run 'TestSequencerOpRethCrash' -v ./op-acceptance-tests/tests/op_reth_sequencer_crash/

op-node (default), all four cells pass, 230s for the package at -parallel 2 (an earlier run of the same cells before the review changes passed too, with the same shape of numbers):

test threshold head at kill persisted tip blocks lost (bound) lost blocks back from L1 rebuilt time
batcher running 7 60 54 6 (12) 6 (the batcher had loaded the head) 0 87s
batcher running 50 92 46 46 (55) 46 (the batcher had loaded the head) 0 119s
batcher stopped 7 63 57 6 (12) 0 6, followers reorged 96s
batcher stopped 50 162 146 16 (55) 0 16, followers reorged 110s

The last cell's sequencer had caught up about 70 blocks of genesis-to-start time with back-to-back blocks, so the kill landed at 162, and reth had deferred the persistence due at 143 to 151 (down to 146) while an earlier save was in flight; the bound absorbs both. The proofs sweep compared 30 to 70 blocks per cell, four accounts each, with no mismatch.

With DEVSTACK_L2CL_KIND=kona-node (kona-node built from this tree's rust/kona at the base commit), the batcher-stopped cells pass and the batcher-running cells fail, so TestSequencerOpRethCrashKeepsPostedBlocksWithBatcherRunning skips on kona-node naming #23386 (see findings):

test threshold head at kill persisted tip blocks lost (bound) result time
batcher running 7 62 57 5 (12) fails: op-geth follower stalls (now skipped) 231s
batcher running 50 90 46 44 (55) fails: sequencer never resumes (now skipped) 312s
batcher stopped 7 61 54 7 (12) pass, followers reorged 106s
batcher stopped 50 90 46 44 (55) pass, followers reorged 138s

The batcher-running rows are from the run before the skip was added; the two cells failed the same way on a second run (threshold 7: nodes not in sync: slower head at 73, must reach 101: unsafe; threshold 50: the same unresolvable safe label, 47 blocks lost). The batcher-stopped rows are from the final code; the package took 139s with the skips.

Corrections to the issue

  • "Produce at least 60 blocks with transactions, so the kill lands with the maximum gap" holds at threshold 7 (the datadir trails by 5 to 7 blocks wherever the kill lands) but not at threshold 50: reth first persists at block 51, down to block 46, and next at block 97, so a kill at block 60 loses 14 blocks and one near block 90 loses about 44. The test kills near block 90 under threshold 50.
  • "Assert op-node resumes sequencing from P rather than H ... before the CL reconnects": op-node re-discovers the EL's chain only on its forkchoice-change path (EngineController.tryUpdateEngineInternal resets on a SYNCING response); the sequencer's build-start path treats SYNCING as a temporary error and retries the same head. With the batcher stopped nothing moves the safe or finalized head, so an op-node left running across the EL crash would retry its stale head indefinitely. The test therefore restarts the sequencer's consensus client with its op-reth, the way the sibling restart tests do, and reads the persisted tip between the two starts. Keeping op-node up across the crash is a separate scenario worth its own issue.
  • "Both followers reorg to the rebuilt chain (ReorgTriggered / ReorgExact) within a bounded time": with the batcher running, the lost blocks the batcher had posted are derived back from L1 as they were and the sequencer continues from them, so the followers reorg only above the last posted block, typically one block, and the test asserts that split exactly instead. The deep reorg every follower must absorb is the batcher-stopped test.

Seams

  • dsl.L2Batcher.Flush asks the batcher to close its open channel and submit what it has loaded now; it does not wait for the data to land.

Follow-ups

  • The workload (alice, bob, the block recorder, driveUntilBlock, accounts) and the reth engine args are now copied in three packages (proofs_history_restart, op_reth_upgrade, this one). Once the stack has landed they belong in the DSL, e.g. a dsl state-churn workload and a sysgo.OpRethWithPersistenceSchedule(threshold, bufferTarget) option.
  • Keeping op-node up across the EL crash (the issue's "before the CL reconnects" reading) is a scenario of its own; see the second correction above for why it would stall with the batcher stopped.

go-code-reviewer was run and every finding was addressed: the kept-prefix check reads the settled chain against bounds instead of a single safe-head read, the reorg checks anchor on the block that held the workload's last transaction, the head is read after its consensus client stopped, and the batcher poll interval the landing wait relies on is set by the test. The general and security reviews were skipped because the change is test infrastructure.

Findings

  • kona-node, batcher running, threshold 50: the restarted kona-node sequencer resets once (Received reset request err=Insert(ForkchoiceUpdateFailed(InvalidForkchoiceState))), derives the posted blocks back from L1, then loops every 5s on Forkchoice update failed: server returned an error response: error code -38002: Invalid forkchoice state (target engine_builder) and Payload fetch failed: server returned an error response: error code -38005: Unsupported fork (target engine), and never builds a block. op-reth's safe label stays unresolvable the whole time, so the test fails with operation failed permanently after 90 attempts: failed to determine L2BlockRef of safe, could not get payload: not found. The datadir came back at block 46 while the pre-crash safe head was far above it; op-node handles the same state through FindL2Heads (Loaded current L2 heads unsafe=46 safe=46).
  • kona-node, batcher running, threshold 7: the restarted sequencer catches up to wall-clock time by rebuilding 58' to 81' back to back, both kona-node followers adopt that chain from gossip (the op-reth follower logs Handling reorg: unwinding and buffering new path ... common_ancestor=57 twice, once for the rebuilt chain and once when derivation brings the posted 58 to 62 back), the op-reth follower settles, but the op-geth follower's unsafe head stops at 78 and never moves: operation failed permanently after 60 attempts: nodes not in sync: slower head at 78, must reach 106: unsafe. Its kona-node logs Payload fetch failed: ... -38005: Unsupported fork every 5s from the reorg on.
  • The batcher-stopped cells pass under kona-node, so the failures need a safe head above the persisted tip or posted blocks to re-derive, which is exactly what a deeper crash gap makes more likely.
  • op-node handles both cells. After the restart it finds the EL's safe label above its tip and clamps safe to the tip (Loaded current L2 heads unsafe=46 safe=46 finalized=0); the batcher logs one sequencer currentL1 reversed wait and never clears its state.
  • Followers running op-node never adopt a gossiped block at or below their unsafe head (DropInapplicableUnsafePayloads), so with the batcher stopped they reorg only once the restarted batcher posts the rebuilt chain and derivation reaches it: about 20s here. kona-node followers adopt it from gossip right away.

Stacked on #23427, #23421 and #23418.

Closes #23396

@einar-oplabs
einar-oplabs requested a review from aakoshh November 10, 2026 14:07
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from 99f634e to bcddde7 Compare November 10, 2026 14:53
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch 2 times, most recently from 09e9bee to 44d938b Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from bcddde7 to 50ea4c8 Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from 44d938b to 604eed8 Compare November 11, 2026 13:59
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from 50ea4c8 to 05e997a Compare November 11, 2026 13:59
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from 604eed8 to fffdfb3 Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch 2 times, most recently from 772f787 to fc7871e Compare November 11, 2026 16:28
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from fffdfb3 to faa379b Compare November 11, 2026 16:28
@sebastianst

Copy link
Copy Markdown
Member

🤖 Fable says: A note from the op-geth decoupling work (#20257). The new test here starts an op-geth follower (ELKind: sysgo.MixedL2ELOpGeth).

op-geth is EOL as of Karst, and #21744 removes the in-process op-geth L2 EL from op-devstack/sysgo (l2_el_opgeth.go, MixedL2ELOpGeth, DEVSTACK_L2EL_KIND). It cannot exist after the go.mod flip to upstream go-ethereum (#20266): the same code would silently run upstream geth as the L2 EL. So please don't add new test or CI dependencies on op-geth; they would have to be removed again before the flip.

Can the follower be op-reth instead?

… unpersisted blocks

op_reth_sequencer_crash halts an op-reth sequencer once its followers
hold its head, resuming for one more workload round when the halt lands
too far past the last workload block, then SIGKILLs it under persistence
thresholds 7 and 50 with the persistence backpressure threshold pinned.
The kill height follows from reth's save schedule and where setup ended.
It reads the tip the datadir kept before the consensus client is back
and checks that the last workload block was within reth's memory buffer,
that the gap stays within the memory buffer plus the backpressure
threshold, that the sequencer rebuilds from that tip, that the followers
forget its peer entry and settle on the resulting chain with the
persisted tip canonical everywhere, that the batcher keeps submitting
until the chain past the crash is safe, and that the proofs follower
indexes the lost branch and its proofs match the op-geth follower's
across the crash. Post-crash traffic comes from freshly funded senders.

With the batcher running, the batcher posts through the halted head
before op-reth is killed, and every lost block comes back from L1. With
the batcher stopped, every lost block is rebuilt and the followers reorg
once the batcher posts the rebuilt chain. The batcher-running test skips
under kona-node naming ethereum-optimism/protocol-team#483, and the
op-geth follower runs a genesis without Karst, which op-geth
v1.101702.4-rc.2 refuses.

sysgo.OpRethWithPersistenceSchedule sets reth's memory buffer target,
persistence threshold and backpressure threshold, and the batcher DSL
gains FlushUntilSafe, which keeps flushing until a node's safe head
reaches a block, since the batcher drops a flush signal when one is
already pending.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23396-sequencer-el-kill branch from fc7871e to d5f6acf Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs force-pushed the einar/23395-reth-upgrade-rollback branch from faa379b to 3b2c3bc Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 13:18
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 13:18
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants