Skip to content

op-acceptance-tests,op-devstack: cover proofs history across an unclean shutdown - #23421

Closed
einar-oplabs wants to merge 1 commit into
einar/23379-proofs-oraclefrom
einar/23380-proofs-unclean-shutdown
Closed

einar-oplabs wants to merge 1 commit into
einar/23379-proofs-oraclefrom
einar/23380-proofs-unclean-shutdown

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

After a SIGKILL an op-reth proofs-history store can hold blocks reth's datadir never persisted, and nothing tested that a node comes back serving correct proofs.

Adds an unclean op-reth stop (StopUngracefully, which fails unless the process died of the SIGKILL), per-node L2 CL options and a shared dsl.BlockRecorderWorkload, and four top-level tests (one per test and kill schedule) of four cells each (index proofs under cl_sync, el_sync, el_sync_after_finality; execute under el_sync): a follower held at an unflushed block and killed must catch up and serve the sequencer's proofs, with op-node's log showing the sync path each EL-sync cell names; and after both op-reths are killed the follower must serve the rebuilt chain's proofs (un-skipped, #23358 is in the base).

Stacked on #23418. Closes #23380. Local verification, findings and notes: detail comment.

Breakdown

Added Removed Net
Implementation 303 8 +295
Tests 609 27 +582
Docs 0 0 0

AI Reviews

  • <ISODATETIME> <COMMITID> Fable 5.1 xhigh
  • <ISODATETIME> <COMMITID> GPT-6.1-Sol xhigh
  • <ISODATETIME> <COMMITID> Opus 5.5 xhigh

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Oct 14, 2026
This was referenced Oct 14, 2026
@opgitgovernance

Copy link
Copy Markdown
Contributor

This pr has been automatically marked as stale and will be closed in 5 days if no updates

@opgitgovernance opgitgovernance added the S-stale Status: Will be closed unless there is activity label Oct 29, 2026
@einar-oplabs einar-oplabs reopened this Nov 6, 2026
@einar-oplabs einar-oplabs removed the S-stale Status: Will be closed unless there is activity label Nov 6, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from c474193 to 35282e5 Compare November 9, 2026 19:35
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 35282e5 to 2d72947 Compare November 10, 2026 13:38
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Claude: Detail moved out of the PR description when the branch was squashed into 2d72947. It is the description as it stood before the squash, unedited, so it describes the earlier commits.


Why

After a SIGKILL, OOM or power loss, an op-reth proofs-history store usually holds blocks reth's datadir has not persisted (reth persists every --engine.persistence-threshold blocks, the store every 5). Nothing reconciles the two at startup, so a node that comes back onto a chain the sequencer rebuilt serves stale eth_getProof data and then crash-loops on ParentHashMismatch (ethereum-optimism/protocol-team#464, fixed by #23358). Nothing tested the unclean-shutdown path at all; these are the acceptance tests for it.

What

op-acceptance-tests/tests/proofs_history_restart (sequencer without proofs history as the eth_getProof reference, follower with proofs history under the cell's flags, 1s L2 blocks, both op-reths with --engine.persistence-threshold and --engine.memory-block-buffer-target=5 passed explicitly):

  • TestProofsHistoryFollowerRecoversFromUncleanShutdown: traffic (transfers back and forth plus a contract that writes NUMBER/TIMESTAMP to two slots) runs until the follower's proofs store has flushed the cell's target block, the follower's op-reth is SIGKILLed at the cell's kill point and its CL stopped. On restart the test reads reth's datadir tip before the CL comes back and requires it below the store's tip (otherwise the cell is not exercising the crash gap), re-peers EL and CL, waits for the follower to be in sync with the sequencer, drives ten more blocks and sweeps VerifyHistoricalProofsMatch from the datadir tip to the last traffic block for the message passer, both EOAs and the contract's two slots.
  • TestProofsHistoryFollowerServesRebuiltBlocksAfterUncleanShutdown: same, but the batcher is stopped up front and the sequencer's op-reth is SIGKILLed right after the follower's, while its own unsafe blocks above the follower's store tip are still in memory; the CLs are stopped once both processes are gone. The restarted sequencer must have its datadir tip below the store tip and at or above the block that completed the workload's funding, and re-sequences the lost heights with different contents (ReorgTriggered on the store-tip block); only then is the follower restarted. Skipped naming protocol-team#464, see below.

Matrix, as table-driven subtests: threshold {7, 50} x proofs path {index = verification-interval=0, execute = =1} x restart sync mode {CL sync, EL sync}. The kill point is paired with the threshold instead of crossing the matrix again: threshold 7 kills mid-buffer (store tip 25, head 27, so the store loses blocks of its own on top of reth's), threshold 50 kills right after the store flushed block 60 (store tip = head, the store's lead over reth is largest). Observed datadir tips at the kill: 21 (once 19 or 20) at threshold 7, 46 at threshold 50, so the store is 4 to 6 and 14 blocks ahead of reth. The sequencer's datadir tip equalled the follower's in every cell.

op-devstack seams (for #23395, #23396 and later PRs):

  • sysgo.SubProcess.Kill(): SIGKILL with no interrupt, bounded wait for the exit (controlledKillWait), same pipe cleanup as Stop. A process that had already died on its own is reaped with a Sub-process exited before it could be killed warning rather than reported as killed, so a crash before the kill stays visible. OpReth.StopUngracefully() exposes it; Stop() stays graceful.

  • stack.UngracefulLifecycle (StopUngracefully()), forwarded by the preset l2ELFrontend, and dsl.L2ELNode.StopUngracefully() which fails the test for a node that cannot be killed (in-process op-geth).

  • sysgo.MixedSingleChainNodeSpec.L2CLOpts: per-node op-node options passed to startL2CLNode (the mixed runtime passed none before, so every follower ran CL sync). sysgo.L2CLVerifierSyncMode(mode) is the option the EL-sync cells use. Ignored for kona-node like OpRethOpts is for op-geth.

  • dsl.EOA.DeployBlockRecorder(): the NUMBER/TIMESTAMP recorder contract both proofs-history packages deploy, moved out of proofs_history_match so it is not copied a second time.

Unit tests: TestSubProcessKillEndsProcessThatIgnoresInterrupts (a sh that traps INT and TERM exits at once, no interrupt is sent, the sub-process restarts afterwards), TestSubProcessKillOfExitedProcess (warns instead of claiming a kill, second kill is a no-op), TestL2CLVerifierSyncMode.

Local verification

Debug op-reth built from this branch, DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> go test -count=1 -timeout 40m -parallel 2 -v -run <Test> ./op-acceptance-tests/tests/proofs_history_restart/:

--- PASS: TestProofsHistoryFollowerRecoversFromUncleanShutdown
    threshold_7-index-cl_sync-kill_mid_buffer      (62.01s)   datadir tip 21, store tip 25, head 27; sweep 21..47
    threshold_7-execute-cl_sync-kill_mid_buffer    (57.10s)   21 / 25 / 27; sweep 21..47
    threshold_7-index-el_sync-kill_mid_buffer      (51.78s)   19 / 25 / 27; sweep 19..45
    threshold_7-execute-el_sync-kill_mid_buffer    (51.83s)   21 / 25 / 27; sweep 21..44
    threshold_50-index-cl_sync-kill_after_flush    (92.36s)   46 / 60 / 60; sweep 46..84
    threshold_50-execute-cl_sync-kill_after_flush  (91.73s)   46 / 60 / 60; sweep 46..82
    threshold_50-index-el_sync-kill_after_flush    (92.20s)   46 / 60 / 60; sweep 46..78
    threshold_50-execute-el_sync-kill_after_flush  (87.43s)   46 / 60 / 60; sweep 46..78
ok      op-acceptance-tests/tests/proofs_history_restart        329.248s

The second test, run once with the skip removed, fails in all eight cells the same way: the sequencer rebuilds the store-tip block with a different hash, the restarted follower imports the chain up to the store's tip, and its op-reth exits on the next block:

ERROR Critical task `exex` panicked: `ExEx proofs-history crashed: Parent hash mismatch at block 26: expected 0xac2827d2d59b8cccf35d05720843825354f11528a90d04d6a2c86df9264dd9ab, got 0xfa3ba001f8ce6b5b0ea4af8668481d9501a55f9c4d0da786a78c55249234c2b5`   name=proofs-follower
ERROR shutting down due to error  err="Critical task `exex` panicked: ..."
    Error Trace: op-devstack/dsl/l2_el.go:72 ... proofs_history_restart_test.go:355
    Error:       failed to determine L2BlockRef of latest, could not get payload: read tcp ...: connection reset by peer
    Messages:    block not found using block label
--- FAIL: TestProofsHistoryFollowerServesRebuiltBlocksAfterUncleanShutdown
    threshold_7-index-cl_sync-kill_mid_buffer      (55.23s)   block 26: expected 0xac2827..4dd9ab, got 0xfa3ba0..34c2b5
    threshold_7-execute-cl_sync-kill_mid_buffer    (54.78s)   block 26
    threshold_7-index-el_sync-kill_mid_buffer      (50.90s)   block 26
    threshold_7-execute-el_sync-kill_mid_buffer    (49.20s)   block 26
    threshold_50-index-cl_sync-kill_after_flush    (85.86s)   block 61: expected 0xa8efca..28e6c1, got 0x3db150..ba7192
    threshold_50-execute-cl_sync-kill_after_flush  (88.87s)   block 61
    threshold_50-index-el_sync-kill_after_flush    (80.87s)   block 61
    threshold_50-execute-el_sync-kill_after_flush  (84.00s)   block 61
FAIL    op-acceptance-tests/tests/proofs_history_restart        281.623s

"expected" is the hash of the store's tip block on the chain that was killed, "got" the rebuilt block at the same height; the ExEx had skipped the rebuilt blocks below the tip by height. The test's own failure is the sync wait losing the follower's RPC, with the panic in the log right above it.

After the review changes (both op-reths killed back to back before the CLs stop, store tip re-read right before the kill, workload setup-block precondition, recorder contract in the DSL), one cell of each was re-run: threshold_7-index-el_sync-kill_mid_buffer of the first test passed (52s, datadir tip 20, store tip 25, head 27), threshold_50-index-cl_sync-kill_after_flush of the second, un-skipped, failed the same way (Parent hash mismatch at block 61; workload setup block 16, sequencer datadir tip 46), and proofs_history_match/threshold_7-masking_0-index passed with the moved recorder helper.

go test -count=1 ./op-devstack/sysgo/ -run 'TestSubProcess|TestL2CLVerifierSyncMode'   # ok

CI budget: with -parallel 1 the eight cells of the first test take about 10 minutes back to back on this machine, and the second test the same once #23358 un-skips it, so the package stays under the 30-minute package timeout with about 10 minutes to spare. If CI boxes turn out slower, the rebuild test can move to its own package.

Not run: the kona-node CL variant (DEVSTACK_L2CL_KIND=kona-node; no kona-node binary built locally). Under kona-node the CL-sync cells skip in code, since kona-node has no CL sync mode, and the EL-sync cells run; KonaNode has no StopUngracefully yet, the tests only kill op-reth.

Skipped cells

Findings and deviations from the issue

  • The first test passes on develop. Implement CI release pipeline #464 says the same-blocks case is harmless and the runs confirm it: the ExEx skips the re-imported blocks by height and the store's copies match. The issue expected both tests to fail; only the rebuilt-blocks one does. The first test stays unskipped as the regression test for the recovery path itself and should keep passing with op-reth: reconcile the proofs store with reth's chain at startup #23358, which re-indexes those blocks instead of skipping them.
  • The issue's "about 55 blocks lost at threshold 50 after 60 blocks" is not what reth does: with buffer target 5 it persists once in-memory blocks exceed the threshold and keeps 5, so after block 51 the datadir is at 46 and stays there until block 97. The store's lead over reth after 60 blocks is 14, which reproduces Implement CI release pipeline #464 just as well; the kill height stays at 60 as the issue asks.
  • The follower's CL is stopped gracefully alongside the EL kill and restarted after it (op-node runs in-process and cannot be killed). Reading reth's head between the EL and CL starts is what gives the test reth's datadir tip without a log hook.
  • TestResyncing in rust/op-reth/tests/proofs/core stays as the graceful variant.
  • --engine.memory-block-buffer-target is passed explicitly next to the threshold so the gap does not depend on reth's defaults (develop's reth 2.5.2 pin: threshold 7, target 5).

Stacked on #23418.

Closes #23380

@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch 2 times, most recently from a1ecc68 to af31ccb Compare November 11, 2026 12:28
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch 2 times, most recently from 8f5a66c to 272c097 Compare November 11, 2026 15:04
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 272c097 to 259913d Compare November 11, 2026 16:28
…an shutdown

SubProcess.Kill sends SIGKILL, waits a bounded time for the exit and
fails unless the process died of that signal, so a crash before a
deliberate kill fails the caller; OpReth.StopUngracefully,
stack.UngracefulLifecycle and L2ELNode.StopUngracefully expose it.
MixedSingleChainNodeSpec gains per-node L2CLOpts, with
L2CLVerifierSyncMode and L2CLPostFinalizationELSync, and kona-node
rejects options it cannot honor. The DSL gains Enode and ForgetPeer, so
a node can drop a peer's trusted entry before it dials again, and
BlockRecorderWorkload, the transfer and block-recorder load the restart,
upgrade and crash tests share.

Four top-level tests, one per test and kill schedule, run four cells
each: the index proofs path under cl_sync, el_sync (post-finalization EL
sync on) and el_sync_after_finality, and the execute path under el_sync.
A proofs-history follower is held at a block its store has not flushed
and killed there, restarted, and must catch up and serve proofs matching
the sequencer's across the gap; under op-node the EL-sync cells also
require op-node's log to show the sync path the cell names. In the
rebuilt-chain tests both op-reths are killed back to back so the lost
heights, which include the last workload block, are rebuilt with
different contents, and the follower must serve the rebuilt chain's
proofs from below the sequencer's persisted tip.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23380-proofs-unclean-shutdown branch from 259913d to 7f21b30 Compare November 12, 2026 17:25
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 13, 2026 11:44
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 13, 2026 11:44
@einar-oplabs einar-oplabs changed the title op-acceptance-tests: cover proofs history across an unclean shutdown op-acceptance-tests,op-devstack: cover proofs history across an unclean shutdown Nov 13, 2026
@joshklop
joshklop added this pull request to stack #24332 November 21, 2026 22:26
@joshklop joshklop closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

op-reth: acceptance test for proofs history across an unclean shutdown

3 participants