Skip to content

kona-engine,kona-node!: align reset recovery with op-node - #23548

Closed
joshklop wants to merge 2 commits into
developfrom
codex/kona-safe-reset-checkpoints
Closed

joshklop wants to merge 2 commits into
developfrom
codex/kona-safe-reset-checkpoints

Conversation

@joshklop

@joshklop joshklop commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

kona's sync-start search could keep unsafe blocks built on replaced L1 origins and chose different safe boundaries from op-node. After EL sync it also walked every L2 block back to genesis, one RPC at a time (days on a mainnet-sized chain), and then derived again from genesis, where blob data has long expired. Recovery now follows op-node's FindL2Heads: its origin, epoch and sequence checks, previous-safe and epoch-boundary limits, finality floor and reorg-depth limit, and its two early recoveries.

When EL sync leaves safe and finalized at genesis, or the execution layer was rewound below its safe or finalized label, the unsafe head stays at the synced tip and safe and finalized are placed 12h of L2 blocks behind it, like op-node's L2HeadsForELSyncWithOffset. The tip arrived over P2P and may not be batched to L1 yet, so it is never labelled safe or finalized. The new --syncmode.offset-el-safe flag sets the offset, with op-node's 12h default.

An L2 block missing while the execution layer catches up is retried, long walks log their progress, and an L1 reorg below the finalized block's origin fails with its own error.

On kona-node, TestUnsafeGapFillAfterUnsafeReorg_RestartL2CL and _RestartCLP2P now pass, because a restarted verifier drops unsafe blocks built on a replaced L1 origin, so their skips are removed. TestUnsafeGapFillAfterSafeReorg still fails on kona-node and keeps its skip.

The derivation-side reset fixes that were in this PR moved to #23858 and #23859.

Closes ethereum-optimism/protocol-team#575
Closes ethereum-optimism/protocol-team#577
Closes ethereum-optimism/protocol-team#578
Closes #22276
Closes #22277

Related to protocol-team#328.

BREAKING CHANGE: After EL sync, or when the execution layer's safe or finalized label is ahead of its chain, kona-node now sets safe and finalized 12h of L2 blocks behind the synced tip. Before, finalized stayed at genesis and safe was found by walking back from the tip. Set --syncmode.offset-el-safe to change the offset. find_starting_forkchoice takes the offset as a new argument, and DEFAULT_EL_SYNC_SAFE_OFFSET is op-node's default. Exhaustive SyncStartError matches must handle ChainViewChanged (retry with a fresh L1 view), ReorgTooDeep (recovery exceeds five sync lookback windows) and FinalizedL1OriginReplaced. BlockNotFound is now retryable.

🤖 Generated with Claude Code

@joshklop
joshklop requested a review from a team as a code owner October 21, 2026 06:18
@joshklop
joshklop force-pushed the codex/kona-safe-reset-checkpoints branch 3 times, most recently from 1f7b377 to 7d71e38 Compare October 27, 2026 04:02
@joshklop joshklop changed the title kona-engine!: retain completed safe checkpoints across resets kona-engine!: align reset recovery with op-node Oct 27, 2026
@joshklop
joshklop added this pull request to stack #23674 October 27, 2026 04:11
@joshklop
joshklop force-pushed the codex/kona-safe-reset-checkpoints branch from 3998dd1 to 0fa061e Compare October 27, 2026 17:40
@joshklop
joshklop removed this pull request from stack #23674 October 27, 2026 18:22

@teddyknox teddyknox left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has a bug that leaves a freshly EL-synced node stuck: when EL sync finishes, kona resets with the safe and finalized heads still at genesis, and the new safe head search then walks every L2 block back to genesis one RPC at a time (days on a mainnet-sized chain) before restarting derivation from genesis, where blob data has long expired.

The cause is the new ready rule, which needs cursor <= previous_safe, so the walk only stops at finality. The reset is triggered in engine/actor.rs (finalized_head() == L2BlockInfo::default() → begin_reset). op-node avoids this with the two early returns at the top of FindL2Heads: the RecoverMinSeqWindows → L2HeadsForELSyncWithOffset branch, and the recovery when the unsafe head is behind safe or finalized. Neither is ported, and the below_finalized case currently asserts a fatal error where op-node recovers. A test with finalized = safe = genesis and the unsafe head far ahead would catch it.

Smaller things:

  • The second commit is now covered by #23628, which adopts the reset head and resumes after every reset, so it can go in the rebase. The signal_applied gate might be worth a follow-up, since develop resumes even when the pipeline reset failed.
  • op-node retries every FindL2Heads error. Here everything except RpcError/ChainViewChanged is fatal. That's fine for ReorgTooDeep and the epoch checks, but BlockNotFound on an L2 parent (from a pruned or still-syncing EL) is now reachable over a much longer walk.

@joshklop
joshklop force-pushed the codex/kona-safe-reset-checkpoints branch 3 times, most recently from 667e463 to 1747cbd Compare November 3, 2026 02:40
kona's sync-start search could keep unsafe blocks built on replaced L1
origins, and chose different safe boundaries from op-node. After EL sync it
also walked every L2 block back to genesis, one RPC at a time, before
deriving again from genesis.

Port op-node's FindL2Heads:
- the walk's plausible-origin, epoch and sequence checks, previous-safe and
  epoch-boundary limits, finality floor and maximum reorg depth;
- the early recovery when EL sync leaves safe and finalized at genesis on a
  chain older than fourteen sequencing windows, or the execution layer was
  rewound below its safe or finalized label. Like op-node's
  L2HeadsForELSyncWithOffset, the unsafe head stays at the synced tip and
  safe and finalized are placed an offset behind it. The tip arrived over
  P2P and may not be batched yet, so it is never labelled safe or final.

The offset is the new --syncmode.offset-el-safe flag
(KONA_NODE_SYNCMODE_OFFSET_EL_SAFE), which defaults to 12h like op-node's.

Retry when a canonical L1 parent vanishes during the walk, and when an L2
block is missing while the execution layer catches up. Log progress during
long walks. A replaced L1 origin at the finalized block is reported as its
own error.

Related to ethereum-optimism/protocol-team#328.

BREAKING CHANGE: After EL sync, or when the execution layer's safe or
finalized label is ahead of its chain, kona-node now sets safe and
finalized 12h of L2 blocks behind the synced tip. Before, finalized stayed
at genesis and safe was found by walking back from the tip. Set
--syncmode.offset-el-safe to change the offset.

find_starting_forkchoice takes the offset as a new argument;
DEFAULT_EL_SYNC_SAFE_OFFSET matches op-node's default. Exhaustive
SyncStartError matches must handle ChainViewChanged (retry with a fresh L1
view), ReorgTooDeep (recovery exceeds five sync lookback windows) and
FinalizedL1OriginReplaced. BlockNotFound is now treated as retryable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop force-pushed the codex/kona-safe-reset-checkpoints branch from 1747cbd to 5bdce91 Compare November 3, 2026 03:28
@joshklop joshklop changed the title kona-engine!: align reset recovery with op-node kona-engine,kona-node!: align reset recovery with op-node Nov 3, 2026
TestUnsafeGapFillAfterUnsafeReorg_RestartL2CL and _RestartCLP2P failed on
kona-node because a restarted verifier kept unsafe blocks built on a
replaced L1 origin. The op-node FindL2Heads port in this branch drops them,
and both tests now pass on kona-node.

TestUnsafeGapFillAfterSafeReorg still fails on kona-node and keeps its skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop force-pushed the codex/kona-safe-reset-checkpoints branch from a1f3427 to ae5aef3 Compare November 3, 2026 04:28
@joshklop

joshklop commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of smaller PRs, one per root cause (tracked in ethereum-optimism/protocol-team#589, #590 and #588):

@joshklop joshklop closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants