Repository navigation
Conversation
1f7b377 to
7d71e38
Compare
3998dd1 to
0fa061e
Compare
teddyknox
left a comment
There was a problem hiding this comment.
This has a bug that leaves a freshly EL-synced node stuck: when EL sync finishes, kona resets with the safe and finalized heads still at genesis, and the new safe head search then walks every L2 block back to genesis one RPC at a time (days on a mainnet-sized chain) before restarting derivation from genesis, where blob data has long expired.
The cause is the new ready rule, which needs cursor <= previous_safe, so the walk only stops at finality. The reset is triggered in engine/actor.rs (finalized_head() == L2BlockInfo::default() → begin_reset). op-node avoids this with the two early returns at the top of FindL2Heads: the RecoverMinSeqWindows → L2HeadsForELSyncWithOffset branch, and the recovery when the unsafe head is behind safe or finalized. Neither is ported, and the below_finalized case currently asserts a fatal error where op-node recovers. A test with finalized = safe = genesis and the unsafe head far ahead would catch it.
Smaller things:
- The second commit is now covered by #23628, which adopts the reset head and resumes after every reset, so it can go in the rebase. The
signal_appliedgate might be worth a follow-up, since develop resumes even when the pipeline reset failed. - op-node retries every
FindL2Headserror. Here everything exceptRpcError/ChainViewChangedis fatal. That's fine forReorgTooDeepand the epoch checks, butBlockNotFoundon an L2 parent (from a pruned or still-syncing EL) is now reachable over a much longer walk.
667e463 to
1747cbd
Compare
kona's sync-start search could keep unsafe blocks built on replaced L1 origins, and chose different safe boundaries from op-node. After EL sync it also walked every L2 block back to genesis, one RPC at a time, before deriving again from genesis. Port op-node's FindL2Heads: - the walk's plausible-origin, epoch and sequence checks, previous-safe and epoch-boundary limits, finality floor and maximum reorg depth; - the early recovery when EL sync leaves safe and finalized at genesis on a chain older than fourteen sequencing windows, or the execution layer was rewound below its safe or finalized label. Like op-node's L2HeadsForELSyncWithOffset, the unsafe head stays at the synced tip and safe and finalized are placed an offset behind it. The tip arrived over P2P and may not be batched yet, so it is never labelled safe or final. The offset is the new --syncmode.offset-el-safe flag (KONA_NODE_SYNCMODE_OFFSET_EL_SAFE), which defaults to 12h like op-node's. Retry when a canonical L1 parent vanishes during the walk, and when an L2 block is missing while the execution layer catches up. Log progress during long walks. A replaced L1 origin at the finalized block is reported as its own error. Related to ethereum-optimism/protocol-team#328. BREAKING CHANGE: After EL sync, or when the execution layer's safe or finalized label is ahead of its chain, kona-node now sets safe and finalized 12h of L2 blocks behind the synced tip. Before, finalized stayed at genesis and safe was found by walking back from the tip. Set --syncmode.offset-el-safe to change the offset. find_starting_forkchoice takes the offset as a new argument; DEFAULT_EL_SYNC_SAFE_OFFSET matches op-node's default. Exhaustive SyncStartError matches must handle ChainViewChanged (retry with a fresh L1 view), ReorgTooDeep (recovery exceeds five sync lookback windows) and FinalizedL1OriginReplaced. BlockNotFound is now treated as retryable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1747cbd to
5bdce91
Compare
TestUnsafeGapFillAfterUnsafeReorg_RestartL2CL and _RestartCLP2P failed on kona-node because a restarted verifier kept unsafe blocks built on a replaced L1 origin. The op-node FindL2Heads port in this branch drops them, and both tests now pass on kona-node. TestUnsafeGapFillAfterSafeReorg still fails on kona-node and keeps its skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a1f3427 to
ae5aef3
Compare
|
Closing in favor of smaller PRs, one per root cause (tracked in ethereum-optimism/protocol-team#589, #590 and #588):
|
kona's sync-start search could keep unsafe blocks built on replaced L1 origins and chose different safe boundaries from op-node. After EL sync it also walked every L2 block back to genesis, one RPC at a time (days on a mainnet-sized chain), and then derived again from genesis, where blob data has long expired. Recovery now follows op-node's
FindL2Heads: its origin, epoch and sequence checks, previous-safe and epoch-boundary limits, finality floor and reorg-depth limit, and its two early recoveries.When EL sync leaves safe and finalized at genesis, or the execution layer was rewound below its safe or finalized label, the unsafe head stays at the synced tip and safe and finalized are placed 12h of L2 blocks behind it, like op-node's
L2HeadsForELSyncWithOffset. The tip arrived over P2P and may not be batched to L1 yet, so it is never labelled safe or finalized. The new--syncmode.offset-el-safeflag sets the offset, with op-node's 12h default.An L2 block missing while the execution layer catches up is retried, long walks log their progress, and an L1 reorg below the finalized block's origin fails with its own error.
On kona-node,
TestUnsafeGapFillAfterUnsafeReorg_RestartL2CLand_RestartCLP2Pnow pass, because a restarted verifier drops unsafe blocks built on a replaced L1 origin, so their skips are removed.TestUnsafeGapFillAfterSafeReorgstill fails on kona-node and keeps its skip.The derivation-side reset fixes that were in this PR moved to #23858 and #23859.
Closes ethereum-optimism/protocol-team#575
Closes ethereum-optimism/protocol-team#577
Closes ethereum-optimism/protocol-team#578
Closes #22276
Closes #22277
Related to protocol-team#328.
BREAKING CHANGE: After EL sync, or when the execution layer's safe or finalized label is ahead of its chain, kona-node now sets safe and finalized 12h of L2 blocks behind the synced tip. Before, finalized stayed at genesis and safe was found by walking back from the tip. Set
--syncmode.offset-el-safeto change the offset.find_starting_forkchoicetakes the offset as a new argument, andDEFAULT_EL_SYNC_SAFE_OFFSETis op-node's default. ExhaustiveSyncStartErrormatches must handleChainViewChanged(retry with a fresh L1 view),ReorgTooDeep(recovery exceeds five sync lookback windows) andFinalizedL1OriginReplaced.BlockNotFoundis now retryable.🤖 Generated with Claude Code