Repository navigation
Description
Part of ethereum-optimism/protocol-team#545.
Summary
When kona-node's execution client is behind its own labels, an L2 block lookup that finds nothing becomes an engine reset. After an EL datadir wipe under a running kona-node, this turns into a loop: every reset walks the sync start back to genesis and re-derives the chain, and finalization stays at genesis. op-node, driving the same EL through the same scenario, resets once and lets the EL sync. Not yet reproduced deterministically in a unit test. The evidence is #23440's CI follow-up and two local runs.
Evidence (#23440, TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync)
The follower's op-reth is stopped, its datadir wiped and restarted while its kona-node keeps running. CircleCI job 5705185 failed at the finalized wait after 621 s: expected head for label=finalized to advance to target=59, but got current=0.
- The first consolidation after the wipe finds no block in the empty engine (
Received None block for 186,Consolidate(MissingUnsafeL2Block(186))). That error hasResetseverity. - The reset reads the wiped engine's labels (
unsafe=0 safe=0 finalized=0). The sync-start search walks back to the finalized block, genesis (Found suitable L2 safe block l2_safe=0 is_finalized=true is_genesis=true), so derivation restarts at L1 genesis and rebuilds the chain itself instead of letting the EL sync. - Whenever the derived safe head catches up with the EL's canonical tip while the gossip-fed unsafe head is ahead of it, the next consolidation finds
Noneagain and resets again. Locally this happened 93 and 94 times in four minutes; in CI, every 4 to 8 s. - Each reset clears the
L2Finalizerqueue, so finalized stays at genesis until a finalize task lands between two resets (223 s and 282 s locally). In CI the task that landed hitBlockNotSafeand the node exited. That crash is fixed separately by kona-engine: keep delegated finalization pending during EL catch-up #23908.
The issue also noted that the follower's op-reth answered null by number for block 348 while it had received payload 361, and committed blocks in bursts of about six. That makes "unsafe ahead, block below it missing" a normal transient state during EL catch-up, not only after a wipe.
Mechanism
ConsolidateTask::consolidatereturnsMissingUnsafeL2Blockwhen the EL has no block at the safe target's height. Its severity isReset.- The reset's sync-start search stops at the EL's finalized block. After a wipe that is genesis, so the derivation pipeline restarts from L1 genesis.
- The same path exists in delegated derivation: a delegated safe target below a gossip-advanced unsafe head, while the EL is syncing, resets too. For comparison, op-node's
FollowSourcetreats a not-found local block below its unsafe head as "EL sync in progress" and only updates local-safe.
Expected behavior / acceptance criteria
- Reproduce with kona-engine tasks and mocked Engine API responses: a consolidation target below the unsafe head whose block the EL does not have yet, while the EL is syncing.
- A block missing below the unsafe head because the EL is still syncing does not reset the engine. Compare with op-node's CL-sync and follow-source handling, and decide whether kona should wait, retarget EL sync, or reset only once.
- After an EL datadir wipe under a running kona-node, the node lets the EL sync instead of re-deriving from genesis in a loop, and finalization resumes.
- op-acceptance-tests,op-devstack: cover forkchoice updates with a stale safe or finalized hash #23440's datadir-wipe cell passes under kona-node, and its kona skip (which names op-reth: acceptance tests for forkchoice updates with a stale safe or finalized hash #23399) is removed or renamed to this issue.
- Genuine divergence, where a block is missing because the chain reorged, still resets.
Related
- op-acceptance-tests,op-devstack: cover forkchoice updates with a stale safe or finalized hash #23440: the acceptance test and the CI follow-up this issue comes from.
- kona-engine: keep delegated finalization pending during EL catch-up #23908: fixes the
Finalize(BlockNotSafe)exit seen in the same run. - kona-node: verify delegated derivation parity against op-node #23912: the delegated-parity verification. Its catch-up runs can hit this loop.
- ethereum-optimism/protocol-team#574: kona treats SYNCING as applied for unsafe payloads. A related way for the unsafe head to run ahead of the EL.
Activity
kona-engine: deliver and retry delegated forkchoice updates
einar-oplabs commentedon Nov 9, 2026
Draft fix for the derived-attributes path: #24006, stacked on #24005. #23900 already covers the delegated path.
When a derived block below the unsafe head is missing, the engine resends the unchanged forkchoice:
SYNCING: the task waits without a reset, so the finalization queue survives. Unsafe imports keep running while it waits. If the EL'slatestblock stays unchanged for 300s, the engine resets anyway.VALID: the block is looked up once more, and the engine resets only if it is still missing.el_sync_finishedstays true after a datadir wipe, so it asks the EL instead.The PR says "Part of" because the datadir-wipe kona skip lives in #23440. Remove it there once both land, and rerun that cell under kona-node to verify the live recovery.
einar-oplabs commentedon Nov 11, 2026
Claude: Status of #24006 (derived-attributes path) at head
1e1db47917, 2026-11-11.What's implemented
--l2.el-sync-stall-timeout).Checks: CI is green and 767 unit tests pass.
Open findings (Fable 5.1 xhigh, round 5)
BlockNotFound, which is Critical, and the node exits.The #23440 datadir-wipe cell is still pending.
kona-engine: wait for EL sync instead of resetting on a missing conso…
kona-engine: wait for EL sync instead of resetting on a missing conso…