Skip to content

kona-node: missing L2 block during EL catch-up resets the engine in a loop #23931

Open

Description

@einar-oplabs

Part of ethereum-optimism/protocol-team#545.

Summary

When kona-node's execution client is behind its own labels, an L2 block lookup that finds nothing becomes an engine reset. After an EL datadir wipe under a running kona-node, this turns into a loop: every reset walks the sync start back to genesis and re-derives the chain, and finalization stays at genesis. op-node, driving the same EL through the same scenario, resets once and lets the EL sync. Not yet reproduced deterministically in a unit test. The evidence is #23440's CI follow-up and two local runs.

Evidence (#23440, TestFollowerResyncsFromScratchUnderConsensusClientThatKeptItsHeads/el_sync)

The follower's op-reth is stopped, its datadir wiped and restarted while its kona-node keeps running. CircleCI job 5705185 failed at the finalized wait after 621 s: expected head for label=finalized to advance to target=59, but got current=0.

  1. The first consolidation after the wipe finds no block in the empty engine (Received None block for 186, Consolidate(MissingUnsafeL2Block(186))). That error has Reset severity.
  2. The reset reads the wiped engine's labels (unsafe=0 safe=0 finalized=0). The sync-start search walks back to the finalized block, genesis (Found suitable L2 safe block l2_safe=0 is_finalized=true is_genesis=true), so derivation restarts at L1 genesis and rebuilds the chain itself instead of letting the EL sync.
  3. Whenever the derived safe head catches up with the EL's canonical tip while the gossip-fed unsafe head is ahead of it, the next consolidation finds None again and resets again. Locally this happened 93 and 94 times in four minutes; in CI, every 4 to 8 s.
  4. Each reset clears the L2Finalizer queue, so finalized stays at genesis until a finalize task lands between two resets (223 s and 282 s locally). In CI the task that landed hit BlockNotSafe and the node exited. That crash is fixed separately by kona-engine: keep delegated finalization pending during EL catch-up #23908.

The issue also noted that the follower's op-reth answered null by number for block 348 while it had received payload 361, and committed blocks in bursts of about six. That makes "unsafe ahead, block below it missing" a normal transient state during EL catch-up, not only after a wipe.

Mechanism

  • ConsolidateTask::consolidate returns MissingUnsafeL2Block when the EL has no block at the safe target's height. Its severity is Reset.
  • The reset's sync-start search stops at the EL's finalized block. After a wipe that is genesis, so the derivation pipeline restarts from L1 genesis.
  • The same path exists in delegated derivation: a delegated safe target below a gossip-advanced unsafe head, while the EL is syncing, resets too. For comparison, op-node's FollowSource treats a not-found local block below its unsafe head as "EL sync in progress" and only updates local-safe.

Expected behavior / acceptance criteria

Related

Activity

  1. self-assigned this
    on Nov 4, 2026
  2. einar-oplabs commented on Nov 9, 2026

    @einar-oplabs
    ContributorAuthor

    Draft fix for the derived-attributes path: #24006, stacked on #24005. #23900 already covers the delegated path.

    When a derived block below the unsafe head is missing, the engine resends the unchanged forkchoice:

    • SYNCING: the task waits without a reset, so the finalization queue survives. Unsafe imports keep running while it waits. If the EL's latest block stays unchanged for 300s, the engine resets anyway.
    • VALID: the block is looked up once more, and the engine resets only if it is still missing.
    • op-node difference: op-node waits only during its initial EL sync. Kona's el_sync_finished stays true after a datadir wipe, so it asks the EL instead.

    The PR says "Part of" because the datadir-wipe kona skip lives in #23440. Remove it there once both land, and rerun that cell under kona-node to verify the live recovery.

  3. einar-oplabs commented on Nov 11, 2026

    @einar-oplabs
    ContributorAuthor

    Claude: Status of #24006 (derived-attributes path) at head 1e1db47917, 2026-11-11.

    What's implemented

    • A derived block missing while the EL answers SYNCING parks the consolidation with a 2s retry; other task kinds keep running.
    • Every 30s the engine reads the EL's sync progress and probes the forkchoice. A VALID probe consolidates the block and records the EL as synced.
    • An EL idle at the block's parent gets the block built from attributes.
    • Progress is logged every 5 minutes.
    • The engine resets after 24h with no progress (--l2.el-sync-stall-timeout).

    Checks: CI is green and 767 unit tests pass.

    Open findings (Fable 5.1 xhigh, round 5)

    • Finalize tasks now run ahead of a parked consolidation. After an EL datadir wipe, a by-number finalize hits BlockNotFound, which is Critical, and the node exits.
    • An EL idle on a fork at the parent's height fails the build as Temporary every time and never reaches the stall reset.
    • The failure metric increments every 2s during a healthy wait.
    • A stall timeout of 0 still waits 30s.

    The #23440 datadir-wipe cell is still pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions