Skip to content

kona-engine: wait for EL sync instead of resetting on a missing consolidation block - #24006

Open
einar-oplabs wants to merge 1 commit into
developfrom
einar/23931-el-sync-reset-loop
Open

einar-oplabs wants to merge 1 commit into
developfrom
einar/23931-el-sync-reset-loop

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

Kona-node resets whenever derived attributes name a block the EL lacks below the unsafe head, so after an EL datadir wipe it re-derives from genesis in a loop and finalization stays at genesis.

What

  • While the EL answers SYNCING the task waits (new Wait severity): it is parked with a 2s retry, other task kinds keep running, and nothing resets, so the finalization queue survives.
  • Every 30s it reads the EL's sync progress and resends the forkchoice, so an EL that reached the unsafe head is consolidated; an EL idle at the block's parent gets the block built from attributes.
  • It resets once progress stops for 24 h (--l2.el-sync-stall-timeout), as neither op-geth nor op-reth reports progress while downloading headers; progress is logged every 5 minutes.

Breakdown

Added Removed Net
Implementation 726 85 641
Tests 1111 56 1055
Docs 1 0 1

Closes

Part of #24100; the datadir-wipe cell that checks this lives in #23440. Targets develop directly. This PR is independent of the delegated-derivation stack (#23947 → #24004 → #24005 → #24007 → #24065) and was taken out of it.

  1. Closes: kona-node: missing L2 block during EL catch-up resets the engine in a loop #23931

AI Reviews

  • 2026-11-10T14:18:13Z ee7955c Fable 5.1 xhigh
  • 2026-11-11T13:20:26Z 0debebb Fable 5.1 xhigh
  • 2026-11-11T14:31:28Z 03d18af Fable 5.1 xhigh
  • 2026-11-11T16:38:12Z 1e1db47 Fable 5.1 xhigh

BREAKING CHANGE: EngineTaskErrorSeverity::Wait; ConsolidateTaskError::{MissingUnsafeL2BlockDuringElSync, MissingUnsafeL2BlockProbeFailed}; Engine::drain takes a DrainScope; EngineConfig gains el_sync_stall_timeout; ConsolidateTask::new is non-const and takes the stall timeout.

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Nov 9, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23911-delegated-cross-safe branch from 60e90ee to a3be2c0 Compare November 9, 2026 18:36
@einar-oplabs
einar-oplabs force-pushed the einar/23931-el-sync-reset-loop branch from 72cd709 to 6193b6a Compare November 9, 2026 18:36
@einar-oplabs
einar-oplabs force-pushed the einar/23931-el-sync-reset-loop branch from 6193b6a to ee7955c Compare November 10, 2026 14:04
@einar-oplabs
einar-oplabs force-pushed the einar/23911-delegated-cross-safe branch from a3be2c0 to f8b9000 Compare November 10, 2026 14:04
@einar-oplabs
einar-oplabs force-pushed the einar/23911-delegated-cross-safe branch from 3c2d3be to e2a4306 Compare November 11, 2026 17:17
@einar-oplabs
einar-oplabs force-pushed the einar/23931-el-sync-reset-loop branch from 1e1db47 to 4e2aeb9 Compare November 11, 2026 17:17
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 11, 2026 17:17
@einar-oplabs
einar-oplabs requested review from a team as code owners November 11, 2026 17:17
@joshklop

Copy link
Copy Markdown
Contributor

This isn't directly related to delegated derivation. Maybe we should take a step back and design the el sync handling a bit more.

@joshklop

Copy link
Copy Markdown
Contributor

🤖 Claude, for @joshklop: (Posted as a conversation comment because Josh has an unsubmitted review on this PR.) Review of aaee7e6ce1. The queue mechanics hold: a parked task is never lost or duplicated, consolidations keep their order behind it, and there's no busy loop. An INVALID answer mid-wait still resets, and the build path names the parent by hash. There are two problems. A Finalize that runs ahead of the waiting consolidation can stop the node (first comment, confirmed in the code). And an EL that rolled back with no p2p now stalls for 24 h where it used to reset in seconds. This only affects derivation mode, because delegated input never returns Wait.

1. core.rs:128

Bug: this lets a Finalize run ahead of the waiting consolidation, although Finalize normally runs after Consolidate. In derivation mode Finalize looks blocks up by number, and block_missing (finalize/task.rs:39) turns a block the EL lacks into BlockNotFound, which is Critical. The engine actor's drain returns it, and the actor exits.

Scenario: the EL datadir is wiped, the safe head is S, and consolidating S+1 waits. L1 finalizes some n ≤ S. The Finalize passes the safe_head >= n check, eth_getBlockByNumber(n) returns null, and the node stops. Before this PR the missing block reset at once and cleared the queue, so keeping the finalization queue alive is exactly what exposes this. No test runs a Finalize during a wait. Two options: hold Finalize behind the waiting task, or treat BlockNotFound as Skip while a task waits.

2. task.rs:502

Consider an EL that rolled back after an unclean shutdown, to a head L below the derived block's parent, and can't fetch the unsafe head (EL p2p off, which is common for sequencers and isolated replicas). It answers SYNCING, eth_syncing stays false at L, and it isn't idle at the parent. So the task waits the full --l2.el-sync-stall-timeout (24 h) with safe and unsafe frozen, where it used to reset in seconds. op-node waits only while IsEngineInitialELSyncing() (op-node/rollup/attributes/attributes.go:205-221) and resets otherwise. Suggest resetting quickly when the EL reports no sync and its latest block stays below the parent without moving.

3. el_sync_progress.rs:42

This can't tell an idle EL from a healthy one downloading headers. op-geth reports false in that phase (as this file's doc says), and op-reth's engine-tree download of a small gap doesn't change eth_syncing either. So after a gossip gap, the build-from-attributes path pulls the EL's sync target back to the parent every 30 s. It isn't a regression (the old reset did the same), and it can't insert on a wrong parent, because the build names the parent by hash.

4. core.rs:147

[nit] This replacement is dead code. Only Consolidate returns Wait, same-kind tasks are held behind the waiting one, and derivation has one attribute set in flight, so waiting is always None here. parking_a_task_requeues_the_one_waiting_before_it tests a state that can't happen. A single waiting: Option<(QueuedTask, Instant)> holding its deadline inside Engine could replace DrainScope, has_work_ahead_of_waiting_task and the actor's waiting_retry_at.

5. providers.rs:86

[nit] 0 is accepted and resets on the second check, 30 s in. Worth rejecting or documenting. Separately, any change in the reading restarts the stall timer, including a lower block or a switch between syncing and not syncing. So an EL whose eth_syncing keeps flipping never stalls. That's unlikely, but worth knowing.

@joshklop
joshklop removed this pull request from stack #24203 November 19, 2026 02:00
@joshklop
joshklop changed the base branch from einar/23911-delegated-cross-safe to develop November 19, 2026 02:00
@joshklop
joshklop force-pushed the einar/23931-el-sync-reset-loop branch from aaee7e6 to 5f85a12 Compare November 19, 2026 02:00
@mintlify

mintlify Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
Optimism-Docs 🟢 Ready View Preview Nov 19, 2026, 11:30 PM

@mintlify

mintlify Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
Optimism-Docs 🟡 Building – Nov 19, 2026, 2:00 AM

…lidation block

Kona-node resets whenever derived attributes name a block the EL lacks
below the unsafe head, so after an EL datadir wipe it re-derives from
genesis in a loop and finalization stays at genesis.

While the EL answers `SYNCING`, the task now waits (new `Wait`
severity): it is parked with its own 2s retry deadline, tasks of other
kinds (imports, builds, finalizations) keep running, and nothing resets,
so the finalization queue survives. Each retry looks the block up again;
every 30s it also reads the EL's sync progress and resends the unchanged
forkchoice, so an EL that reached the unsafe head is consolidated, and a
`VALID` answer records the EL as synced. An EL idle at the derived
block's parent, reporting no sync, gets the block built from the
attributes instead. Progress, or its absence, is logged every 5 minutes;
the engine resets once progress and latest block stop moving for 24 h
(`--l2.el-sync-stall-timeout`), as neither op-geth nor op-reth reports
progress while downloading headers. Failed lookups and probes are
Temporary. The delegated path is #23900.

Part of #23931.

BREAKING CHANGE: `EngineTaskErrorSeverity::Wait`;
`ConsolidateTaskError::{MissingUnsafeL2BlockDuringElSync,
MissingUnsafeL2BlockProbeFailed}`; `Engine::drain` takes a `DrainScope`;
`EngineConfig` gains `el_sync_stall_timeout` (default
`DEFAULT_EL_SYNC_STALL_TIMEOUT`); `ConsolidateTask::new` is non-const
and takes the stall timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
staging - docs/public-docs — 51be765c Deployed Nov 19, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kona-node: missing L2 block during EL catch-up resets the engine in a loop

2 participants