Skip to content

kona-engine,kona-node-service: retry delegated forkchoice updates and prevent finalized-head regression - #23900

Merged
einar-oplabs merged 11 commits into
developfrom
einar/23676-delegated-forkchoice
Nov 12, 2026
Merged

einar-oplabs merged 11 commits into
developfrom
einar/23676-delegated-forkchoice

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Why

Kona-node's delegated derivation could leave the EL's safe head behind its own and never retry a forkchoice update after SYNCING; a stale or forked delegate could move the finalized head backward or name a non-canonical finalized block, stopping or resetting kona every poll.

What

Breakdown

Added Removed Net
Implementation 380 132 248
Tests 1156 84 1072
Docs 0 0 0

Closes

  1. kona-node: reliably deliver and retry delegated forkchoice updates #23676
  2. kona-node: prevent delegated finalized-head regression #23677

AI Reviews

BREAKING CHANGE: EngineTaskErrorSeverity::Skip and FinalizeTaskError::{BehindFinalizedHead, BlockNumberMismatch, ConflictsWithFinalizedHead, NotCanonical} are new; exhaustive matches must handle them. A wrong-height block is no longer BlockNotFound. DelegateDerivationActor::new takes the engine's state receiver.

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Nov 3, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23676-delegated-forkchoice branch from 2417e68 to 537b357 Compare November 3, 2026 23:38
@einar-oplabs einar-oplabs changed the title kona-engine!: deliver and retry delegated forkchoice updates kona-engine: deliver and retry delegated forkchoice updates Nov 5, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23676-delegated-forkchoice branch 2 times, most recently from 12a2dec to b628609 Compare November 5, 2026 22:11
@einar-oplabs

einar-oplabs commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT-6.1-Sol (xhigh): Reviewed b628609e6f8c33e12466cb237d4c1193fc753380; no actionable findings.

Checked delegated safe-head delivery, the distinction between applied and syncing forkchoices, intermediate span-batch updates, and reset invalidation of the acknowledged forkchoice.

  • All 371 library tests across kona-engine and kona-node-service passed locally, including unchanged-target SYNCING → VALID recovery and the real engine actor test.
  • All 122 PR checks were green at review time.

The initial review covered this PR independently; combined validation is recorded below.

Combined validation update (2026-11-06): locally merged the current heads of #23899 (bb36509bc5), #23900 (b628609e6f), #23908 (0c2fa31049), and #23947 (64d55f160c) into test commit 215a22332b. All four PR branches are unchanged.

The merge preserves both sets of finalization guards and their regression tests. Test fixtures account for the acknowledged forkchoice status and the new identity-mismatch error.

  • 401 Rust library tests passed across kona-engine and kona-node-service.
  • op-devstack/sysgo tests and formatting checks passed.
  • Clippy passed with --all-targets --all-features -- -D warnings; it emitted the existing workspace warning about the removed clippy::from_iter_instead_of_collect lint.
  • All three follow_l2 acceptance scenarios passed under both op-node + op-reth and kona-node + op-reth, using freshly built Rust binaries from the combined checkout: safe/finalized/current-L1 convergence, reorg recovery, and operation without CL P2P.

Nextest flagged an open output handle in one passing, unchanged attribute-format test; its isolated rerun passed without the flag. The first acceptance attempts stopped at setup because the new checkout lacked the Cannon helper fixture; the complete runs above passed after restoring that fixture.

@einar-oplabs
einar-oplabs force-pushed the einar/23676-delegated-forkchoice branch from b628609 to 7a290fc Compare November 6, 2026 23:27
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 7, 2026 00:10
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 7, 2026 00:10

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GPT-6.1-Sol (xhigh): Reviewed 7a290fc5579c87f7f9f13fe2a45adac92baf34cc against fa3efe5b49ecc42007dfd4fcb79bd5d70f78f98f. No actionable findings.

Checked forced delivery for repeated delegated targets after SYNCING, unchanged forkchoice delivery during reset, the missing-block path preserving the unsafe sync target, and the distinction between delegated consolidation and intermediate span-batch attributes. Reviewed the RPC expectations and actor-level coverage alongside the production paths.

Validation: all 435 tests in kona-engine and kona-node-service passed at stack tip 454481f562f45c8c93022a6ed6090a226d12ce3c, including the downstream finalization changes in #23899 and #23908. Full acceptance devnets were not rerun during this review.

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fable 5.1 (xhigh): Reviewed 7a290fc557. One stack-ordering issue: the new follow path plus the same poll's finalize can exit the node if this PR lands before #23908. The rest are non-blocking nits on logging, duplication, ordering and test fixtures.

A delegated safe head below the unsafe head was treated like an
intermediate span-batch block: it moved the consensus labels without a
forkchoice update. An update that left the labels unchanged was also
skipped, but a SYNCING answer moves the labels, so an identical retry
never reached the execution layer. And while the execution layer was
still syncing and lacked the delegated safe block, consolidation reset
the engine, which restarted the sync (the delegated case of #23931).

Only derived attributes inside a span batch now defer the update. The
delegated consolidation paths and the reset always send their
forkchoice, so each delegate poll is the retry after a SYNCING answer
or an execution-layer restart. A delegated safe head whose block the
syncing execution layer does not have yet moves only the safe labels,
as in op-node's follow-source mode, instead of resetting; the missing
block is logged at warn. One helper applies a delegated safe head with
or without the unsafe head.

The finalize that follows such a poll still fails on the same syncing
execution layer until #23908 skips it, so this lands together with
#23899 and #23908.

`test_utils::l1_info_deposit` replaces three inline copies of the
L1-info deposit, and the delegated consolidation table asserts what
reaches the block sink.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23676-delegated-forkchoice branch from bf83ae6 to c1c8522 Compare November 7, 2026 04:37
A lagging delegate could move finality backward, and a delegate on
another fork could name a finalized block that is not the execution
layer's canonical block. The first is irreversible. The second either
stopped the node or failed every forkchoice update and reset the
engine on each poll.

The finalize task now looks the target up by number and finalizes a
pinned hash only when it is the canonical block at that height. A
target behind the finalized head, a different block at the finalized
height, or a pinned hash that is not canonical fails with an error of
the new `Skip` severity: the engine drops the task, counts it as a
failure, and the engine actor warns once per distinct error until a
reset. A target equal to the finalized head needs no lookup or
forkchoice update, and each finality guard runs once per kind of
target. A block served at another height is a distinct
`BlockNumberMismatch`.

The delegate actor also rejects a whole sync status whose finalized
L2 block is behind, or conflicts with, the engine's, so a lagging or
diverged delegate's safe head does not reach the engine either. The
baseline is the finalized head the engine applied, read from the state
it publishes, not the last target forwarded to it: a target the engine
skipped must not make the canonical block at its height look like a
conflict, which would stop the delegate's safe head too.

BREAKING CHANGE: `EngineTaskErrorSeverity` has a new `Skip` variant,
and `FinalizeTaskError` has new variants `BehindFinalizedHead`,
`BlockNumberMismatch`, `ConflictsWithFinalizedHead` and
`NotCanonical`; exhaustive matches must handle them. A block served at
another height is now `BlockNumberMismatch` instead of `BlockNotFound`.
`DerivationActorRequest::ProcessEngineSyncCompletionRequest` carries an
`EngineSyncState`, and `EngineDerivationClient::notify_sync_completed`
takes one. `DelegateDerivationActor::new` takes the engine's state
receiver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
joshklop
joshklop previously approved these changes Nov 11, 2026
@joshklop

Copy link
Copy Markdown
Contributor

The safe head can still move backward (pre-existing, outside this PR). op-node promotes the cross-safe head only when it advances (L2BlockRefAdvances). Kona applies whatever the delegate reports, so a delegate that restarts and briefly reports a lower safe head moves kona's safe head back. #23899 covers this for the finalized head only. This could be a follow-up.

`synchronize` skips a forkchoice update that leaves the labels
unchanged; `synchronize_unconditionally` always sends it. Both share a
private `send_forkchoice`. The reset and the delegated paths of the
consolidate task call `synchronize_unconditionally`; the insert,
finalize and derived consolidate paths call `synchronize`.

The consolidate task's thin wrappers `reconcile_to_safe_head`,
`follow_safe_head_during_el_sync` and `apply_delegated_safe_head` are
inlined at their call sites, keeping their logs and comments.
Behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
einar-oplabs and others added 5 commits November 12, 2026 14:30
`ensure_not_behind_finalized_head` and
`ensure_no_conflict_with_finalized_head` each had one caller, so their
checks move inline. A new test pins a target above the safe head: it
fails with `BlockNotSafe` before any lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ProcessEngineSyncCompletionRequest` and `notify_sync_completed` carry
the local-safe head again instead of the whole `EngineSyncState`. Both
consumers only read the local-safe head; the delegate actor takes its
finalized baseline from the engine's published state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its safe

Such a snapshot is inconsistent: the engine would refuse to finalize
it. op-node's follow mode rejects it the same way. A test also pins
that any finalized block, genesis included, is accepted before the
engine establishes finality, and the doc on the finality check now
states that it only catches a fork at the engine's finalized height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test sends stale finalizations through the actor: it keeps
running, an identical repeat keeps the same skip in memory, and a
reset forgets it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Opus 5.5 (high): On the safe head moving backward: agreed, and out of scope for this PR. #24004 is the follow-up. There, a delegated safe head below the engine's safe head is ignored unless the EL proves divergence, which matches op-node promoting only when L2BlockRefAdvances.

…d-finality

kona-engine: prevent delegated finalized-head regression
@einar-oplabs
einar-oplabs added this pull request to the merge queue Nov 12, 2026
@einar-oplabs einar-oplabs changed the title kona-engine: deliver and retry delegated forkchoice updates kona-engine,kona-node-service: retry delegated forkchoice updates and prevent finalized-head regression Nov 12, 2026
Merged via the queue into develop with commit b636ba6 Nov 12, 2026
131 of 132 checks passed
@einar-oplabs
einar-oplabs deleted the einar/23676-delegated-forkchoice branch November 12, 2026 23:35
einar-oplabs added a commit that referenced this pull request Nov 13, 2026
joshklop pushed a commit that referenced this pull request Nov 19, 2026
…lidation block

Kona-node resets whenever derived attributes name a block the EL lacks
below the unsafe head, so after an EL datadir wipe it re-derives from
genesis in a loop and finalization stays at genesis.

While the EL answers `SYNCING`, the task now waits (new `Wait`
severity): it is parked with its own 2s retry deadline, tasks of other
kinds (imports, builds, finalizations) keep running, and nothing resets,
so the finalization queue survives. Each retry looks the block up again;
every 30s it also reads the EL's sync progress and resends the unchanged
forkchoice, so an EL that reached the unsafe head is consolidated, and a
`VALID` answer records the EL as synced. An EL idle at the derived
block's parent, reporting no sync, gets the block built from the
attributes instead. Progress, or its absence, is logged every 5 minutes;
the engine resets once progress and latest block stop moving for 24 h
(`--l2.el-sync-stall-timeout`), as neither op-geth nor op-reth reports
progress while downloading headers. Failed lookups and probes are
Temporary. The delegated path is #23900.

Part of #23931.

BREAKING CHANGE: `EngineTaskErrorSeverity::Wait`;
`ConsolidateTaskError::{MissingUnsafeL2BlockDuringElSync,
MissingUnsafeL2BlockProbeFailed}`; `Engine::drain` takes a `DrainScope`;
`EngineConfig` gains `el_sync_stall_timeout` (default
`DEFAULT_EL_SYNC_STALL_TIMEOUT`); `ConsolidateTask::new` is non-const
and takes the stall timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshklop pushed a commit that referenced this pull request Nov 19, 2026
…lidation block

Kona-node resets whenever derived attributes name a block the EL lacks
below the unsafe head, so after an EL datadir wipe it re-derives from
genesis in a loop and finalization stays at genesis.

While the EL answers `SYNCING`, the task now waits (new `Wait`
severity): it is parked with its own 2s retry deadline, tasks of other
kinds (imports, builds, finalizations) keep running, and nothing resets,
so the finalization queue survives. Each retry looks the block up again;
every 30s it also reads the EL's sync progress and resends the unchanged
forkchoice, so an EL that reached the unsafe head is consolidated, and a
`VALID` answer records the EL as synced. An EL idle at the derived
block's parent, reporting no sync, gets the block built from the
attributes instead. Progress, or its absence, is logged every 5 minutes;
the engine resets once progress and latest block stop moving for 24 h
(`--l2.el-sync-stall-timeout`), as neither op-geth nor op-reth reports
progress while downloading headers. Failed lookups and probes are
Temporary. The delegated path is #23900.

Part of #23931.

BREAKING CHANGE: `EngineTaskErrorSeverity::Wait`;
`ConsolidateTaskError::{MissingUnsafeL2BlockDuringElSync,
MissingUnsafeL2BlockProbeFailed}`; `Engine::drain` takes a `DrainScope`;
`EngineConfig` gains `el_sync_stall_timeout` (default
`DEFAULT_EL_SYNC_STALL_TIMEOUT`); `ConsolidateTask::new` is non-const
and takes the stall timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants