Skip to content

kona-engine: keep delegated finalization pending during EL catch-up - #23908

Merged
joshklop merged 24 commits into
developfrom
einar/23675-delegated-catchup
Nov 17, 2026
Merged

joshklop merged 24 commits into
developfrom
einar/23675-delegated-catchup

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Why

Kona exits when it cannot apply a finalization target yet: during delegated EL catch-up (#23675), or after a reset lowers the safe head below a pending target (#23757, #23440).

What

  • A target above the safe head is skipped instead of stopping the node, as op-node's promoteFinalized does: BlockNotSafe { safe, target } now has Skip severity.
  • A pinned target the EL answers null for is skipped as the new NotYetAvailable (the 4s delegate poll resubmits it); a lookup that fails with an error is Temporary.
  • While derivation is idle, a by-number target skipped after a reset is retried on each L1 head instead of waiting for L1 finality to move; a finalized L1 block moving backwards is logged and ignored.

Breakdown

Added Removed Net
Implementation 119 44 75
Tests 618 57 561
Docs 0 0 0

Closes

Part of ethereum-optimism/protocol-team#545; follows up on #23678. Follow-ups: #24119, #24120. Stack, in merge order: #23908 → #23947 → #24004 → #24005 → #24006 → #24007 → #24065 (#23900 and #23899 merged).

  1. Closes: kona-node: keep delegated finalization pending during EL catch-up #23675
  2. Closes: kona-node: fatal Finalize(BlockNotSafe) during delegated startup/reset recovery #23757

AI Reviews

  • 2026-11-07T00:26:10Z 15ca278 GPT-6.1-Sol xhigh
  • 2026-11-07T00:42:24Z 15ca278 Fable 5.1 xhigh

BREAKING CHANGE: FinalizeTaskError::NotYetAvailable { target } is new; exhaustive matches must handle it. BlockNotSafe is now BlockNotSafe { safe, target } and no longer stops the node.

🤖 Generated with Claude Code

@einar-oplabs

einar-oplabs commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT-6.1-Sol (xhigh): Reviewed 0c2fa31049b9dacee2ebd49deff50b63ca3b0fcd; no actionable findings.

  • All 375 library tests across kona-engine and kona-node-service passed locally.
  • All 122 PR checks were green at review time.
  • Remaining coverage gap: no single targeted regression test drives delegate polling through engine catch-up and deferred finalization; the unit tests cover the pieces separately.

Combined validation update (2026-11-06): locally merged the current heads of #23899 (bb36509bc5), #23900 (b628609e6f), #23908 (0c2fa31049), and #23947 (64d55f160c) into test commit 215a22332b. All four PR branches are unchanged.

The merge preserves both sets of finalization guards and their regression tests. Test fixtures account for the acknowledged forkchoice status and the new identity-mismatch error.

  • 401 Rust library tests passed across kona-engine and kona-node-service.
  • op-devstack/sysgo tests and formatting checks passed.
  • Clippy passed with --all-targets --all-features -- -D warnings; it emitted the existing workspace warning about the removed clippy::from_iter_instead_of_collect lint.
  • All three follow_l2 acceptance scenarios passed under both op-node + op-reth and kona-node + op-reth, using freshly built Rust binaries from the combined checkout: safe/finalized/current-L1 convergence, reorg recovery, and operation without CL P2P.

Nextest flagged an open output handle in one passing, unchanged attribute-format test; its isolated rerun passed without the flag. The first acceptance attempts stopped at setup because the new checkout lacked the Cannon helper fixture; the complete runs above passed after restoring that fixture.

@einar-oplabs
einar-oplabs force-pushed the einar/23675-delegated-catchup branch from 0c2fa31 to 15ca278 Compare November 6, 2026 23:27
@einar-oplabs
einar-oplabs changed the base branch from develop to einar/23677-delegated-finality November 6, 2026 23:29
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 7, 2026 00:10
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 7, 2026 00:10

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GPT-6.1-Sol (xhigh): Reviewed 15ca278f190ed7635c51dcc6d18307e311562c1b against 0e0cdf7c305abd6f5c2dc0b5e38f30351d979906. No actionable findings.

Checked that above-safe and unavailable pinned targets leave the queue without blocking catch-up imports, that RPC transport failures retain their retry severity, and that repeated delegate polls resubmit targets. Traced local finalization through confirmed attributes, reset clearing, retained L1 finality, and retry on later L1 head updates; reviewed the recovery tables and real-actor tests.

Validation: all 435 tests in kona-engine and kona-node-service passed at stack tip 454481f562f45c8c93022a6ed6090a226d12ce3c. Full acceptance devnets were not rerun during this review.

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fable 5.1 (xhigh): Reviewed 15ca278f19. Main point: any JSON-RPC error response for a pinned target is now a skip, so an EL that persistently errors stalls delegated finalization with one warning and no error detail. Then a question on the drained by-number entry, test-coverage gaps, and doc and naming nits.

A delegated safe head below the unsafe head was treated like an
intermediate span-batch block: it moved the consensus labels without a
forkchoice update. An update that left the labels unchanged was also
skipped, but a SYNCING answer moves the labels, so an identical retry
never reached the execution layer. And while the execution layer was
still syncing and lacked the delegated safe block, consolidation reset
the engine, which restarted the sync (the delegated case of #23931).

Only derived attributes inside a span batch now defer the update. The
delegated consolidation paths and the reset always send their
forkchoice, so each delegate poll is the retry after a SYNCING answer
or an execution-layer restart. A delegated safe head whose block the
syncing execution layer does not have yet moves only the safe labels,
as in op-node's follow-source mode, instead of resetting; the missing
block is logged at warn. One helper applies a delegated safe head with
or without the unsafe head.

The finalize that follows such a poll still fails on the same syncing
execution layer until #23908 skips it, so this lands together with
#23899 and #23908.

`test_utils::l1_info_deposit` replaces three inline copies of the
L1-info deposit, and the delegated consolidation table asserts what
reaches the block sink.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@einar-oplabs
einar-oplabs force-pushed the einar/23677-delegated-finality branch from 1cdca29 to cb2fc8d Compare November 7, 2026 04:37
@einar-oplabs
einar-oplabs force-pushed the einar/23675-delegated-catchup branch from ce455d4 to 9db5f8b Compare November 7, 2026 04:37
einar-oplabs and others added 2 commits November 9, 2026 18:10
A lagging delegate could move finality backward, and a delegate on
another fork could name a finalized block that is not the execution
layer's canonical block. The first is irreversible. The second either
stopped the node or failed every forkchoice update and reset the
engine on each poll.

The finalize task now looks the target up by number and finalizes a
pinned hash only when it is the canonical block at that height. A
target behind the finalized head, a different block at the finalized
height, or a pinned hash that is not canonical fails with an error of
the new `Skip` severity: the engine drops the task, counts it as a
failure, and the engine actor warns once per distinct error until a
reset. A target equal to the finalized head needs no lookup or
forkchoice update, and each finality guard runs once per kind of
target. A block served at another height is a distinct
`BlockNumberMismatch`.

The delegate actor also rejects a whole sync status whose finalized
L2 block is behind, or conflicts with, the engine's, so a lagging or
diverged delegate's safe head does not reach the engine either. The
baseline is the finalized head the engine applied, read from the state
it publishes, not the last target forwarded to it: a target the engine
skipped must not make the canonical block at its height look like a
conflict, which would stop the delegate's safe head too.

BREAKING CHANGE: `EngineTaskErrorSeverity` has a new `Skip` variant,
and `FinalizeTaskError` has new variants `BehindFinalizedHead`,
`BlockNumberMismatch`, `ConflictsWithFinalizedHead` and
`NotCanonical`; exhaustive matches must handle them. A block served at
another height is now `BlockNumberMismatch` instead of `BlockNotFound`.
`DerivationActorRequest::ProcessEngineSyncCompletionRequest` carries an
`EngineSyncState`, and `EngineDerivationClient::notify_sync_completed`
takes one. `DelegateDerivationActor::new` takes the engine's state
receiver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A finalization target the engine cannot apply yet no longer stops
kona-node or holds the engine queue. A target above the safe head
fails with `BlockNotSafe`, now of `Skip` severity, whatever its source,
as op-node's promoteFinalized does: a delegate can run ahead of the
local safe head, and a reset can lower the safe head below a target
that was safe when requested (#23757, #23440). A hash-pinned target
the execution layer has no block for yet fails with the new skipped
`NotYetAvailable`, which carries the answer, whether the lookup
returned null or an error response, so a retried lookup cannot hold
the queue ahead of the imports that let the execution layer catch up.
The delegate poll resubmits its target every 4s.

A by-number target from L1 finality that a reset skipped is no longer
lost until L1 finality moves again: like op-node's retry when
derivation goes idle, the finalizer keeps the highest finalized L1
number and finalizes newly safe blocks against it on each L1 head
update, which batches the blocks confirmed since the last update. A
by-number target whose block is missing still fails with
`BlockNotFound`.

Tests cover the finalizer, a block derived again after a reset, a
delegated poll during execution-layer sync at the actor level, and
the error variant of each unavailable-block answer;
`test_utils::expect_sync_start` replaces two copies of the reset's RPC
script.

BREAKING CHANGE: `FinalizeTaskError` has a new variant,
`NotYetAvailable { engine_answer, target }`; exhaustive matches must
handle it. `BlockNotSafe` keeps its name but has `Skip` severity
instead of `Critical`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshklop
joshklop previously approved these changes Nov 16, 2026
@joshklop
joshklop disabled auto-merge November 16, 2026 20:35
@joshklop
joshklop added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@joshklop
joshklop added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@joshklop
joshklop added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 17, 2026
@joshklop
joshklop added this pull request to the merge queue Nov 17, 2026
@joshklop
joshklop removed this pull request from the merge queue due to a manual request Nov 17, 2026
@joshklop
joshklop marked this pull request as draft November 17, 2026 15:59
joshklop and others added 2 commits November 17, 2026 08:00
develop now inserts pre-Canyon payloads with engine_newPayloadV2 (#24092), so
finalization_target_survives_resets_below_it must expect that method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop marked this pull request as ready for review November 17, 2026 16:06
joshklop
joshklop previously approved these changes Nov 17, 2026
@joshklop
joshklop enabled auto-merge November 17, 2026 16:08
geoknee
geoknee previously approved these changes Nov 17, 2026
@joshklop
joshklop added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 17, 2026
@joshklop
joshklop enabled auto-merge November 17, 2026 16:36
Resolve conflicts with #23846, which replaced DerivationDelegateProvider with
SyncStatusSource: keep the module-level poll interval beside the new trait,
and give the test delegate a &'static str error so the fetch-failure case
still fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop dismissed stale reviews from geoknee and themself via 973f8af November 17, 2026 16:43
@joshklop
joshklop added this pull request to the merge queue Nov 17, 2026
Merged via the queue into develop with commit d6b23e5 Nov 17, 2026
127 checks passed
@joshklop
joshklop deleted the einar/23675-delegated-catchup branch November 17, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kona-node: fatal Finalize(BlockNotSafe) during delegated startup/reset recovery kona-node: keep delegated finalization pending during EL catch-up

3 participants