Skip to content

kona-engine,kona-node-service: recover when the EL rejects a delegated forkchoice update - #24065

Open
einar-oplabs wants to merge 1 commit into
einar/23912-delegated-parity-scenariosfrom
einar/24061-delegated-el-rejection-recovery
Open

einar-oplabs wants to merge 1 commit into
einar/23912-delegated-parity-scenariosfrom
einar/24061-delegated-el-rejection-recovery

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

A delegate whose chain does not descend from kona-node's finalized block can leave the EL on its fork after a rejected forkchoice update, because op-geth and op-reth make the head canonical before rejecting it.

What

  • A rejected delegated update that moved the head restores the engine's forkchoice while the EL holds the finalized block by hash; a restore answered SYNCING or rejected resets instead.
  • After a rejection on a chain forked below finality, the delegate actor holds the delegate's heads back until the engine's finality moves, every five minutes, or a reset.
  • The sync start refuses a chain off the finalized block (op-node ReorgFinalizedErr) and a reset restores and walks again, so a delegate's head cannot stop the node.

Breakdown

Added Removed Net
Implementation 724 64 660
Tests 1266 59 1207
Docs 10 0 10

Closes

Part of ethereum-optimism/protocol-team#545. Stack, in merge order: #23900 → #23899 → #23908 → #23947 → #24004 → #24005 → #24007 → #24065.

  1. Closes: kona-node: recover when the EL rejects a delegated forkchoice update #24061

AI Reviews

  • 2026-11-11T15:16:29Z fef2fef Fable 5.1 xhigh
  • 2026-11-11T16:42:18Z 858ed5f Fable 5.1 xhigh

BREAKING CHANGE: ConsolidateTaskError gains DelegatedForkchoiceRejected and FailedToFetchFinalizedBlock; SynchronizeTaskError gains FinalizedAheadOfSafe; SyncStartError gains MismatchedFinalizedBlock { finalized, walked }, FinalizedHeightUnavailable, FinalizedL1OriginUnavailable, FinalizedL1OriginNotServedByHash and FinalizedL1OriginNotCanonical; EngineResetError gains FinalityNotRestored; DelegatedRejection and SafeLabel are exported.

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Nov 11, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/24061-delegated-el-rejection-recovery branch from 858ed5f to f78e2ef Compare November 11, 2026 17:17
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 11, 2026 17:17
@einar-oplabs
einar-oplabs requested review from a team as code owners November 11, 2026 17:17
@joshklop
joshklop added this pull request to stack #24203 November 16, 2026 20:35
@joshklop
joshklop force-pushed the einar/24061-delegated-el-rejection-recovery branch from f78e2ef to 0cf7192 Compare November 19, 2026 00:54
@mintlify

mintlify Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
Optimism-Docs 🟢 Ready View Preview Nov 19, 2026, 12:54 AM

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: Review of 0cf719221f. I found no regression that strands a normal or sequencer node at sync start. Transient RPC errors, a lagging L1 and an inconsistent EL at the finalized height are all retried. The restore path has no unbounded loop, and the 5-minute hold can't hold forever. The first comment is confirmed against op-node's code. The two comments on el_applied_last_forkchoice come from reading the whole stack at its tip, and the fix for both belongs here.

l2.assert_finished();
}

/// A walk that misses the finalized block stops the node, like op-node's `ReorgFinalizedErr`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: The parity claim is wrong: op-node doesn't stop here. onResetEngineRequest (engine_controller.go:1431-1438) turns any FindL2Heads error into a ResetEvent, and onResetEvent (sync_deriver.go:227-231) requests another reset, so op-node keeps retrying.

l2_block_info_at(el_finalized)
};
if !client.l2_block_is_stored(anchor.block_info.hash).await.map_err(SyncStartError::from)? {
return Err(mismatch.into());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: After a restart, the anchor is the EL's own finalized label, which L2ForkchoiceState::current fetched by tag moments earlier. A by-hash miss on it is an EL inconsistency of the same kind as FinalizedHeightUnavailable, which is retried. But SyncStart(_) is not temporary (line 376), so this case stops the node. Suggest treating the restart case as temporary, or at least correcting the test comment at line 558.

let rejection = if is_chain_below_finality {
DelegatedRejection::ChainBelowFinality
} else {
DelegatedRejection::HeadLabels

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: HeadLabels replaces the reset that used to recover from stale engine labels (see the removed rejected_forkchoice_for_delegated_heads_resets_the_engine). The new doc on safe_label_on_engine_chain says a stale safe label makes the EL reject the next update. That rejection moves the head, so it's classified here and nothing resets. With a stale safe label and a delegate whose cross-safe stays below it, every 4 s poll sends new heads, the EL rejects them, and the engine restores. The EL head jumps forward and back, and safe stays frozen until cross-safe passes the label. Consider resetting after N consecutive HeadLabels rejections. Medium confidence: I couldn't build a concrete production sequence that produces the stale label.

warn!(target: "engine", ?finalized, ?heads, "Execution layer no longer holds the finalized block; resetting");
return rejection;
}
Err(err) => return err,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: An RPC failure here, or in the restore at line 487, returns a temporary error after the rejected update has already made the delegate's head canonical. el_applied_last_forkchoice stays true, because neither a failed synchronize nor the restore records anything. On retry, el_canonical_hash_at reads the delegate's chain as the engine's and matches the reported block. With cross-safe below the safe label, it returns without sending an update, and the delegate actor then memoizes those heads and stops resending them. The EL serves a chain that skips the finalized block until the next finalize forces a reset. Suggest clearing el_applied_last_forkchoice as soon as an update that moved the head comes back InvalidForkchoiceState, then setting it from the restore's answer. (Traced, not tested.)

return rejection;
}
if !moves_head {
warn!(target: "engine", ?heads, "Execution layer rejected the delegated safe label; skipping it for this poll");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: This repeats forever while el_applied_last_forkchoice is false. Example: the unsafe head is on chain A and the delegate has reorged to chain B below it. Each poll takes follow_safe_head_during_el_sync (#24004's gate), sends {head U_A, safe S_B}, the EL rejects S_B, and this returns "skip for this poll". Nothing has changed and the flag is still false, so the next poll does the same. It only ends when the delegate passes U_A's height or a finalize update succeeds. Suggest resending the engine's own forkchoice here and recording the answer, which sets the flag.

let is_proven_on_engine_chain = chain_hash.is_some();
let replaces_stale_safe_label = is_proven_on_engine_chain &&
cross_safe.block_info.number == safe.block_info.number &&
cross_safe.block_info.number <= safe.block_info.number &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: [nit] <= behaves the same as == here, because the < case already returned above, but the diff reads like a behaviour change. Revert it, or drop the earlier guard.

let is_finalized = safe_cursor.block_info.hash == current_fc.finalized.block_info.hash;
if !is_finalized && safe_cursor.block_info.number <= current_fc.finalized.block_info.number
{
return Err(finalized_block_not_on_walked_chain(&current_fc.finalized, &safe_cursor));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: When the EL's head ends up below its finalized label on the same chain (crash or rewind), this reports MismatchedFinalizedBlock, which names a fork although the walked block is an ancestor. The restore then tries the finalized block alone, and retries forever if the EL can't apply it. Not a regression (develop exited with FinalizedAheadOfUnsafe), but op-node handles this case explicitly (start.go, "Unsafe head is behind known finalized").

…d forkchoice update

A delegate whose chain does not descend from kona-node's finalized block
can leave the EL on its fork after a rejected forkchoice update, because
op-geth and op-reth make the head canonical before rejecting it; the
next reset then walks a chain that misses finality.

On every delegated path, a rejected update that moved the head restores
the engine's own forkchoice while the EL holds the finalized block by
hash, and is permanent; a restore answered `SYNCING` or rejected resets
instead. After a rejection on a chain that forks below finality, the
delegate actor holds the delegate's heads back as a whole until the
engine's finality moves, otherwise every five minutes, or until a reset.
A rejection that moved only the safe label restores nothing and is
retried next poll. The sync start refuses a chain off the finalized
block (op-node `ReorgFinalizedErr`), also when the sequencing window
stops the walk above it; a reset restores the engine's finalized block,
or after a restart the EL's own, so a delegate's head cannot stop the
node. A missing block or L1 origin at the finalized height is retried
while the provider lags; an L1 origin on another chain stops the node.
`synchronize` keeps the safe labels at or above finality.

Closes #24061.

BREAKING CHANGE: `ConsolidateTaskError` gains
`DelegatedForkchoiceRejected { heads, rejection }` and
`FailedToFetchFinalizedBlock`; `SynchronizeTaskError` gains
`FinalizedAheadOfSafe`; `SyncStartError::MismatchedFinalizedBlock`
becomes `{ finalized, walked }` and `SyncStartError` gains
`FinalizedHeightUnavailable`, `FinalizedL1OriginUnavailable`,
`FinalizedL1OriginNotServedByHash` and `FinalizedL1OriginNotCanonical`;
`EngineResetError` gains `FinalityNotRestored`; `EngineDerivationClient`
gains `send_rejected_delegated_safe_heads`; `DerivationActorRequest`
gains `ProcessRejectedDelegatedSafeHeads`; `DelegatedRejection` and
`SafeLabel` are exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop removed this pull request from stack #24203 November 19, 2026 02:00
@joshklop
joshklop force-pushed the einar/24061-delegated-el-rejection-recovery branch from 0cf7192 to 8efdff1 Compare November 19, 2026 02:00
@joshklop
joshklop added this pull request to stack #24271 November 19, 2026 02:00
joshklop pushed a commit that referenced this pull request Nov 19, 2026
…safe heads separately

Under interop op-node imports a delegate's local-safe and cross-safe
heads separately; kona-node copies the delegated safe head into both, so
its forkchoice safe hash can run ahead of cross-chain verification.

Kona-node now follows local-safe for the EL lookup, EL sync and reorgs,
and moves the safe label to cross-safe only when it advances. Only the
unsafe head and established finality count as proof of the chain; other
heights are checked against an EL that applied the last forkchoice, and
only at or below the unsafe head. A head that does not extend the unsafe
chain reorgs directly, moving safe to cross-safe, clamped to finalized,
unless the EL holds the current safe label, which a catch-up then keeps;
only a conflict with finalized skips the update. While the EL has not
applied the last forkchoice, safe only moves forward and nothing reorgs.
Inconsistent heads are rejected, and the local-safe L1 origin must be
canonical. Before interop both heads are one block, so non-interop
chains see no change. A rejected update resets the engine; recovery is
#24065.

Closes #23911.

BREAKING CHANGE: `ConsolidateInput::BlockInfo` is replaced by
`Delegated(Box<DelegatedSafeHeads>)`, built with
`DelegatedSafeHeads::try_new`, which returns
`InconsistentSafeHeadsError`; the `From<L2BlockInfo>` conversions are
removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: Second pass at 8efdff13de. The seven comments from the first round are still open and aren't repeated here. The four inline comments are new. Below: complexity, simplifications, and a proposed split.

Complexity. 29 files; +724/−64 implementation, +1266/−59 tests, 12 new public error variants and types. The PR adds four recovery mechanisms that interact:

  1. A rejected delegated update is classified and the engine's forkchoice restored (ConsolidateTask).
  2. The delegated actor holds back rejected heads, released by a 5-minute timer or by finality moving.
  3. The sync start refuses a walk that misses finality; the reset restores and walks again.
  4. A finality backstop in synchronize.

Most of the edge cases are in (1) and (2). #24061 records three Fable review rounds that each found new ones, and the first round here found five more (the HeadLabels loop, a stale el_applied_last_forkchoice twice, the SafeLabel repeat, the restart anchor).

Big simplifications

  • Prevent instead of repair. The delegate's rollup RPC already serves optimism_outputAtBlock. Before sending heads that move the EL's head, the delegated actor could ask the delegate for its block at the engine's finalized height and compare hashes, caching the answer per finalized head (one call per finality change). A delegate whose chain skips kona's finalized block would then never move the EL. Most of (1) and (2) goes away: the ChainBelowFinality classification, is_off_finalized_chain, the consolidate-time restore, the hold timer and the cross-actor message. A rejected update that moves the head could reset as it does today, with (3) as the backstop for a delegate whose answers are inconsistent.
  • Scope to op-node parity first. op-node's FollowSource has no restore. It assumes the source's refs are sane (engine_controller.go:1523) and retries any FindL2Heads failure. (3) without the restore, plus (4), would be the parity piece. "A delegate's head cannot loop the engine reset" goes beyond op-node and could follow once the stack lands.

Small simplifications (in addition to the inline comments and the earlier <= nit)

  • find_starting_forkchoice_restoring_finality chooses between the engine's finalized head and the EL's finalized label as the anchor. After a rejected update, the EL's finalized label is the block the engine last sent it, so el_finalized in both cases looks equivalent and drops a branch and a test case. This would also settle the first round's restart-anchor comment. (Medium confidence.)
  • SafeLabel is exported only to name a field of FinalizedAheadOfSafe. Keeping it crate-private shrinks the breaking change.

Split. Proposed pieces, in merge order, each reviewable on its own:

  1. Refactor only: send_forkchoice / record_forkchoice_reply / with_update, with no behaviour change.
  2. The sync start refuses a walk that misses the finalized block (sync/mod.rs, sync/error.rs, is_temporary, finality_tests). The lower stack doesn't touch these files, so this piece can target develop directly.
  3. The reset restores a forkchoice on the finalized chain and walks again (FinalityNotRestored, recovery_tests).
  4. The FinalizedAheadOfSafe backstop in synchronize, plus the sequencer arm.
  5. Delegated mode only: classify a rejected update, then the hold (consolidate/*, the delegated actor, the request plumbing, docs). With the first big simplification, this piece mostly disappears.

Pieces 2–4 change every kona-node reset or forkchoice update, sequencers and deriving verifiers included. As separate PRs, reviewers can judge that change without the delegated-mode context.

@@ -58,12 +68,12 @@ pub(in crate::task_queue) async fn synchronize_unconditionally(
state_update: EngineSyncStateUpdate,
) -> Result<(), SynchronizeTaskError> {
let new_sync_state = state.sync_state.apply_update(state_update);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: apply_update sets the kona_node_block_labels gauges before the new finality guard and before the EL answers. Until now a reset followed a rejected update and rewrote them. This PR turns a rejected delegated update into a skip, so the safe and local-safe gauges keep showing the rejected delegate heads until the next accepted update. During a chain-below-finality hold, that is five minutes or more. A FinalizedAheadOfSafe refusal does the same. Suggest building new_sync_state with the new with_update here and at line 37, and setting the gauges in send_and_apply_forkchoice next to state.sync_state = new_sync_state.

/// provider without a block at that height lags and is retried; one with a different block there
/// follows another L1 chain, which retrying cannot fix; one with that very block there serves it
/// only by number, as a broken or lagging block-hash index does.
async fn finalized_l1_origin_missing_by_hash(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: This extra L1 lookup only decides whether to retry or stop. FinalizedL1OriginNotCanonical is not temporary (core.rs:284), so a single L1 backend behind a load balancer that is on a stale fork at this height stops the node. op-node retries every FindL2Heads error through ResetEvent (engine_controller.go:1433-1436). Suggest dropping the lookup and folding NotCanonical and NotServedByHash into one temporary FinalizedL1OriginUnavailable. That saves an L1 round trip and two variants from the breaking change, and matches op-node.


/// Tells derivation about delegated safe heads whose forkchoice update the execution layer
/// permanently rejected, so the delegate does not send them again before a reset.
async fn forward_permanently_rejected_delegated_safe_heads(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: [simplification] The rejection reaches the delegated actor through a new request variant, a new EngineDerivationClient method, and an arm in the deriving actor that ignores it. The delegated actor already reads engine_state_rx. If EngineState kept the last permanent rejection (cleared by reset), the delegated actor could read it from the watch, or ConsolidateTask could skip matching heads before any RPC. That removes the channel round trip. It also closes a small race: heads polled before this message is processed still reach the engine, and cost another rejected update and restore.

engine_finalized: self.engine_finalized_head(),
retry_at: Instant::now() + CHAIN_BELOW_FINALITY_RETRY_INTERVAL,
},
DelegatedRejection::HeadLabels | DelegatedRejection::SafeLabel => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: [nit] SafeLabel never reaches this arm: forward_permanently_rejected_delegated_safe_heads returns early for it because is_permanent() is false. As written, the arm suggests a safe-label rejection is held until reset. Suggest unreachable!, or moving SafeLabel out of DelegatedRejection. It only ever skips one poll, so a separate ConsolidateTaskError variant fits it better.

This branch was successfully deployed

1 active deployment
staging - docs/public-docs — 8efdff13 Deployed Nov 19, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kona-node: recover when the EL rejects a delegated forkchoice update

2 participants