Repository navigation
kona-engine,kona-node-service: recover when the EL rejects a delegated forkchoice update - #24065
Conversation
858ed5f to
f78e2ef
Compare
f78e2ef to
0cf7192
Compare
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
joshklop
left a comment
There was a problem hiding this comment.
🤖 Claude, for @joshklop: Review of 0cf719221f. I found no regression that strands a normal or sequencer node at sync start. Transient RPC errors, a lagging L1 and an inconsistent EL at the finalized height are all retried. The restore path has no unbounded loop, and the 5-minute hold can't hold forever. The first comment is confirmed against op-node's code. The two comments on el_applied_last_forkchoice come from reading the whole stack at its tip, and the fix for both belongs here.
| l2.assert_finished(); | ||
| } | ||
|
|
||
| /// A walk that misses the finalized block stops the node, like op-node's `ReorgFinalizedErr`, |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: The parity claim is wrong: op-node doesn't stop here. onResetEngineRequest (engine_controller.go:1431-1438) turns any FindL2Heads error into a ResetEvent, and onResetEvent (sync_deriver.go:227-231) requests another reset, so op-node keeps retrying.
| l2_block_info_at(el_finalized) | ||
| }; | ||
| if !client.l2_block_is_stored(anchor.block_info.hash).await.map_err(SyncStartError::from)? { | ||
| return Err(mismatch.into()); |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: After a restart, the anchor is the EL's own finalized label, which L2ForkchoiceState::current fetched by tag moments earlier. A by-hash miss on it is an EL inconsistency of the same kind as FinalizedHeightUnavailable, which is retried. But SyncStart(_) is not temporary (line 376), so this case stops the node. Suggest treating the restart case as temporary, or at least correcting the test comment at line 558.
| let rejection = if is_chain_below_finality { | ||
| DelegatedRejection::ChainBelowFinality | ||
| } else { | ||
| DelegatedRejection::HeadLabels |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: HeadLabels replaces the reset that used to recover from stale engine labels (see the removed rejected_forkchoice_for_delegated_heads_resets_the_engine). The new doc on safe_label_on_engine_chain says a stale safe label makes the EL reject the next update. That rejection moves the head, so it's classified here and nothing resets. With a stale safe label and a delegate whose cross-safe stays below it, every 4 s poll sends new heads, the EL rejects them, and the engine restores. The EL head jumps forward and back, and safe stays frozen until cross-safe passes the label. Consider resetting after N consecutive HeadLabels rejections. Medium confidence: I couldn't build a concrete production sequence that produces the stale label.
| warn!(target: "engine", ?finalized, ?heads, "Execution layer no longer holds the finalized block; resetting"); | ||
| return rejection; | ||
| } | ||
| Err(err) => return err, |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: An RPC failure here, or in the restore at line 487, returns a temporary error after the rejected update has already made the delegate's head canonical. el_applied_last_forkchoice stays true, because neither a failed synchronize nor the restore records anything. On retry, el_canonical_hash_at reads the delegate's chain as the engine's and matches the reported block. With cross-safe below the safe label, it returns without sending an update, and the delegate actor then memoizes those heads and stops resending them. The EL serves a chain that skips the finalized block until the next finalize forces a reset. Suggest clearing el_applied_last_forkchoice as soon as an update that moved the head comes back InvalidForkchoiceState, then setting it from the restore's answer. (Traced, not tested.)
| return rejection; | ||
| } | ||
| if !moves_head { | ||
| warn!(target: "engine", ?heads, "Execution layer rejected the delegated safe label; skipping it for this poll"); |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: This repeats forever while el_applied_last_forkchoice is false. Example: the unsafe head is on chain A and the delegate has reorged to chain B below it. Each poll takes follow_safe_head_during_el_sync (#24004's gate), sends {head U_A, safe S_B}, the EL rejects S_B, and this returns "skip for this poll". Nothing has changed and the flag is still false, so the next poll does the same. It only ends when the delegate passes U_A's height or a finalize update succeeds. Suggest resending the engine's own forkchoice here and recording the answer, which sets the flag.
| let is_proven_on_engine_chain = chain_hash.is_some(); | ||
| let replaces_stale_safe_label = is_proven_on_engine_chain && | ||
| cross_safe.block_info.number == safe.block_info.number && | ||
| cross_safe.block_info.number <= safe.block_info.number && |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: [nit] <= behaves the same as == here, because the < case already returned above, but the diff reads like a behaviour change. Revert it, or drop the earlier guard.
| let is_finalized = safe_cursor.block_info.hash == current_fc.finalized.block_info.hash; | ||
| if !is_finalized && safe_cursor.block_info.number <= current_fc.finalized.block_info.number | ||
| { | ||
| return Err(finalized_block_not_on_walked_chain(¤t_fc.finalized, &safe_cursor)); |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: When the EL's head ends up below its finalized label on the same chain (crash or rewind), this reports MismatchedFinalizedBlock, which names a fork although the walked block is an ancestor. The restore then tries the finalized block alone, and retries forever if the EL can't apply it. Not a regression (develop exited with FinalizedAheadOfUnsafe), but op-node handles this case explicitly (start.go, "Unsafe head is behind known finalized").
…d forkchoice update A delegate whose chain does not descend from kona-node's finalized block can leave the EL on its fork after a rejected forkchoice update, because op-geth and op-reth make the head canonical before rejecting it; the next reset then walks a chain that misses finality. On every delegated path, a rejected update that moved the head restores the engine's own forkchoice while the EL holds the finalized block by hash, and is permanent; a restore answered `SYNCING` or rejected resets instead. After a rejection on a chain that forks below finality, the delegate actor holds the delegate's heads back as a whole until the engine's finality moves, otherwise every five minutes, or until a reset. A rejection that moved only the safe label restores nothing and is retried next poll. The sync start refuses a chain off the finalized block (op-node `ReorgFinalizedErr`), also when the sequencing window stops the walk above it; a reset restores the engine's finalized block, or after a restart the EL's own, so a delegate's head cannot stop the node. A missing block or L1 origin at the finalized height is retried while the provider lags; an L1 origin on another chain stops the node. `synchronize` keeps the safe labels at or above finality. Closes #24061. BREAKING CHANGE: `ConsolidateTaskError` gains `DelegatedForkchoiceRejected { heads, rejection }` and `FailedToFetchFinalizedBlock`; `SynchronizeTaskError` gains `FinalizedAheadOfSafe`; `SyncStartError::MismatchedFinalizedBlock` becomes `{ finalized, walked }` and `SyncStartError` gains `FinalizedHeightUnavailable`, `FinalizedL1OriginUnavailable`, `FinalizedL1OriginNotServedByHash` and `FinalizedL1OriginNotCanonical`; `EngineResetError` gains `FinalityNotRestored`; `EngineDerivationClient` gains `send_rejected_delegated_safe_heads`; `DerivationActorRequest` gains `ProcessRejectedDelegatedSafeHeads`; `DelegatedRejection` and `SafeLabel` are exported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0cf7192 to
8efdff1
Compare
…safe heads separately Under interop op-node imports a delegate's local-safe and cross-safe heads separately; kona-node copies the delegated safe head into both, so its forkchoice safe hash can run ahead of cross-chain verification. Kona-node now follows local-safe for the EL lookup, EL sync and reorgs, and moves the safe label to cross-safe only when it advances. Only the unsafe head and established finality count as proof of the chain; other heights are checked against an EL that applied the last forkchoice, and only at or below the unsafe head. A head that does not extend the unsafe chain reorgs directly, moving safe to cross-safe, clamped to finalized, unless the EL holds the current safe label, which a catch-up then keeps; only a conflict with finalized skips the update. While the EL has not applied the last forkchoice, safe only moves forward and nothing reorgs. Inconsistent heads are rejected, and the local-safe L1 origin must be canonical. Before interop both heads are one block, so non-interop chains see no change. A rejected update resets the engine; recovery is #24065. Closes #23911. BREAKING CHANGE: `ConsolidateInput::BlockInfo` is replaced by `Delegated(Box<DelegatedSafeHeads>)`, built with `DelegatedSafeHeads::try_new`, which returns `InconsistentSafeHeadsError`; the `From<L2BlockInfo>` conversions are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshklop
left a comment
There was a problem hiding this comment.
🤖 Claude, for @joshklop: Second pass at 8efdff13de. The seven comments from the first round are still open and aren't repeated here. The four inline comments are new. Below: complexity, simplifications, and a proposed split.
Complexity. 29 files; +724/−64 implementation, +1266/−59 tests, 12 new public error variants and types. The PR adds four recovery mechanisms that interact:
- A rejected delegated update is classified and the engine's forkchoice restored (
ConsolidateTask). - The delegated actor holds back rejected heads, released by a 5-minute timer or by finality moving.
- The sync start refuses a walk that misses finality; the reset restores and walks again.
- A finality backstop in
synchronize.
Most of the edge cases are in (1) and (2). #24061 records three Fable review rounds that each found new ones, and the first round here found five more (the HeadLabels loop, a stale el_applied_last_forkchoice twice, the SafeLabel repeat, the restart anchor).
Big simplifications
- Prevent instead of repair. The delegate's rollup RPC already serves
optimism_outputAtBlock. Before sending heads that move the EL's head, the delegated actor could ask the delegate for its block at the engine's finalized height and compare hashes, caching the answer per finalized head (one call per finality change). A delegate whose chain skips kona's finalized block would then never move the EL. Most of (1) and (2) goes away: theChainBelowFinalityclassification,is_off_finalized_chain, the consolidate-time restore, the hold timer and the cross-actor message. A rejected update that moves the head could reset as it does today, with (3) as the backstop for a delegate whose answers are inconsistent. - Scope to op-node parity first. op-node's
FollowSourcehas no restore. It assumes the source's refs are sane (engine_controller.go:1523) and retries anyFindL2Headsfailure. (3) without the restore, plus (4), would be the parity piece. "A delegate's head cannot loop the engine reset" goes beyond op-node and could follow once the stack lands.
Small simplifications (in addition to the inline comments and the earlier <= nit)
find_starting_forkchoice_restoring_finalitychooses between the engine's finalized head and the EL's finalized label as the anchor. After a rejected update, the EL's finalized label is the block the engine last sent it, soel_finalizedin both cases looks equivalent and drops a branch and a test case. This would also settle the first round's restart-anchor comment. (Medium confidence.)SafeLabelis exported only to name a field ofFinalizedAheadOfSafe. Keeping it crate-private shrinks the breaking change.
Split. Proposed pieces, in merge order, each reviewable on its own:
- Refactor only:
send_forkchoice/record_forkchoice_reply/with_update, with no behaviour change. - The sync start refuses a walk that misses the finalized block (
sync/mod.rs,sync/error.rs,is_temporary,finality_tests). The lower stack doesn't touch these files, so this piece can targetdevelopdirectly. - The reset restores a forkchoice on the finalized chain and walks again (
FinalityNotRestored,recovery_tests). - The
FinalizedAheadOfSafebackstop insynchronize, plus the sequencer arm. - Delegated mode only: classify a rejected update, then the hold (
consolidate/*, the delegated actor, the request plumbing, docs). With the first big simplification, this piece mostly disappears.
Pieces 2–4 change every kona-node reset or forkchoice update, sequencers and deriving verifiers included. As separate PRs, reviewers can judge that change without the delegated-mode context.
| @@ -58,12 +68,12 @@ pub(in crate::task_queue) async fn synchronize_unconditionally( | |||
| state_update: EngineSyncStateUpdate, | |||
| ) -> Result<(), SynchronizeTaskError> { | |||
| let new_sync_state = state.sync_state.apply_update(state_update); | |||
There was a problem hiding this comment.
🤖 Claude, for @joshklop: apply_update sets the kona_node_block_labels gauges before the new finality guard and before the EL answers. Until now a reset followed a rejected update and rewrote them. This PR turns a rejected delegated update into a skip, so the safe and local-safe gauges keep showing the rejected delegate heads until the next accepted update. During a chain-below-finality hold, that is five minutes or more. A FinalizedAheadOfSafe refusal does the same. Suggest building new_sync_state with the new with_update here and at line 37, and setting the gauges in send_and_apply_forkchoice next to state.sync_state = new_sync_state.
| /// provider without a block at that height lags and is retried; one with a different block there | ||
| /// follows another L1 chain, which retrying cannot fix; one with that very block there serves it | ||
| /// only by number, as a broken or lagging block-hash index does. | ||
| async fn finalized_l1_origin_missing_by_hash( |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: This extra L1 lookup only decides whether to retry or stop. FinalizedL1OriginNotCanonical is not temporary (core.rs:284), so a single L1 backend behind a load balancer that is on a stale fork at this height stops the node. op-node retries every FindL2Heads error through ResetEvent (engine_controller.go:1433-1436). Suggest dropping the lookup and folding NotCanonical and NotServedByHash into one temporary FinalizedL1OriginUnavailable. That saves an L1 round trip and two variants from the breaking change, and matches op-node.
|
|
||
| /// Tells derivation about delegated safe heads whose forkchoice update the execution layer | ||
| /// permanently rejected, so the delegate does not send them again before a reset. | ||
| async fn forward_permanently_rejected_delegated_safe_heads( |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: [simplification] The rejection reaches the delegated actor through a new request variant, a new EngineDerivationClient method, and an arm in the deriving actor that ignores it. The delegated actor already reads engine_state_rx. If EngineState kept the last permanent rejection (cleared by reset), the delegated actor could read it from the watch, or ConsolidateTask could skip matching heads before any RPC. That removes the channel round trip. It also closes a small race: heads polled before this message is processed still reach the engine, and cost another rejected update and restore.
| engine_finalized: self.engine_finalized_head(), | ||
| retry_at: Instant::now() + CHAIN_BELOW_FINALITY_RETRY_INTERVAL, | ||
| }, | ||
| DelegatedRejection::HeadLabels | DelegatedRejection::SafeLabel => { |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: [nit] SafeLabel never reaches this arm: forward_permanently_rejected_delegated_safe_heads returns early for it because is_permanent() is false. As written, the arm suggests a safe-label rejection is held until reset. Suggest unreachable!, or moving SafeLabel out of DelegatedRejection. It only ever skips one poll, so a separate ConsolidateTaskError variant fits it better.
Why
A delegate whose chain does not descend from kona-node's finalized block can leave the EL on its fork after a rejected forkchoice update, because op-geth and op-reth make the head canonical before rejecting it.
What
SYNCINGor rejected resets instead.ReorgFinalizedErr) and a reset restores and walks again, so a delegate's head cannot stop the node.Breakdown
Closes
Part of ethereum-optimism/protocol-team#545. Stack, in merge order: #23900 → #23899 → #23908 → #23947 → #24004 → #24005 → #24007 → #24065.
AI Reviews
BREAKING CHANGE:
ConsolidateTaskErrorgainsDelegatedForkchoiceRejectedandFailedToFetchFinalizedBlock;SynchronizeTaskErrorgainsFinalizedAheadOfSafe;SyncStartErrorgainsMismatchedFinalizedBlock { finalized, walked },FinalizedHeightUnavailable,FinalizedL1OriginUnavailable,FinalizedL1OriginNotServedByHashandFinalizedL1OriginNotCanonical;EngineResetErrorgainsFinalityNotRestored;DelegatedRejectionandSafeLabelare exported.🤖 Generated with Claude Code