Repository navigation
kona-engine,kona-node-service: retry delegated forkchoice updates and prevent finalized-head regression - #23900
Conversation
2417e68 to
537b357
Compare
12a2dec to
b628609
Compare
|
GPT-6.1-Sol (xhigh): Reviewed Checked delegated safe-head delivery, the distinction between applied and syncing forkchoices, intermediate span-batch updates, and reset invalidation of the acknowledged forkchoice.
The initial review covered this PR independently; combined validation is recorded below. Combined validation update (2026-11-06): locally merged the current heads of #23899 ( The merge preserves both sets of finalization guards and their regression tests. Test fixtures account for the acknowledged forkchoice status and the new identity-mismatch error.
Nextest flagged an open output handle in one passing, unchanged attribute-format test; its isolated rerun passed without the flag. The first acceptance attempts stopped at setup because the new checkout lacked the Cannon helper fixture; the complete runs above passed after restoring that fixture. |
b628609 to
7a290fc
Compare
einar-oplabs
left a comment
There was a problem hiding this comment.
GPT-6.1-Sol (xhigh): Reviewed 7a290fc5579c87f7f9f13fe2a45adac92baf34cc against fa3efe5b49ecc42007dfd4fcb79bd5d70f78f98f. No actionable findings.
Checked forced delivery for repeated delegated targets after SYNCING, unchanged forkchoice delivery during reset, the missing-block path preserving the unsafe sync target, and the distinction between delegated consolidation and intermediate span-batch attributes. Reviewed the RPC expectations and actor-level coverage alongside the production paths.
Validation: all 435 tests in kona-engine and kona-node-service passed at stack tip 454481f562f45c8c93022a6ed6090a226d12ce3c, including the downstream finalization changes in #23899 and #23908. Full acceptance devnets were not rerun during this review.
einar-oplabs
left a comment
There was a problem hiding this comment.
Fable 5.1 (xhigh): Reviewed 7a290fc557. One stack-ordering issue: the new follow path plus the same poll's finalize can exit the node if this PR lands before #23908. The rest are non-blocking nits on logging, duplication, ordering and test fixtures.
A delegated safe head below the unsafe head was treated like an intermediate span-batch block: it moved the consensus labels without a forkchoice update. An update that left the labels unchanged was also skipped, but a SYNCING answer moves the labels, so an identical retry never reached the execution layer. And while the execution layer was still syncing and lacked the delegated safe block, consolidation reset the engine, which restarted the sync (the delegated case of #23931). Only derived attributes inside a span batch now defer the update. The delegated consolidation paths and the reset always send their forkchoice, so each delegate poll is the retry after a SYNCING answer or an execution-layer restart. A delegated safe head whose block the syncing execution layer does not have yet moves only the safe labels, as in op-node's follow-source mode, instead of resetting; the missing block is logged at warn. One helper applies a delegated safe head with or without the unsafe head. The finalize that follows such a poll still fails on the same syncing execution layer until #23908 skips it, so this lands together with #23899 and #23908. `test_utils::l1_info_deposit` replaces three inline copies of the L1-info deposit, and the delegated consolidation table asserts what reaches the block sink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bf83ae6 to
c1c8522
Compare
A lagging delegate could move finality backward, and a delegate on another fork could name a finalized block that is not the execution layer's canonical block. The first is irreversible. The second either stopped the node or failed every forkchoice update and reset the engine on each poll. The finalize task now looks the target up by number and finalizes a pinned hash only when it is the canonical block at that height. A target behind the finalized head, a different block at the finalized height, or a pinned hash that is not canonical fails with an error of the new `Skip` severity: the engine drops the task, counts it as a failure, and the engine actor warns once per distinct error until a reset. A target equal to the finalized head needs no lookup or forkchoice update, and each finality guard runs once per kind of target. A block served at another height is a distinct `BlockNumberMismatch`. The delegate actor also rejects a whole sync status whose finalized L2 block is behind, or conflicts with, the engine's, so a lagging or diverged delegate's safe head does not reach the engine either. The baseline is the finalized head the engine applied, read from the state it publishes, not the last target forwarded to it: a target the engine skipped must not make the canonical block at its height look like a conflict, which would stop the delegate's safe head too. BREAKING CHANGE: `EngineTaskErrorSeverity` has a new `Skip` variant, and `FinalizeTaskError` has new variants `BehindFinalizedHead`, `BlockNumberMismatch`, `ConflictsWithFinalizedHead` and `NotCanonical`; exhaustive matches must handle them. A block served at another height is now `BlockNumberMismatch` instead of `BlockNotFound`. `DerivationActorRequest::ProcessEngineSyncCompletionRequest` carries an `EngineSyncState`, and `EngineDerivationClient::notify_sync_completed` takes one. `DelegateDerivationActor::new` takes the engine's state receiver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The safe head can still move backward (pre-existing, outside this PR). op-node promotes the cross-safe head only when it advances ( |
`synchronize` skips a forkchoice update that leaves the labels unchanged; `synchronize_unconditionally` always sends it. Both share a private `send_forkchoice`. The reset and the delegated paths of the consolidate task call `synchronize_unconditionally`; the insert, finalize and derived consolidate paths call `synchronize`. The consolidate task's thin wrappers `reconcile_to_safe_head`, `follow_safe_head_during_el_sync` and `apply_delegated_safe_head` are inlined at their call sites, keeping their logs and comments. Behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ensure_not_behind_finalized_head` and `ensure_no_conflict_with_finalized_head` each had one caller, so their checks move inline. A new test pins a target above the safe head: it fails with `BlockNotSafe` before any lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ProcessEngineSyncCompletionRequest` and `notify_sync_completed` carry the local-safe head again instead of the whole `EngineSyncState`. Both consumers only read the local-safe head; the delegate actor takes its finalized baseline from the engine's published state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its safe Such a snapshot is inconsistent: the engine would refuse to finalize it. op-node's follow mode rejects it the same way. A test also pins that any finalized block, genesis included, is accepted before the engine establishes finality, and the doc on the finality check now states that it only catches a fork at the engine's finalized height. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test sends stale finalizations through the actor: it keeps running, an identical repeat keeps the same skip in memory, and a reset forgets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Opus 5.5 (high): On the safe head moving backward: agreed, and out of scope for this PR. #24004 is the follow-up. There, a delegated safe head below the engine's safe head is ignored unless the EL proves divergence, which matches op-node promoting only when |
…d-finality kona-engine: prevent delegated finalized-head regression
…lidation block Kona-node resets whenever derived attributes name a block the EL lacks below the unsafe head, so after an EL datadir wipe it re-derives from genesis in a loop and finalization stays at genesis. While the EL answers `SYNCING`, the task now waits (new `Wait` severity): it is parked with its own 2s retry deadline, tasks of other kinds (imports, builds, finalizations) keep running, and nothing resets, so the finalization queue survives. Each retry looks the block up again; every 30s it also reads the EL's sync progress and resends the unchanged forkchoice, so an EL that reached the unsafe head is consolidated, and a `VALID` answer records the EL as synced. An EL idle at the derived block's parent, reporting no sync, gets the block built from the attributes instead. Progress, or its absence, is logged every 5 minutes; the engine resets once progress and latest block stop moving for 24 h (`--l2.el-sync-stall-timeout`), as neither op-geth nor op-reth reports progress while downloading headers. Failed lookups and probes are Temporary. The delegated path is #23900. Part of #23931. BREAKING CHANGE: `EngineTaskErrorSeverity::Wait`; `ConsolidateTaskError::{MissingUnsafeL2BlockDuringElSync, MissingUnsafeL2BlockProbeFailed}`; `Engine::drain` takes a `DrainScope`; `EngineConfig` gains `el_sync_stall_timeout` (default `DEFAULT_EL_SYNC_STALL_TIMEOUT`); `ConsolidateTask::new` is non-const and takes the stall timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lidation block Kona-node resets whenever derived attributes name a block the EL lacks below the unsafe head, so after an EL datadir wipe it re-derives from genesis in a loop and finalization stays at genesis. While the EL answers `SYNCING`, the task now waits (new `Wait` severity): it is parked with its own 2s retry deadline, tasks of other kinds (imports, builds, finalizations) keep running, and nothing resets, so the finalization queue survives. Each retry looks the block up again; every 30s it also reads the EL's sync progress and resends the unchanged forkchoice, so an EL that reached the unsafe head is consolidated, and a `VALID` answer records the EL as synced. An EL idle at the derived block's parent, reporting no sync, gets the block built from the attributes instead. Progress, or its absence, is logged every 5 minutes; the engine resets once progress and latest block stop moving for 24 h (`--l2.el-sync-stall-timeout`), as neither op-geth nor op-reth reports progress while downloading headers. Failed lookups and probes are Temporary. The delegated path is #23900. Part of #23931. BREAKING CHANGE: `EngineTaskErrorSeverity::Wait`; `ConsolidateTaskError::{MissingUnsafeL2BlockDuringElSync, MissingUnsafeL2BlockProbeFailed}`; `Engine::drain` takes a `DrainScope`; `EngineConfig` gains `el_sync_stall_timeout` (default `DEFAULT_EL_SYNC_STALL_TIMEOUT`); `ConsolidateTask::new` is non-const and takes the stall timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
Kona-node's delegated derivation could leave the EL's safe head behind its own and never retry a forkchoice update after SYNCING; a stale or forked delegate could move the finalized head backward or name a non-canonical finalized block, stopping or resetting kona every poll.
What
synchronizesends only changed labels andsynchronize_unconditionallyalways sends; resets and delegated consolidation use the latter, so each 4s delegate poll retries after SYNCING (op-node fix(op-node): pipeline reset to handle EL restarts gracefully #20310), and a delegated safe head a syncing EL lacks moves only the safe labels (delegated case of kona-node: missing L2 block during EL catch-up resets the engine in a loop #23931).Skipseverity, and the delegate actor rejects a status whose finalized L2 is above its safe L2 or behind or conflicting with the engine's applied finality.BlockNotFound/BlockNotSafeskips during catch-up. Part of ethereum-optimism/protocol-team#545; follows up on kona-node!: deliver and retry delegated forkchoice updates #23680 and kona-node!: prevent finalized-head regression from stale targets #23679. Stack: kona-engine,kona-node-service: retry delegated forkchoice updates and prevent finalized-head regression #23900 (+kona-engine: prevent delegated finalized-head regression #23899) → kona-engine: keep delegated finalization pending during EL catch-up #23908 → op-devstack,op-acceptance-tests: run the follow-source tests against kona-node #23947 → kona-engine,kona-node-service: keep a stale delegated safe head from moving safe below finality #24004 → kona-engine,kona-node-service: import delegated local-safe and cross-safe heads separately #24005 → kona-engine: wait for EL sync instead of resetting on a missing consolidation block #24006 → kona-node-service,op-acceptance-tests: exercise delegated derivation recovery end to end #24007 → kona-engine,kona-node-service: recover when the EL rejects a delegated forkchoice update #24065.Breakdown
Closes
AI Reviews
BREAKING CHANGE:
EngineTaskErrorSeverity::SkipandFinalizeTaskError::{BehindFinalizedHead, BlockNumberMismatch, ConflictsWithFinalizedHead, NotCanonical}are new; exhaustive matches must handle them. A wrong-height block is no longerBlockNotFound.DelegateDerivationActor::newtakes the engine's state receiver.🤖 Generated with Claude Code