Repository navigation
Description
Summary
Kona's delegated sequencer on joshklop-2-0 exited six times during startup/recovery with EngineTask(Finalize(BlockNotSafe)). The last crash immediately followed a second engine reset that restored a safe head below the finalized head just reported by the delegate. The next startup recovered and finalized the same delegated block successfully.
The fatal failure is confirmed from Kubernetes termination status and logs. A reset/delegated-update ordering problem is strongly suggested, but the exact task interleaving has not yet been reproduced or proven.
Observed deployment and impact
- Observed October 28, 2026, on the
joshklop-2-0personal devnet, Kona sequencerseq-2, configured with delegated derivation. - Deployed Kona image tag:
3faff12ecf1de0cd61fd35de1e38127db4a1ed7a; image digest:sha256:72cf937cc93f7c93131ed9e6ac20f6068941650c54ef861e3f48118bb8aae5a2. - Execution client: op-reth v2.7.0. Rollup block time: 1 second.
- Six matching service failures at 12:52:53.024, 12:52:55.342, 12:53:08.541, 12:53:32.482, 12:54:25.467, and 12:55:48.598 UTC.
- Kubernetes recorded six container restarts; the last instance ran from 12:55:46 to 12:55:48 UTC and terminated with reason
Error, exit code 1, notOOMKilled. - The current container started at 12:58:31 UTC and subsequently remained healthy. The network was producing blocks normally at investigation time; this report does not establish a network-wide outage during the crash interval.
Last-crash sequence
Selected logs below omit hashes, ANSI formatting, and unrelated fields; timestamps and heights are preserved.
12:55:46.585909 Loaded current L2 EL forkchoice state unsafe=136795 safe=134984 finalized=134984
12:55:47.526736 Found suitable L2 safe block l2_safe=134984
12:55:47.527276 Finished execution layer sync [safe/finalized=134984]
12:55:47.527310 Sent reset signal to derivation actor
12:55:47.590466 Processed sync status from delegate safe_l2=137189 finalized_l2=136079
12:55:47.593637 Loaded current L2 EL forkchoice state unsafe=136795 safe=134984 finalized=134984
12:55:48.597671 Found suitable L2 safe block l2_safe=134984
12:55:48.598261 Sent reset signal to derivation actor
12:55:48.598279 Critical error draining engine tasks err=Finalize(BlockNotSafe)
12:55:48.598381 Critical error in sub-routine: EngineTask(Finalize(BlockNotSafe))
12:55:48.598445 Failed to start rollup node service: EngineTask(Finalize(BlockNotSafe))
Error: EngineTask(Finalize(BlockNotSafe))
On the successful startup at 12:58:32 UTC, reset again selected safe 134984; the delegate then reported safe 137189 and finalized 136079, and Kona logged Updated finalized head ... number=136079 at 12:58:32.981733. The finalized block hash matched the failed startup's delegate report. This argues against permanently invalid delegate data, but does not by itself prove the precise race.
Mechanism and remaining uncertainty
At the deployed revision, FinalizeTask::execute returns BlockNotSafe when the requested finalization height exceeds the engine's current safe head. FinalizeTaskError::severity classifies this as Critical, terminating the service.
The observed order is consistent with a delegated finalization request crossing a reset boundary and executing before the corresponding safe promotion has been restored. The fatal log does not include the task's target height: 136079 is the immediately preceding delegated target, not a directly logged task argument. The queue/consolidation/reset interleaving still needs a deterministic reproduction.
Expected behavior / acceptance criteria
- Add a regression test through the actual engine queue/actor reset path: receive delegated safe/finalized updates, reset below the pending finalized target, then process pending work. The transient ordering must not terminate Kona.
- Reconcile, invalidate, or defer stale pending finalization work across reset boundaries without blocking unsafe imports or creating a tight retry/reset loop.
- Once the matching safe state and finalized block are available, eventual finalization succeeds; cover repeated resets and missing-then-present execution-client data.
- Preserve the invariant that finalized is on the accepted safe chain. Do not simply remove the guard, clamp an arbitrary target, substitute a different block at the same height, or silently accept conflicting finality.
- Preserve hash-pinned block identity validation and fail safely for genuinely inconsistent or conflicting finalized data.
Related work and scope
- Delegated derivation reliability tracker: https://github.com/ethereum-optimism/protocol-team/issues/545.
- kona-node: keep delegated finalization pending during EL catch-up #23675 address missing delegated finalized blocks during EL catch-up (
BlockNotFound); the existing catch-up implementation leavesBlockNotSafecritical and does not demonstrate this reset-ordering case is fixed. - kona-node: reliably deliver and retry delegated forkchoice updates #23676 address safe-head forkchoice delivery and retrying
SYNCING; these may affect the preconditions, but coverage of this specific crash has not been established. - op-acceptance-tests,op-devstack: cover forkchoice updates with a stale safe or finalized hash #23440's CI follow-up documents the same fatal
Finalize(BlockNotSafe)after a reset lowers safe during datadir-wipe recovery, and explicitly calls for a dedicated issue. The live trigger here is startup with delegated derivation; no datadir wipe was established. - kona-node: L2Finalizer emits finalization targets above the cross-safe head #22582 concerns native interop finalizer targets above cross-safe. This report concerns delegated startup/reset recovery, so that issue alone does not establish coverage.
Activity
einar-oplabs commentedon Nov 4, 2026
Draft fix: #23908 (also closes #23675). A delegated hash-pinned finalize target above the local safe head is now logged and skipped instead of returning
BlockNotSafe(Critical), and the next delegate poll resubmits it, matching op-node'spromoteFinalized. Finality still only moves along the accepted safe chain, and by-number finalization from local L1 finality still fails hard.Regression test
delegated_finalization_survives_resets_below_targetdrives the realEngineActor. A reset restores safe below the pinned target, the target is skipped, an unsafe import still runs, and a second reset repeats the drop. The same target then finalizes once the delegated safe head arrives, and the actor goes idle with no retry or reset loop. With only the guard removed, the test fails withFinalize(BlockNotSafe), the devnet error.Not covered: the #23440 datadir-wipe variant, if it goes through by-number finalization, and the delegate actor's own poll loop, which has no unit-test harness.
kona-engine: keep delegated finalization pending during EL catch-up
kona-engine: keep delegated finalization pending during EL catch-up
kona-engine: keep delegated finalization pending during EL catch-up
kona-engine: keep delegated finalization pending during EL catch-up