Skip to content

kona-node: fatal Finalize(BlockNotSafe) during delegated startup/reset recovery #23757

Closed

Description

@joshklop

Summary

Kona's delegated sequencer on joshklop-2-0 exited six times during startup/recovery with EngineTask(Finalize(BlockNotSafe)). The last crash immediately followed a second engine reset that restored a safe head below the finalized head just reported by the delegate. The next startup recovered and finalized the same delegated block successfully.

The fatal failure is confirmed from Kubernetes termination status and logs. A reset/delegated-update ordering problem is strongly suggested, but the exact task interleaving has not yet been reproduced or proven.

Observed deployment and impact

  • Observed October 28, 2026, on the joshklop-2-0 personal devnet, Kona sequencer seq-2, configured with delegated derivation.
  • Deployed Kona image tag: 3faff12ecf1de0cd61fd35de1e38127db4a1ed7a; image digest: sha256:72cf937cc93f7c93131ed9e6ac20f6068941650c54ef861e3f48118bb8aae5a2.
  • Execution client: op-reth v2.7.0. Rollup block time: 1 second.
  • Six matching service failures at 12:52:53.024, 12:52:55.342, 12:53:08.541, 12:53:32.482, 12:54:25.467, and 12:55:48.598 UTC.
  • Kubernetes recorded six container restarts; the last instance ran from 12:55:46 to 12:55:48 UTC and terminated with reason Error, exit code 1, not OOMKilled.
  • The current container started at 12:58:31 UTC and subsequently remained healthy. The network was producing blocks normally at investigation time; this report does not establish a network-wide outage during the crash interval.

Last-crash sequence

Selected logs below omit hashes, ANSI formatting, and unrelated fields; timestamps and heights are preserved.

12:55:46.585909 Loaded current L2 EL forkchoice state unsafe=136795 safe=134984 finalized=134984
12:55:47.526736 Found suitable L2 safe block l2_safe=134984
12:55:47.527276 Finished execution layer sync [safe/finalized=134984]
12:55:47.527310 Sent reset signal to derivation actor
12:55:47.590466 Processed sync status from delegate safe_l2=137189 finalized_l2=136079
12:55:47.593637 Loaded current L2 EL forkchoice state unsafe=136795 safe=134984 finalized=134984
12:55:48.597671 Found suitable L2 safe block l2_safe=134984
12:55:48.598261 Sent reset signal to derivation actor
12:55:48.598279 Critical error draining engine tasks err=Finalize(BlockNotSafe)
12:55:48.598381 Critical error in sub-routine: EngineTask(Finalize(BlockNotSafe))
12:55:48.598445 Failed to start rollup node service: EngineTask(Finalize(BlockNotSafe))
Error: EngineTask(Finalize(BlockNotSafe))

On the successful startup at 12:58:32 UTC, reset again selected safe 134984; the delegate then reported safe 137189 and finalized 136079, and Kona logged Updated finalized head ... number=136079 at 12:58:32.981733. The finalized block hash matched the failed startup's delegate report. This argues against permanently invalid delegate data, but does not by itself prove the precise race.

Mechanism and remaining uncertainty

At the deployed revision, FinalizeTask::execute returns BlockNotSafe when the requested finalization height exceeds the engine's current safe head. FinalizeTaskError::severity classifies this as Critical, terminating the service.

The observed order is consistent with a delegated finalization request crossing a reset boundary and executing before the corresponding safe promotion has been restored. The fatal log does not include the task's target height: 136079 is the immediately preceding delegated target, not a directly logged task argument. The queue/consolidation/reset interleaving still needs a deterministic reproduction.

Expected behavior / acceptance criteria

  • Add a regression test through the actual engine queue/actor reset path: receive delegated safe/finalized updates, reset below the pending finalized target, then process pending work. The transient ordering must not terminate Kona.
  • Reconcile, invalidate, or defer stale pending finalization work across reset boundaries without blocking unsafe imports or creating a tight retry/reset loop.
  • Once the matching safe state and finalized block are available, eventual finalization succeeds; cover repeated resets and missing-then-present execution-client data.
  • Preserve the invariant that finalized is on the accepted safe chain. Do not simply remove the guard, clamp an arbitrary target, substitute a different block at the same height, or silently accept conflicting finality.
  • Preserve hash-pinned block identity validation and fail safely for genuinely inconsistent or conflicting finalized data.

Related work and scope

Activity

  1. einar-oplabs commented on Nov 4, 2026

    @einar-oplabs
    Contributor

    Draft fix: #23908 (also closes #23675). A delegated hash-pinned finalize target above the local safe head is now logged and skipped instead of returning BlockNotSafe (Critical), and the next delegate poll resubmits it, matching op-node's promoteFinalized. Finality still only moves along the accepted safe chain, and by-number finalization from local L1 finality still fails hard.

    Regression test delegated_finalization_survives_resets_below_target drives the real EngineActor. A reset restores safe below the pinned target, the target is skipped, an unsafe import still runs, and a second reset repeats the drop. The same target then finalizes once the delegated safe head arrives, and the actor goes idle with no retry or reset loop. With only the guard removed, the test fails with Finalize(BlockNotSafe), the devnet error.

    Not covered: the #23440 datadir-wipe variant, if it goes through by-number finalization, and the delegate actor's own poll loop, which has no unit-test harness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions