Skip to content

kona-node-service,op-acceptance-tests: exercise delegated derivation recovery end to end - #24007

Open
einar-oplabs wants to merge 6 commits into
einar/23911-delegated-cross-safefrom
einar/23912-delegated-parity-scenarios
Open

einar-oplabs wants to merge 6 commits into
einar/23911-delegated-cross-safefrom
einar/23912-delegated-parity-scenarios

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

#23947 runs the follow-source tests against kona-node, but they pass on develop too, so catch-up, SYNCING→VALID, stale finality and reset recovery are still unexercised.

What

Breakdown

Added Removed Net
Implementation 29 16 13
Tests 865 33 832
Docs 0 0 0

Closes

Part of ethereum-optimism/protocol-team#545; real-EL fault-injection gaps are listed there. Stack, in merge order: #23900 → #23899 → #23908 → #23947 → #24004 → #24005 → #24007 → #24065.

  1. Closes: kona-node: verify delegated derivation parity against op-node #23912

AI Reviews

  • 2026-11-10T14:16:29Z 2601b40 Fable 5.1 xhigh
  • 2026-11-11T13:20:56Z 8497400 Fable 5.1 xhigh
  • 2026-11-11T14:27:43Z 1dfdbb7 Fable 5.1 xhigh
  • 2026-11-11T16:37:35Z 6e50e73 Fable 5.1 xhigh

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Nov 9, 2026
@einar-oplabs
einar-oplabs force-pushed the einar/23912-delegated-parity-scenarios branch from 00fd8fc to 77e5719 Compare November 9, 2026 18:36
@einar-oplabs
einar-oplabs force-pushed the einar/23912-delegated-parity-scenarios branch from 77e5719 to 2601b40 Compare November 10, 2026 14:04
@einar-oplabs
einar-oplabs force-pushed the einar/23912-delegated-parity-scenarios branch from 6e50e73 to e12273b Compare November 11, 2026 17:17
@einar-oplabs
einar-oplabs marked this pull request as ready for review November 11, 2026 17:17
@einar-oplabs
einar-oplabs requested a review from a team as a code owner November 11, 2026 17:17
@joshklop
joshklop added this pull request to stack #24203 November 16, 2026 20:35
einar-oplabs and others added 5 commits November 17, 2026 10:12
…kona-node

The follow_l2 tests skipped on kona-node because the single-chain test
sequencer requires an op-node CL, and the two-verifiers preset applied
DEVSTACK_L2CL_KIND to every node. The preset now always runs the
sequencer and the deriving verifier "b" on op-node, and only the
follow-source verifier "c" takes the env-selected CL. Under kona-node
that checks kona's follow-source mode against an op-node upstream;
under op-node every node keeps its CL. The preset also no longer starts
a proposer or challenger, which no follow_l2 test uses, and accepts
only the options that runtime honors.

The single-chain CL starters take the CL kind as an argument instead of
reading DEVSTACK_L2CL_KIND, each preset resolves it once, and an
unknown kind, the empty string included, fails instead of running
op-node. Presets with interop at genesis pin their sequencer to
op-node, which they already required.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TestFollowL2_ReorgRecovery could pass with the follow-source verifier and
the deriving verifier both reset to genesis. The follow-source verifier
must now hold the reorged block as local-safe before the L1 reorg, replace
it in its EL with the deriving verifier's block, and move its local-safe
head past that height. TestFollowL2_WithoutCLP2P also checks that the
finalized head follows the deriving verifier after reconnecting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResolveMixedL2CLKind takes the test handle and fails on a value other than
op-node or kona-node, so an invalid kind fails before any node starts
instead of when the first node of that kind starts. The two-verifiers
runtime reads the follow-source verifier's kind into a local in place of
singleChainTwoVerifiersCLKinds, the multi-node runtimes call the exported
minimal-runtime constructors, the CL-kind starters use FailNowf, and the
README says that some presets pin nodes to op-node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…moving safe below finality

A stale delegate snapshot can report a safe head below kona-node's
finalized head, and kona-node applies it, moving safe behind its own
finality.

Kona-node now drops a delegate status whose labels are out of order (a
zero local-safe head counts as unreported) or that breaks the engine's
applied finality, and the engine skips a delegated safe head below, or
conflicting with, established finality. A delegated safe head below safe
is ignored, and forwarded only once per engine view, unless an EL that
applied the last forkchoice holds a different block at that height,
which proves divergence and reorgs the engine above finality. Labels
left with unsafe below safe are repaired by the next delegated head
above finality. Repeated rejection warnings are deduplicated. A rejected
delegated forkchoice update resets the engine as on develop; recovery is
#24065 (#24061). Known window: after a delegate L1 reorg whose fork
point is above its new safe head, safe stays on the old fork until the
delegate passes it.

Closes #23910.

BREAKING CHANGE: `ConsolidateTaskError` gains `BreaksFinality` and
`FailedToFetchDelegatedSafeBlock`;
`FinalizeTaskError::{BehindFinalizedHead, ConflictsWithFinalizedHead}`
become `BreaksFinality(FinalityViolation)`; `FinalityViolation` is
exported; `EngineState` gains `el_applied_last_forkchoice`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…safe heads separately

Under interop op-node imports a delegate's local-safe and cross-safe
heads separately; kona-node copies the delegated safe head into both, so
its forkchoice safe hash can run ahead of cross-chain verification.

Kona-node now follows local-safe for the EL lookup, EL sync and reorgs,
and moves the safe label to cross-safe only when it advances. Only the
unsafe head and established finality count as proof of the chain; other
heights are checked against an EL that applied the last forkchoice, and
only at or below the unsafe head. A head that does not extend the unsafe
chain reorgs directly, moving safe to cross-safe, clamped to finalized,
unless the EL holds the current safe label, which a catch-up then keeps;
only a conflict with finalized skips the update. While the EL has not
applied the last forkchoice, safe only moves forward and nothing reorgs.
Inconsistent heads are rejected, and the local-safe L1 origin must be
canonical. Before interop both heads are one block, so non-interop
chains see no change. A rejected update resets the engine; recovery is
#24065.

Closes #23911.

BREAKING CHANGE: `ConsolidateInput::BlockInfo` is replaced by
`Delegated(Box<DelegatedSafeHeads>)`, built with
`DelegatedSafeHeads::try_new`, which returns
`InconsistentSafeHeadsError`; the `From<L2BlockInfo>` conversions are
removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop force-pushed the einar/23912-delegated-parity-scenarios branch from e12273b to ecb3f40 Compare November 19, 2026 00:54

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: Review of ecb3f40431. No bugs. The only non-test change moves the retry and poll constants to module level with the same values. The scenarios run on paused time against an in-process mock, so they look deterministic, and any unexpected EL request fails them. The catch-up and regressed-finality scenarios do fail without their fixes. The acceptance test pauses the sequencer cleanly and compares full block references.

Coverage gap: no scenario sends a delegated safe head below the unsafe head that the EL lacks (follow_safe_head_during_el_sync), so WAIT_RETRY_INTERVAL only feeds QUIET_TIME. #24006's Wait only happens in derivation mode, so its only coverage is the unit tests in queue_tests.rs. If either path is one of #545's completion criteria, it's still uncovered end to end.

};

/// How long the engine may take to reach the labels a step expects.
const LABELS_DEADLINE: Duration = Duration::from_secs(1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: [nit] LABELS_DEADLINE is a fixed 1 s. That's deterministic today. But a future scenario that reaches its labels only after a backoff (100+200+400+800 ms > 1 s) or a 2 s wait would fail with "did not reach the expected labels", which hides the cause. Deriving it from QUIET_TIME would avoid that.

if number == 0 {
return L1_ORIGIN;
}
BlockInfo { hash: B256::with_last_byte(number as u8), number, ..Default::default() }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: [nit] number as u8 silently gives duplicate hashes past block 255. It's fine at today's CHAIN_LENGTH, but deriving the hash from the full number, or asserting the bound, would make it safe.


/// Local-safe and cross-safe labels move separately; finalizing above cross-safe is rejected whole.
#[tokio::test(start_paused = true)]
async fn local_safe_ahead_of_cross_safe_moves_each_label_separately_and_finalizes_below_cross_safe()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: [nit] This scenario reports L1_ORIGIN as current_l1 but never calls assert_origin. Also, its "finalized 25 above safe 22" step is rejected by the delegate's FinalizedAheadOfSafe check, so the engine never sees it. "finalizes below cross-safe" therefore describes delegate validation, not engine behaviour.

…recovery end to end

#23947 runs the follow-source tests against kona-node, but they pass on
develop too, so catch-up, SYNCING→VALID, stale finality and reset
recovery are still unexercised.

This adds end-to-end scenarios that run the real delegate and engine
actors against a scripted delegate and a strict Engine API mock that
checks every forkchoice update and lookup by hash, one per #545
completion criterion, plus local-safe ahead of cross-safe and a reorg
that the EL first answers `SYNCING`. Each step waits for the engine's
published labels and ends with a quiet window longer than the engine's
longest retry, the poll cadence and the published derivation origin are
asserted, and a stopped actor reports its own panic. A follow_l2
acceptance test pauses the sequencer once kona-node follows its source
(pausing earlier strands it at genesis, #24030), waits on both layers
with progress-bounded waits, and compares kona-node's labels with its EL
and the sequencer's chain by hash. The scenarios found the
forwarded-versus-applied finalized bug fixed in #23899. Production code
changes only in retry and poll-interval constants becoming module-level.

Closes #23912. Gaps that need real-EL fault injection are listed on
ethereum-optimism/protocol-team#545.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop removed this pull request from stack #24203 November 19, 2026 02:00
@joshklop
joshklop changed the base branch from einar/23931-el-sync-reset-loop to einar/23911-delegated-cross-safe November 19, 2026 02:00
@joshklop
joshklop requested a review from a team as a code owner November 19, 2026 02:00
@joshklop
joshklop force-pushed the einar/23912-delegated-parity-scenarios branch from ecb3f40 to 8854cc0 Compare November 19, 2026 02:00
@joshklop
joshklop added this pull request to stack #24271 November 19, 2026 02:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kona-node: verify delegated derivation parity against op-node

2 participants