Repository navigation
kona-node-service,op-acceptance-tests: exercise delegated derivation recovery end to end - #24007
einar-oplabs wants to merge 6 commits into
Conversation
00fd8fc to
77e5719
Compare
77e5719 to
2601b40
Compare
6e50e73 to
e12273b
Compare
…kona-node The follow_l2 tests skipped on kona-node because the single-chain test sequencer requires an op-node CL, and the two-verifiers preset applied DEVSTACK_L2CL_KIND to every node. The preset now always runs the sequencer and the deriving verifier "b" on op-node, and only the follow-source verifier "c" takes the env-selected CL. Under kona-node that checks kona's follow-source mode against an op-node upstream; under op-node every node keeps its CL. The preset also no longer starts a proposer or challenger, which no follow_l2 test uses, and accepts only the options that runtime honors. The single-chain CL starters take the CL kind as an argument instead of reading DEVSTACK_L2CL_KIND, each preset resolves it once, and an unknown kind, the empty string included, fails instead of running op-node. Presets with interop at genesis pin their sequencer to op-node, which they already required. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TestFollowL2_ReorgRecovery could pass with the follow-source verifier and the deriving verifier both reset to genesis. The follow-source verifier must now hold the reorged block as local-safe before the L1 reorg, replace it in its EL with the deriving verifier's block, and move its local-safe head past that height. TestFollowL2_WithoutCLP2P also checks that the finalized head follows the deriving verifier after reconnecting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResolveMixedL2CLKind takes the test handle and fails on a value other than op-node or kona-node, so an invalid kind fails before any node starts instead of when the first node of that kind starts. The two-verifiers runtime reads the follow-source verifier's kind into a local in place of singleChainTwoVerifiersCLKinds, the multi-node runtimes call the exported minimal-runtime constructors, the CL-kind starters use FailNowf, and the README says that some presets pin nodes to op-node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…moving safe below finality A stale delegate snapshot can report a safe head below kona-node's finalized head, and kona-node applies it, moving safe behind its own finality. Kona-node now drops a delegate status whose labels are out of order (a zero local-safe head counts as unreported) or that breaks the engine's applied finality, and the engine skips a delegated safe head below, or conflicting with, established finality. A delegated safe head below safe is ignored, and forwarded only once per engine view, unless an EL that applied the last forkchoice holds a different block at that height, which proves divergence and reorgs the engine above finality. Labels left with unsafe below safe are repaired by the next delegated head above finality. Repeated rejection warnings are deduplicated. A rejected delegated forkchoice update resets the engine as on develop; recovery is #24065 (#24061). Known window: after a delegate L1 reorg whose fork point is above its new safe head, safe stays on the old fork until the delegate passes it. Closes #23910. BREAKING CHANGE: `ConsolidateTaskError` gains `BreaksFinality` and `FailedToFetchDelegatedSafeBlock`; `FinalizeTaskError::{BehindFinalizedHead, ConflictsWithFinalizedHead}` become `BreaksFinality(FinalityViolation)`; `FinalityViolation` is exported; `EngineState` gains `el_applied_last_forkchoice`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…safe heads separately Under interop op-node imports a delegate's local-safe and cross-safe heads separately; kona-node copies the delegated safe head into both, so its forkchoice safe hash can run ahead of cross-chain verification. Kona-node now follows local-safe for the EL lookup, EL sync and reorgs, and moves the safe label to cross-safe only when it advances. Only the unsafe head and established finality count as proof of the chain; other heights are checked against an EL that applied the last forkchoice, and only at or below the unsafe head. A head that does not extend the unsafe chain reorgs directly, moving safe to cross-safe, clamped to finalized, unless the EL holds the current safe label, which a catch-up then keeps; only a conflict with finalized skips the update. While the EL has not applied the last forkchoice, safe only moves forward and nothing reorgs. Inconsistent heads are rejected, and the local-safe L1 origin must be canonical. Before interop both heads are one block, so non-interop chains see no change. A rejected update resets the engine; recovery is #24065. Closes #23911. BREAKING CHANGE: `ConsolidateInput::BlockInfo` is replaced by `Delegated(Box<DelegatedSafeHeads>)`, built with `DelegatedSafeHeads::try_new`, which returns `InconsistentSafeHeadsError`; the `From<L2BlockInfo>` conversions are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e12273b to
ecb3f40
Compare
joshklop
left a comment
There was a problem hiding this comment.
🤖 Claude, for @joshklop: Review of ecb3f40431. No bugs. The only non-test change moves the retry and poll constants to module level with the same values. The scenarios run on paused time against an in-process mock, so they look deterministic, and any unexpected EL request fails them. The catch-up and regressed-finality scenarios do fail without their fixes. The acceptance test pauses the sequencer cleanly and compares full block references.
Coverage gap: no scenario sends a delegated safe head below the unsafe head that the EL lacks (follow_safe_head_during_el_sync), so WAIT_RETRY_INTERVAL only feeds QUIET_TIME. #24006's Wait only happens in derivation mode, so its only coverage is the unit tests in queue_tests.rs. If either path is one of #545's completion criteria, it's still uncovered end to end.
| }; | ||
|
|
||
| /// How long the engine may take to reach the labels a step expects. | ||
| const LABELS_DEADLINE: Duration = Duration::from_secs(1); |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: [nit] LABELS_DEADLINE is a fixed 1 s. That's deterministic today. But a future scenario that reaches its labels only after a backoff (100+200+400+800 ms > 1 s) or a 2 s wait would fail with "did not reach the expected labels", which hides the cause. Deriving it from QUIET_TIME would avoid that.
| if number == 0 { | ||
| return L1_ORIGIN; | ||
| } | ||
| BlockInfo { hash: B256::with_last_byte(number as u8), number, ..Default::default() } |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: [nit] number as u8 silently gives duplicate hashes past block 255. It's fine at today's CHAIN_LENGTH, but deriving the hash from the full number, or asserting the bound, would make it safe.
|
|
||
| /// Local-safe and cross-safe labels move separately; finalizing above cross-safe is rejected whole. | ||
| #[tokio::test(start_paused = true)] | ||
| async fn local_safe_ahead_of_cross_safe_moves_each_label_separately_and_finalizes_below_cross_safe() |
There was a problem hiding this comment.
🤖 Claude, for @joshklop: [nit] This scenario reports L1_ORIGIN as current_l1 but never calls assert_origin. Also, its "finalized 25 above safe 22" step is rejected by the delegate's FinalizedAheadOfSafe check, so the engine never sees it. "finalizes below cross-safe" therefore describes delegate validation, not engine behaviour.
…recovery end to end #23947 runs the follow-source tests against kona-node, but they pass on develop too, so catch-up, SYNCING→VALID, stale finality and reset recovery are still unexercised. This adds end-to-end scenarios that run the real delegate and engine actors against a scripted delegate and a strict Engine API mock that checks every forkchoice update and lookup by hash, one per #545 completion criterion, plus local-safe ahead of cross-safe and a reorg that the EL first answers `SYNCING`. Each step waits for the engine's published labels and ends with a quiet window longer than the engine's longest retry, the poll cadence and the published derivation origin are asserted, and a stopped actor reports its own panic. A follow_l2 acceptance test pauses the sequencer once kona-node follows its source (pausing earlier strands it at genesis, #24030), waits on both layers with progress-bounded waits, and compares kona-node's labels with its EL and the sequencer's chain by hash. The scenarios found the forwarded-versus-applied finalized bug fixed in #23899. Production code changes only in retry and poll-interval constants becoming module-level. Closes #23912. Gaps that need real-EL fault injection are listed on ethereum-optimism/protocol-team#545. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ecb3f40 to
8854cc0
Compare
Why
#23947 runs the follow-source tests against kona-node, but they pass on develop too, so catch-up, SYNCING→VALID, stale finality and reset recovery are still unexercised.
What
Breakdown
Closes
Part of ethereum-optimism/protocol-team#545; real-EL fault-injection gaps are listed there. Stack, in merge order: #23900 → #23899 → #23908 → #23947 → #24004 → #24005 → #24007 → #24065.
AI Reviews
🤖 Generated with Claude Code