Skip to content

op-devstack,op-acceptance-tests: run the follow-source tests against kona-node - #23947

Merged
geoknee merged 3 commits into
developfrom
einar/23912-delegated-parity-verification
Nov 19, 2026
Merged

geoknee merged 3 commits into
developfrom
einar/23912-delegated-parity-verification

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Why

The follow-source acceptance tests skipped on kona-node, so kona's follow-source mode was never checked against an op-node upstream.

What

  • The two-verifiers preset keeps the sequencer and the deriving verifier b on op-node; only the follow-source verifier c takes DEVSTACK_L2CL_KIND, and it no longer starts a proposer or challenger.
  • ResolveMixedL2CLKind fails on an unknown DEVSTACK_L2CL_KIND before any node starts, and the CL starters take the resolved kind; presets with interop at genesis pin their sequencer to op-node (op-devstack: run kona-node as sequencer on interop-at-genesis single-chain presets #24121).
  • All three follow_l2 tests run on kona-node; TestFollowL2_ReorgRecovery now requires c to recover past the reorged block.

Breaking Changes

  • sysgo.ResolveMixedL2CLKind now takes a devtest.T.

Breakdown

Added Removed Net
Implementation 131 59 72
Tests 85 45 40
Docs 1 1 0

Closes

Part of #23912 and ethereum-optimism/protocol-team#545. Stack, in merge order: #23908 → #23947 → #24004 → #24005 → #24007 → #24065 (#23900 and #23899 merged). Follow-ups: #24121, #24122.

  1. Closes: op-acceptance-tests: run TestFollowL2_Safe_Finalized_CurrentL1 against kona-node #22272
  2. Closes: op-acceptance-tests: run TestFollowL2_ReorgRecovery against kona-node #22273
  3. Closes: op-acceptance-tests: run TestFollowL2_WithoutCLP2P against kona-node #22274

AI Reviews

  • 2026-11-07T00:26:11Z 454481f GPT-6.1-Sol xhigh
  • 2026-11-07T00:55:48Z 454481f Fable 5.1 xhigh

🤖 Generated with Claude Code

@einar-oplabs

einar-oplabs commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT-6.1-Sol (xhigh): Reviewed 64d55f160c27dc5f16fa51a6a633eb642f626510; no actionable findings.

Checked that the sequencer and deriving verifier b stay on op-node, the follow-source verifier c still honors DEVSTACK_L2CL_KIND, and the existing configuration options and peer connections are preserved.

Validation:

  • go test -count=1 ./op-devstack/sysgo passed.
  • All three follow_l2 acceptance tests passed with op-node + op-reth and with kona-node + op-reth: safe/finalized/current-L1 convergence, reorg recovery, and operation without CL P2P.
  • All 56 PR checks were green at review time.

The initial acceptance runs rebuilt the Go harness from this PR and used existing local Rust binaries from the 23912-kona-merged workspace. These tests check the follow-source behavior; they do not directly cover the catch-up, stale-finality, or repeated-SYNCING regressions addressed by the related engine PRs.

Combined validation update (2026-11-06): locally merged the current heads of #23899 (bb36509bc5), #23900 (b628609e6f), #23908 (0c2fa31049), and #23947 (64d55f160c) into test commit 215a22332b. All four PR branches are unchanged.

The merge preserves both sets of finalization guards and their regression tests. Test fixtures account for the acknowledged forkchoice status and the new identity-mismatch error.

  • 401 Rust library tests passed across kona-engine and kona-node-service.
  • op-devstack/sysgo tests and formatting checks passed.
  • Clippy passed with --all-targets --all-features -- -D warnings; it emitted the existing workspace warning about the removed clippy::from_iter_instead_of_collect lint.
  • All three follow_l2 acceptance scenarios passed under both op-node + op-reth and kona-node + op-reth, using freshly built Rust binaries from the combined checkout: safe/finalized/current-L1 convergence, reorg recovery, and operation without CL P2P.

Nextest flagged an open output handle in one passing, unchanged attribute-format test; its isolated rerun passed without the flag. The first acceptance attempts stopped at setup because the new checkout lacked the Cannon helper fixture; the complete runs above passed after restoring that fixture.

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GPT-6.1-Sol (xhigh): Reviewed 454481f562f45c8c93022a6ed6090a226d12ce3c against 15ca278f190ed7635c51dcc6d18307e311562c1b. No actionable findings.

Checked the changed CL-starter callers, explicit client selection across presets, the op-node sequencer/deriving-verifier and requested follow-source-verifier topology, callers of the two-verifier preset, and its frontend handling when proposer/challenger services are absent. Reviewed all three retained follow-source scenarios and their setup.

Validation: go test ./op-devstack/sysgo -run 'Test(KonaOptionsRejectedBeforeStartup|SingleChainTwoVerifiersCLKinds)$' -count=1 passed. All 435 kona-engine/kona-node-service tests also passed at this stack tip. The three full follow-source acceptance devnets were inspected but not rerun during this review.

@einar-oplabs einar-oplabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fable 5.1 (xhigh): Reviewed 454481f562. Main point: an unknown DEVSTACK_L2CL_KIND still runs everything on op-node and reports success, so a typo in the kona CI variant tests nothing, while the EL starter already rejects unknown kinds. Then the latent interop-at-genesis failure under the kona variant, the fault-proofs drop in the two-verifiers runtime, and smaller duplication, ordering, comment and test nits.

Comment thread op-devstack/sysgo/singlechain_build.go Resolved

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This increases kona-node coverage and I found no regressions. One existing test is weaker on kona than it looks, though, and there's a cheap way to get more kona coverage out of this preset.

Coverage. No node that ran kona-node before this PR runs op-node after it:

  • Before this PR the two-verifiers preset failed under kona before any test logic ran, because replaceSingleChainTestSequencer requires b to be an *OpNode. So pinning the sequencer and b to op-node loses nothing.
  • The new op-node pin for interop at genesis matches what the code already did, since world.Interop is only set when cfg.InteropAtGenesis is.
  • Every other preset gets the same CL kind as before.

The kona CI job (memory-all-kona-op-reth-fusaka) runs the whole acceptance tree, and its run on 16fe67d passed all three follow_l2 tests.

Local verification at 16fe67d. Lint, build, vet and the new unit tests pass. Reverting default: to the op-node branch makes TestUnknownL2CLKindRejectedBeforeStartup fail, as it should. I ran follow_l2 3 times with DEVSTACK_L2CL_KIND=kona-node (9/9 pass) and once with op-node (3/3). The kona-node logs confirm c ran kona-node with KONA_NODE_L2_FOLLOW_SOURCE set. With a follow source set, kona builds only DelegateDerivationActor and no pipeline (rollup.rs:295), so c can't pass by deriving from L1 itself.

1. TestFollowL2_ReorgRecovery can pass on kona without c having recovered

In 2 of the 3 kona runs, the reorged L2 block (14) was c's own safe head. Kona put the new chain's P2P blocks on top of it, got Insert(ForkchoiceUpdateFailed(InvalidForkchoiceState)), and reset its safe head to l2_safe=0. b had also just reset after the L1 reorg, so the final check passed with both at genesis:

Nodes in sync at matching head node1_id=c node2_id=b label=local-safe head=428b2d..7550cc:0

The test ended about 2s later. In the op-node run, c kept safe=14 and caught up with b at block 32. Whether kona c would have caught up is unverified.

The gap is in the test: before the reorg it only waits for L2ELB.Reached(eth.Safe, ...), never for c. I'd suggest three changes:

  • Before the reorg, require L2CLC to reach l2BlockBeforeReorg as local-safe.
  • After it, wait until L2ELC's block at l2BlockBeforeReorg.Number changes and equals b's.
  • Require c's safe head to get past that height, so the in-sync check can't match at genesis.

Kona resetting to finalized here, where op-node keeps its safe head, looks like a real kona/op-node divergence in follow mode. It's probably for a separate kona fix, but this PR is what makes these tests the kona coverage for it, so the test should catch it.

2. b could take DEVSTACK_L2CL_KIND too

b is pinned to op-node only by the type assertion in replaceSingleChainTestSequencer (singlechain_variants.go:252). The test sequencer only needs b's RPC URLs, and startTestSequencerForRPCs already wires one that way for a possibly-kona primary (singlechain_runtime.go:244). Switching to it would let b follow the env too, which would exercise:

  • kona as the follow-source upstream: it serves optimism_syncStatus, the only call a follower makes upstream;
  • kona following kona;
  • op-node following kona, if the per-node kinds are made configurable.

Nothing covers any of these today. singleChainTwoVerifiersCLKinds already holds per-node kinds, so this is cheap.

3. Nothing asserts that c follows rather than derives

It's true today, but a regression to deriving wouldn't fail these tests. The preset accepts global L2CL options, so NewStallableFollowSourceProxies().L2CLOption() would work here; it's already used in the supernode follow_l2 tests. Stall c's source, assert c's local-safe and finalized heads stop while b's advance, then unstall and assert c catches up.

Smaller suggestions

  • TestFollowL2_WithoutCLP2P: also check finalized is in sync with b after reconnecting.
  • Kona's delegated actor skips two checks op-node makes on the upstream status (driver.go:470-480): that safe isn't ahead of local-safe, and that finalized isn't ahead of safe. Worth a kona unit test or issue.

Nits

  • The singleChainTwoVerifiersCLKinds struct and TestSingleChainTwoVerifiersCLKinds restate a constant. A local followKind := ResolveMixedL2CLKind() with the explanation as a comment would do.
  • newMinimalRuntime(t, cfg, resolveSequencerCLKind(cfg)) (singlechain_variants.go:35, :40) is the same as calling NewMinimalRuntimeWithConfig(t, cfg) (and the no-fault-proofs equivalent).
  • In the conductor runtime, "A non-candidate node, always stock" now sits above a call that takes clKind. Maybe "with stock op-reth".
  • t.Require().FailNow(msg, "unknown L2 CL kind %q", clKind) could be FailNowf, matching mixed_runtime.go.
  • An invalid DEVSTACK_L2CL_KIND is only caught when the first node of that kind starts. In two-verifiers that's after L1, the sequencer, b and the test sequencer are up. Validating once in ResolveMixedL2CLKind would fail fast. Fine as a follow-up.
  • op-devstack/README.md:113 says DEVSTACK_L2CL_KIND selects kona-node "as default L2 CL node". A sentence saying some presets pin nodes to op-node would help.

@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Opus 5.5 (high): Review items, by commit:

@joshklop
joshklop marked this pull request as draft November 16, 2026 20:47
Base automatically changed from einar/23675-delegated-catchup to develop November 17, 2026 18:03
@einar-oplabs
einar-oplabs dismissed joshklop’s stale review November 17, 2026 18:03

The merge-base changed after approval.

einar-oplabs and others added 3 commits November 17, 2026 10:12
…kona-node

The follow_l2 tests skipped on kona-node because the single-chain test
sequencer requires an op-node CL, and the two-verifiers preset applied
DEVSTACK_L2CL_KIND to every node. The preset now always runs the
sequencer and the deriving verifier "b" on op-node, and only the
follow-source verifier "c" takes the env-selected CL. Under kona-node
that checks kona's follow-source mode against an op-node upstream;
under op-node every node keeps its CL. The preset also no longer starts
a proposer or challenger, which no follow_l2 test uses, and accepts
only the options that runtime honors.

The single-chain CL starters take the CL kind as an argument instead of
reading DEVSTACK_L2CL_KIND, each preset resolves it once, and an
unknown kind, the empty string included, fails instead of running
op-node. Presets with interop at genesis pin their sequencer to
op-node, which they already required.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TestFollowL2_ReorgRecovery could pass with the follow-source verifier and
the deriving verifier both reset to genesis. The follow-source verifier
must now hold the reorged block as local-safe before the L1 reorg, replace
it in its EL with the deriving verifier's block, and move its local-safe
head past that height. TestFollowL2_WithoutCLP2P also checks that the
finalized head follows the deriving verifier after reconnecting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResolveMixedL2CLKind takes the test handle and fails on a value other than
op-node or kona-node, so an invalid kind fails before any node starts
instead of when the first node of that kind starts. The two-verifiers
runtime reads the follow-source verifier's kind into a local in place of
singleChainTwoVerifiersCLKinds, the multi-node runtimes call the exported
minimal-runtime constructors, the CL-kind starters use FailNowf, and the
README says that some presets pin nodes to op-node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshklop
joshklop force-pushed the einar/23912-delegated-parity-verification branch from 6bcb337 to aafe0f0 Compare November 17, 2026 18:13
@joshklop
joshklop marked this pull request as ready for review November 17, 2026 18:13
@joshklop

Copy link
Copy Markdown
Contributor

@OptimismBot re-approve

@joshklop

Copy link
Copy Markdown
Contributor

@otto-op-centaur re-approve

@joshklop

Copy link
Copy Markdown
Contributor

@op-centaur re-approve

@otto-op-centaur

Copy link
Copy Markdown

otto-op-centaur[bot] did not re-approve this pull request: the latest approval was not dismissed as stale.

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude, for @joshklop: Review of aafe0f0151. Mergeable as is, with three optional nits below. go build and go vet on the touched packages pass, and so do the op-devstack/sysgo tests for the CL kind. Every singleChainRuntimeSpec sets SequencerCLKind, and each runtime resolves the kind before L1 starts.

Comment thread op-devstack/README.md Resolved
Comment thread op-devstack/sysgo/mixed_runtime_test.go Resolved
Comment thread op-devstack/sysgo/mixed_runtime_test.go Resolved
@joshklop

Copy link
Copy Markdown
Contributor

@op-centaur re-approve

@otto-op-centaur

Copy link
Copy Markdown

otto-op-centaur[bot] did not re-approve this pull request: the latest approval was not dismissed as stale.

@joshklop

Copy link
Copy Markdown
Contributor

@op-centaur re-approve

@otto-op-centaur

Copy link
Copy Markdown

otto-op-centaur[bot] did not re-approve this pull request: the latest approval was not dismissed as stale.

@joshklop
joshklop self-requested a review November 19, 2026 01:14
@joshklop

Copy link
Copy Markdown
Contributor

@op-centaur re-approve

@otto-op-centaur

Copy link
Copy Markdown

otto-op-centaur[bot] did not re-approve this pull request: the latest approval was not dismissed as stale.

@joshklop
joshklop removed this pull request from stack #24203 November 19, 2026 02:00
@joshklop
joshklop added this pull request to stack #24271 November 19, 2026 02:00
@geoknee
geoknee added this pull request to the merge queue Nov 19, 2026
Merged via the queue into develop with commit 140f498 Nov 19, 2026
68 checks passed
@geoknee
geoknee deleted the einar/23912-delegated-parity-verification branch November 19, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants