Skip to content

op-devstack,op-acceptance-tests: skip presets that start no kona-node in the kona acceptance job - #24245

Merged
nonsense merged 4 commits into
developfrom
ae/acceptance-kona-skip-opnode-only-presets
Nov 20, 2026
Merged

nonsense merged 4 commits into
developfrom
ae/acceptance-kona-skip-opnode-only-presets

Conversation

@nonsense

@nonsense nonsense commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

The kona-node acceptance job (memory-all-kona-op-reth-fusaka) repeats most of the op-node job unchanged. DEVSTACK_L2CL_KIND=kona-node selects the CL only in the single-chain presets. The supernode runtimes and the interop runtimes without a supernode start op-node and op-supernode CLs whatever the variable says. On develop that is 158 of 289 top-level tests and about 10,200 of 15,400 test-seconds in the kona job. Their kona-job timings match the op-node job's (10,243 s vs 10,316 s). Each memory-all job costs about 3,000 credits per run, and there are about 2,560 runs a month.

How the duplication started. Until #19787, both memory-all jobs ran only the base gate. That was a list of single-chain packages (base, ecotone, fjord, isthmus, jovian) with no interop, supernode or proofs tests. #19787 removed gates so that every implementation variant runs all acceptance tests. The interop and supernode runtimes never read DEVSTACK_L2CL_KIND, so that change also made the kona job re-run every one of their tests on op-node. Each interop and supernode test added since has grown the duplication. This PR reverses that part of #19787 and keeps the rest: the kona job still runs every test that starts a kona-node.

With this PR, the 14 presets built on those runtimes (NewSimpleInterop, NewTwoL2SupernodeInterop, NewSingleChainInteropNoSupernode and the others) skip the test under kona-node before they start anything. The skip is the first line of each of those preset constructors (skipOpNodeOnlyPresetOnKonaNode), so the preset shows which tests don't run in the kona job. The kona job now runs on 2 nodes instead of 4. The remaining work, about 5,100 test-seconds, is bounded by the longest kona test (TestExecuteStep_CannonKona, about 400 s), so 2 nodes running 16 tests at once should take no longer than 4 did. The estimated saving is 4–5M credits a month. Skipping without the node cut would save about 1M. The op-node job is unchanged, and so is coverage: every skipped run starts the same processes as its op-node twin.

Policy change for the acceptance-test owners to sign off. docs/ai/acceptance-tests.md said that a test pinning every node's CL kind should run in both jobs rather than skip one to save CI time. That rule came from #23856, which removed SkipOnKonaNode from the 7 tests that start both CLs and accepted running them twice. It stays for the tests that start both CLs explicitly (safedb_parity, derivation/confirmations), which still run in both jobs. Whether they should also skip one job is left to the owners. The new rule covers presets whose runtime starts no kona-node: they skip themselves under kona-node in the preset constructors, so new interop and supernode tests need no per-test handling, and SkipOnKonaNode keeps listing only real kona-node gaps. When kona-node or lokahi can run these topologies, remove the skipOpNodeOnlyPresetOnKonaNode calls from those presets. No code outside op-devstack/presets calls these runtimes directly.

Not skipped, deliberately: newDefaultSingleChainWorld with PresetConfig.InteropAtGenesis also gives the sequencer an op-node. No preset accepts that option today, and the multi-node variants built on the same runtime add verifiers that follow DEVSTACK_L2CL_KIND. A skip there could drop kona coverage.

Side effects:

  • op-up --interop with DEVSTACK_L2CL_KIND=kona-node now stops with the skip message, because op-up treats a skip as an error. Before, it silently started op-node CLs.
  • Out-of-tree suites that run these tests with DEVSTACK_L2CL_KIND=kona-node skip them too.
  • This PR's own kona run still splits tests by develop's timings, which count the newly skipped tests at full length. The two nodes can be unevenly loaded until the change is on develop.

op-devstack/presets/op_node_only_test.go lists both sides: the 14 presets must skip under kona-node, and the 11 presets that start the selected CL must not. A probe stops each preset at its first temp dir, so neither check starts a system. Removing the call from one preset, or adding it to NewMinimal, fails the test.

Review findings that were not applied:

  • ci-config-reviewer asked for the per-node wall time and memory to be checked on this PR's run. That check is done after CI runs.

🤖 Generated with Claude Code

@nonsense nonsense changed the title op-devstack, op-acceptance-tests: skip presets that start no kona-node in the kona acceptance job op-devstack,op-acceptance-tests: skip presets that start no kona-node in the kona acceptance job Nov 18, 2026
@nonsense
nonsense marked this pull request as ready for review November 18, 2026 12:32
@nonsense
nonsense requested a review from a team as a code owner November 18, 2026 12:32
Comment thread docs/ai/acceptance-tests.md Resolved
@sebastianst sebastianst self-assigned this Nov 19, 2026
nonsense and others added 4 commits November 19, 2026 10:59
…e in the kona acceptance job

The supernode runtimes and the interop runtimes without a supernode start
only op-node and op-supernode CLs, whatever DEVSTACK_L2CL_KIND says. Under
kona-node their tests repeated the op-node acceptance job exactly: 158 of
289 top-level tests and about 10,200 of 15,400 test-seconds of the kona job
on develop. Their sysgo constructors now skip the test under kona-node
before starting anything. Tests that start both CLs explicitly still run in
both jobs, and SkipOnKonaNode stays the list of real kona-node gaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the presets that start no kona-node skipped under kona-node, the kona
job has about 5,100 test-seconds left, a third of before. Its wall time is
bounded by its longest test, about 400 s, so 2 nodes running 16 tests at
once finish no later than 4 did. The scheduled shipped-binaries copy of the
job inherits the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the kona-node skip from six sysgo runtime constructors to the first
line of the 14 preset constructors whose runtimes start only op-node and
op-supernode CLs (NewSimpleInterop, NewTwoL2SupernodeInterop,
NewSingleChainInteropNoSupernode and the rest). A reader of a preset now
sees that its tests don't run in the kona-node job. Nothing outside
op-devstack/presets calls those runtimes, so the same tests skip.

The test lists both sides: the 14 presets must skip under kona-node, and
the 11 presets that start the selected CL must not. A probe T stops each
preset at its first temp dir, so neither case starts a system.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two preset lists were written by hand, so a new preset on neither
list passed without notice: an op-node-only preset that forgot the skip
would repeat the op-node job under kona-node. TestEveryPresetIsClassified
parses the package and requires every exported New* func whose first
parameter is a devtest.T to be on exactly one of the op-node-only,
kona-capable or not-a-preset lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nonsense
nonsense force-pushed the ae/acceptance-kona-skip-opnode-only-presets branch from 0db7942 to 372b117 Compare November 19, 2026 10:23
@nonsense
nonsense requested a review from sebastianst November 19, 2026 17:48
@nonsense
nonsense added this pull request to the merge queue Nov 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 20, 2026
@nonsense
nonsense added this pull request to the merge queue Nov 20, 2026
Merged via the queue into develop with commit 5a4b5ff Nov 20, 2026
114 checks passed
@nonsense
nonsense deleted the ae/acceptance-kona-skip-opnode-only-presets branch November 20, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants