Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .circleci/continue/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2901,14 +2901,17 @@ workflows:
# for the gitignored superchain-configs.zip op-acceptance-tests //go:embeds
- prep-superchain
# IN-MEMORY - kona/op-reth
# 2 nodes, not 4: tests on presets that start no kona-node skip here (op-devstack presets'
# skipOpNodeOnlyPresetOnKonaNode), leaving about a third of the test time. The longest
# test, TestExecuteStep_CannonKona at about 400 s, still bounds the wall time.
- op-acceptance-tests: &memory_all_kona
name: memory-all-kona-op-reth-<<matrix.l1_fork>>
image: <<pipeline.parameters.c-rust_base_image>>
matrix:
parameters:
l1_fork: *acceptance_l1_forks
l2_cl_kind: kona-node
parallelism: 4
parallelism: 2
acceptance_test_jobs: "16"
no_output_timeout: 120m
context:
Expand Down
12 changes: 9 additions & 3 deletions docs/ai/acceptance-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,15 @@ CI runs two variants of the `op-acceptance-tests` job: `memory-all-opn-op-reth-<
(op-node + op-reth) and `memory-all-kona-op-reth-<l1_fork>` (kona-node + op-reth).

A test that cannot run under kona-node skips itself in code with `sysgo.SkipOnKonaNode` rather
than being left out of a list. Skip only for a real client gap. A test that pins the CL kind of
every node itself runs the same way in both CL variants; let it run twice rather than skipping
one variant to save CI time.
than being left out of a list. Skip only for a real client gap.

The supernode presets (`NewSimpleInterop`, `NewTwoL2SupernodeInterop`, ...) and the interop
presets without a supernode (`NewSingleChainInteropNoSupernode`, ...) start only op-node and
op-supernode CLs, whatever `DEVSTACK_L2CL_KIND` says, so under kona-node they would repeat the
op-node job. Each of them calls `skipOpNodeOnlyPresetOnKonaNode` first, which skips the test
under kona-node before anything starts, so a new test on those presets needs no handling of its
own. A test that starts both CLs itself, through `sysgo.NewMixedSingleChainRuntime` with
explicit CL kinds, still runs in both jobs.
Comment thread
sebastianst marked this conversation as resolved.
Resolved

### Kona Prestate

Expand Down
2 changes: 1 addition & 1 deletion op-devstack/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ and returns a typed output that the test then may use.
- `DEVSTACK_MONOREPO_ROOT`: Absolute path to a monorepo checkout. Used by out-of-tree acceptance suites (which depend on this module by git rev) so contract-artifact resolution can find `packages/contracts-bedrock` outside the cwd-relative walk. In-tree runs don't need it.

### Rust stack env vars:
- `DEVSTACK_L2CL_KIND=kona-node` to select kona-node as default L2 CL node. Any value other than `op-node` or `kona-node` fails the test before a node starts. Some presets pin nodes to op-node whatever the variable says, such as the sequencer of a chain with interop at genesis, and the sequencer and deriving verifier of the two-verifiers preset.
- `DEVSTACK_L2CL_KIND=kona-node` to select kona-node as default L2 CL node. Any value other than `op-node` or `kona-node` fails the test before a node starts. Some presets pin nodes to op-node whatever the variable says, such as the sequencer of a chain with interop at genesis, and the sequencer and deriving verifier of the two-verifiers preset. The supernode and interop presets start only op-node and op-supernode CLs, so under kona-node they skip the test
- `RUST_BINARY_PATH_KONA_NODE=/home/USERHERE/projects/kona/target/debug/kona-node` to select the kona-node executable to run (skips the build; the path must exist)
- `RUST_BINARY_PATH_OP_RETH=/home/USERHERE/projects/reth/target/release/op-reth` to select the op-reth executable to run (skips the build; the path must exist)

Expand Down
9 changes: 9 additions & 0 deletions op-devstack/presets/interop.go
Original file line number Diff line number Diff line change
Expand Up @@ -139,33 +139,38 @@ func (s *SingleChainInterop) StandardBridge(l2Chain *dsl.L2Network) *dsl.Standar
// NewSimpleInterop creates a fresh SimpleInterop target for the current
// test using the super-root proofs system backed by op-supernode.
func NewSimpleInterop(t devtest.T, opts ...Option) *SimpleInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewSimpleInterop", opts, twoL2SupernodeProofsPresetSupportedOptionKinds)
return simpleInteropFromSupernodeProofsRuntime(t, sysgo.NewTwoL2SupernodeProofsRuntimeWithConfig(t, true, presetCfg))
}

// NewThreeChainInterop creates a three-chain supernode proofs system.
func NewThreeChainInterop(t devtest.T, opts ...Option) *ThreeChainInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewThreeChainInterop", opts, twoL2SupernodeProofsPresetSupportedOptionKinds)
return threeChainInteropFromSupernodeProofsRuntime(t, sysgo.NewThreeL2SupernodeProofsRuntimeWithConfig(t, true, presetCfg))
}

// NewSingleChainInterop creates a fresh SingleChainInterop target for the
// current test using the single-chain super-root proofs system backed by op-supernode.
func NewSingleChainInterop(t devtest.T, opts ...Option) *SingleChainInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewSingleChainInterop", opts, supernodeProofsPresetSupportedOptionKinds)
return singleChainInteropFromSupernodeProofsRuntime(t, sysgo.NewSingleChainSupernodeProofsRuntimeWithConfig(t, true, presetCfg))
}

// NewSimpleInteropIsthmusSuper creates a fresh SimpleInterop target for the current test
// using the Isthmus super-root system backed by op-supernode.
func NewSimpleInteropIsthmusSuper(t devtest.T, opts ...Option) *SimpleInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewSimpleInteropIsthmusSuper", opts, twoL2SupernodeProofsPresetSupportedOptionKinds)
return simpleInteropFromSupernodeProofsRuntime(t, sysgo.NewTwoL2SupernodeProofsRuntimeWithConfig(t, false, presetCfg))
}

// NewSingleChainInteropIsthmusSuper creates a fresh SingleChainInterop target for the
// current test using the single-chain Isthmus super-root system backed by op-supernode.
func NewSingleChainInteropIsthmusSuper(t devtest.T, opts ...Option) *SingleChainInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewSingleChainInteropIsthmusSuper", opts, supernodeProofsPresetSupportedOptionKinds)
return singleChainInteropFromSupernodeProofsRuntime(t, sysgo.NewSingleChainSupernodeProofsRuntimeWithConfig(t, false, presetCfg))
}
Expand All @@ -175,6 +180,7 @@ func NewSingleChainInteropIsthmusSuper(t devtest.T, opts ...Option) *SingleChain
// op-supernode). The op-challenger plays super-cannon-kona games against this op-node
// source. This exercises the "op-node as super root RPC" path end-to-end.
func NewSingleChainInteropNoSupernode(t devtest.T, opts ...Option) *SingleChainInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewSingleChainInteropNoSupernode", opts, singleChainInteropNoSupernodePresetSupportedOptionKinds)
return singleChainInteropNoSupernodeFromRuntime(t, sysgo.NewSingleChainInteropNoSupernodeSuperRootRuntimeWithConfig(t, presetCfg))
}
Expand All @@ -184,6 +190,7 @@ func NewSingleChainInteropNoSupernode(t devtest.T, opts ...Option) *SingleChainI
// running an op-challenger that plays ZK dispute games against that op-node source. This exercises
// the "op-node as super root RPC" path for the ZK game end-to-end.
func NewSingleChainInteropNoSupernodeZKDispute(t devtest.T, opts ...Option) *SingleChainInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewSingleChainInteropNoSupernodeZKDispute", opts, 0)
return singleChainInteropNoSupernodeFromRuntime(t, sysgo.NewSingleChainInteropNoSupernodeZKDisputeRuntimeWithConfig(t, presetCfg))
}
Expand All @@ -194,6 +201,7 @@ func NewSingleChainInteropNoSupernodeZKDispute(t devtest.T, opts ...Option) *Sin
// migration runs. This exercises the initial-deploy path for super-root
// dispute games tracked by ethereum-optimism/optimism#18729.
func NewSingleChainInteropSuperRootAtGenesis(t devtest.T, opts ...Option) *SingleChainInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewSingleChainInteropSuperRootAtGenesis", opts, supernodeProofsPresetSupportedOptionKinds)
return singleChainInteropFromSupernodeProofsRuntime(t, sysgo.NewSingleChainSuperRootAtGenesisRuntimeWithConfig(t, presetCfg))
}
Expand Down Expand Up @@ -239,6 +247,7 @@ type MinimalInteropNoSupernode struct {
// NewMinimalInteropNoSupernode creates a fresh MinimalInteropNoSupernode target for the
// current test.
func NewMinimalInteropNoSupernode(t devtest.T, opts ...Option) *MinimalInteropNoSupernode {
skipOpNodeOnlyPresetOnKonaNode(t)
_, _ = collectSupportedPresetConfig(t, "NewMinimalInteropNoSupernode", opts, 0)
return &MinimalInteropNoSupernode{
Minimal: *minimalFromRuntime(t, sysgo.NewMinimalInteropNoSupernodeRuntime(t)),
Expand Down
17 changes: 17 additions & 0 deletions op-devstack/presets/op_node_only.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package presets

import (
"github.com/ethereum-optimism/optimism/op-devstack/devtest"
"github.com/ethereum-optimism/optimism/op-devstack/sysgo"
)

// skipOpNodeOnlyPresetOnKonaNode skips the test under DEVSTACK_L2CL_KIND=kona-node. A preset calls
// it first when its runtime starts only op-node and op-supernode CLs whatever that variable says,
// as the supernode and interop presets do: under kona-node the test would repeat the op-node job.
// It is separate from sysgo.SkipOnKonaNode so that list names only real kona-node gaps. Remove
// the call from a preset once its runtime starts the CL kind the variable selects.
func skipOpNodeOnlyPresetOnKonaNode(t devtest.T) {
if sysgo.ResolveMixedL2CLKind(t) == sysgo.MixedL2CLKona {
t.Skip("skipping on kona-node: this preset starts only op-node and op-supernode CLs whatever DEVSTACK_L2CL_KIND says")
}
}
161 changes: 161 additions & 0 deletions op-devstack/presets/op_node_only_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
package presets

import (
"go/ast"
"go/parser"
"go/token"
"path/filepath"
"slices"
"strings"
"testing"

"github.com/ethereum-optimism/optimism/op-devstack/devtest"
"github.com/ethereum-optimism/optimism/op-devstack/sysgo"
"github.com/stretchr/testify/require"
)

type presetCase struct {
name string
newPreset func(devtest.T)
}

// opNodeOnlyPresets start only op-node and op-supernode CLs, so under kona-node each must skip
// before it starts anything.
var opNodeOnlyPresets = []presetCase{
{"NewSimpleInterop", func(t devtest.T) { NewSimpleInterop(t) }},
{"NewSimpleInteropIsthmusSuper", func(t devtest.T) { NewSimpleInteropIsthmusSuper(t) }},
{"NewThreeChainInterop", func(t devtest.T) { NewThreeChainInterop(t) }},
{"NewSingleChainInterop", func(t devtest.T) { NewSingleChainInterop(t) }},
{"NewSingleChainInteropIsthmusSuper", func(t devtest.T) { NewSingleChainInteropIsthmusSuper(t) }},
{"NewSingleChainInteropSuperRootAtGenesis", func(t devtest.T) { NewSingleChainInteropSuperRootAtGenesis(t) }},
{"NewSingleChainInteropNoSupernode", func(t devtest.T) { NewSingleChainInteropNoSupernode(t) }},
{"NewSingleChainInteropNoSupernodeZKDispute", func(t devtest.T) { NewSingleChainInteropNoSupernodeZKDispute(t) }},
{"NewMinimalInteropNoSupernode", func(t devtest.T) { NewMinimalInteropNoSupernode(t) }},
{"NewTwoL2Supernode", func(t devtest.T) { NewTwoL2Supernode(t) }},
{"NewTwoL2SupernodeInterop", func(t devtest.T) { NewTwoL2SupernodeInterop(t, 0) }},
{"NewTwoL2SupernodeLightSequencerInterop", func(t devtest.T) { NewTwoL2SupernodeLightSequencerInterop(t, 0) }},
{"NewTwoL2SupernodeFollowL2", func(t devtest.T) { NewTwoL2SupernodeFollowL2(t, 0) }},
{"NewSingleSupernodeWithSyncTester", func(t devtest.T) { NewSingleSupernodeWithSyncTester(t) }},
}

// konaCapablePresets start the CL kind DEVSTACK_L2CL_KIND selects, so they must still run in the
// kona-node job.
var konaCapablePresets = []presetCase{
{"NewMinimal", func(t devtest.T) { NewMinimal(t) }},
{"NewMinimalNoFaultProofs", func(t devtest.T) { NewMinimalNoFaultProofs(t) }},
{"NewMinimalWithConductors", func(t devtest.T) { NewMinimalWithConductors(t) }},
{"NewMinimalWithKona", func(t devtest.T) { NewMinimalWithKona(t) }},
{"NewSimpleWithSyncTester", func(t devtest.T) { NewSimpleWithSyncTester(t) }},
{"NewSingleChainMultiNode", func(t devtest.T) { NewSingleChainMultiNode(t) }},
{"NewSingleChainMultiNodeWithoutCheck", func(t devtest.T) { NewSingleChainMultiNodeWithoutCheck(t) }},
{"NewSingleChainMultiNodeWithoutP2PWithoutCheck", func(t devtest.T) { NewSingleChainMultiNodeWithoutP2PWithoutCheck(t) }},
{"NewSingleChainWithIsolatedVerifier", func(t devtest.T) { NewSingleChainWithIsolatedVerifier(t) }},
{"NewSingleChainMultiNodeWithTestSeq", func(t devtest.T) { NewSingleChainMultiNodeWithTestSeq(t) }},
{"NewSingleChainTwoVerifiersWithoutCheck", func(t devtest.T) { NewSingleChainTwoVerifiersWithoutCheck(t) }},
}

// notPresets take a devtest.T but start nothing themselves.
var notPresets = []string{
"NewMixedSingleChainFrontends", // wraps a runtime the caller already started
}

func TestOpNodeOnlyPresetsSkipOnKonaNode(t *testing.T) {
Comment thread
sebastianst marked this conversation as resolved.
Resolved
t.Setenv("DEVSTACK_L2CL_KIND", string(sysgo.MixedL2CLKona))
for _, p := range opNodeOnlyPresets {
require.Truef(t, runSkipped(t, p.name, p.newPreset), "%s did not skip under kona-node", p.name)
}
}

func TestKonaCapablePresetsRunOnKonaNode(t *testing.T) {
t.Setenv("DEVSTACK_L2CL_KIND", string(sysgo.MixedL2CLKona))
for _, p := range konaCapablePresets {
require.Falsef(t, runSkipped(t, p.name, p.newPreset), "%s skipped under kona-node", p.name)
}
}

// Every exported New* func whose first parameter is a devtest.T must be on exactly one of the three
// lists, so a new preset can't land without a kona-node decision. An unlisted op-node-only preset
// would repeat the op-node job under kona-node.
func TestEveryPresetIsClassified(t *testing.T) {
var listed []string
for _, p := range slices.Concat(opNodeOnlyPresets, konaCapablePresets) {
listed = append(listed, p.name)
}
listed = append(listed, notPresets...)

files, err := filepath.Glob("*.go")
require.NoError(t, err)
fset := token.NewFileSet()
var constructors []string
for _, path := range files {
if strings.HasSuffix(path, "_test.go") {
continue
}
f, err := parser.ParseFile(fset, path, nil, 0)
require.NoError(t, err)
for _, decl := range f.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Recv != nil || !fn.Name.IsExported() || !strings.HasPrefix(fn.Name.Name, "New") {
continue
}
if params := fn.Type.Params.List; len(params) > 0 && isDevtestT(params[0].Type) {
constructors = append(constructors, fn.Name.Name)
}
}
}
require.NotEmpty(t, constructors)
require.ElementsMatch(t, constructors, listed,
"an extra in list A needs adding to one of opNodeOnlyPresets, konaCapablePresets and notPresets; "+
"an extra in list B is listed twice or is not an exported New* func taking a devtest.T")
}

func isDevtestT(expr ast.Expr) bool {
sel, ok := expr.(*ast.SelectorExpr)
if !ok {
return false
}
pkg, ok := sel.X.(*ast.Ident)
return ok && pkg.Name == "devtest" && sel.Sel.Name == "T"
}

func TestSkipOpNodeOnlyPresetOnKonaNodeSkipsOnlyKonaNode(t *testing.T) {
for _, tc := range []struct {
kind sysgo.MixedL2CLKind
wantSkip bool
}{
{"", false},
{sysgo.MixedL2CLOpNode, false},
{sysgo.MixedL2CLKona, true},
} {
t.Setenv("DEVSTACK_L2CL_KIND", string(tc.kind))
require.Equalf(t, tc.wantSkip, runSkipped(t, "kind="+string(tc.kind), skipOpNodeOnlyPresetOnKonaNode),
"DEVSTACK_L2CL_KIND=%q", tc.kind)
}
}

// stopAtStartT stops a preset at its first temp dir, the JWT secret every runtime writes before
// it starts a node, so a test can see whether the preset skipped without starting a system.
type stopAtStartT struct{ devtest.T }

type presetStarted struct{}

func (stopAtStartT) TempDir() string { panic(presetStarted{}) }
func (stopAtStartT) TempDirWithPrefix(string) string { panic(presetStarted{}) }

// runSkipped runs fn in a subtest, stopped at its first temp dir, and reports whether the
// subtest skipped before reaching it.
func runSkipped(t *testing.T, name string, fn func(devtest.T)) bool {
// With this set, devtest turns a skip into an error and carries on.
t.Setenv(devtest.ExpectPreconditionsMet, "")
var sub *testing.T
t.Run(name, func(t *testing.T) {
sub = t
defer func() {
if r := recover(); r != nil && r != (presetStarted{}) {
panic(r)
}
}()
fn(stopAtStartT{devtest.SerialT(t)})
})
return sub.Skipped()
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ type SingleSupernodeWithSyncTester struct {
}

func NewSingleSupernodeWithSyncTester(t devtest.T, opts ...Option) *SingleSupernodeWithSyncTester {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, presetOpts := collectSupportedPresetConfig(
t,
"NewSingleSupernodeWithSyncTester",
Expand Down
3 changes: 3 additions & 0 deletions op-devstack/presets/twol2.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ type TwoL2 struct {
// NewTwoL2Supernode creates a fresh TwoL2 target backed by a shared supernode for the
// current test.
func NewTwoL2Supernode(t devtest.T, opts ...Option) *TwoL2 {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewTwoL2Supernode", opts, twoL2SupernodePresetSupportedOptionKinds)
return twoL2SupernodeFromRuntime(t, sysgo.NewTwoL2SupernodeRuntimeWithConfig(t, presetCfg))
}
Expand Down Expand Up @@ -167,13 +168,15 @@ func (s *TwoL2SupernodeInterop) BootstrapLightSequencersViaVNHandoff() {
// NewTwoL2SupernodeInterop creates a fresh TwoL2SupernodeInterop target for the current
// test.
func NewTwoL2SupernodeInterop(t devtest.T, delaySeconds uint64, opts ...Option) *TwoL2SupernodeInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewTwoL2SupernodeInterop", opts, twoL2SupernodeInteropPresetSupportedOptionKinds)
return twoL2SupernodeInteropFromRuntime(t, sysgo.NewTwoL2SupernodeInteropRuntimeWithConfig(t, delaySeconds, presetCfg))
}

// NewTwoL2SupernodeLightSequencerInterop creates a two-L2 interop setup where
// light op-node CLs sequence blocks and the shared supernode derives safe heads.
func NewTwoL2SupernodeLightSequencerInterop(t devtest.T, delaySeconds uint64, opts ...Option) *TwoL2SupernodeInterop {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewTwoL2SupernodeLightSequencerInterop", opts, twoL2SupernodeLightSequencerPresetSupportedOptionKinds)
return twoL2SupernodeInteropFromRuntime(t, sysgo.NewTwoL2SupernodeLightSequencerInteropRuntimeWithConfig(t, delaySeconds, presetCfg))
}
Expand Down
1 change: 1 addition & 0 deletions op-devstack/presets/twol2_follow_l2.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ type TwoL2SupernodeFollowL2 struct {
// NewTwoL2SupernodeFollowL2 creates a fresh follow-source variant of the two-L2
// supernode interop preset for the current test.
func NewTwoL2SupernodeFollowL2(t devtest.T, delaySeconds uint64, opts ...Option) *TwoL2SupernodeFollowL2 {
skipOpNodeOnlyPresetOnKonaNode(t)
presetCfg, _ := collectSupportedPresetConfig(t, "NewTwoL2SupernodeFollowL2", opts, twoL2SupernodeInteropPresetSupportedOptionKinds)
return twoL2SupernodeFollowL2FromRuntime(t, sysgo.NewTwoL2SupernodeFollowL2RuntimeWithConfig(t, delaySeconds, presetCfg))
}
5 changes: 4 additions & 1 deletion op-devstack/sysgo/singlechain_build.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,10 @@ func applyConfigPrefundedL2(t devtest.T, keys devkeys.Keys, l1ChainID, l2ChainID
l1Config.WithPrefundedAccount(addrFor(devkeys.SystemConfigOwner), *millionEth)
}

// startL2CLForKey starts an L2 CL node of the given kind for the given key.
// startL2CLForKey starts an L2 CL node of the given kind for the given key. The supernode and
// interop runtimes don't use it, so their presets skip under kona-node (presets'
// skipOpNodeOnlyPresetOnKonaNode); a runtime that passes it the kind DEVSTACK_L2CL_KIND selects
// must not keep that skip in its presets, or the kona-node job skips its tests.
func startL2CLForKey(
t devtest.T,
keys devkeys.Keys,
Expand Down
Loading