Skip to content

kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers - #23787

Merged
einar-oplabs merged 7 commits into
developfrom
einar/safedb-remaining-validation
Nov 12, 2026
Merged

einar-oplabs merged 7 commits into
developfrom
einar/safedb-remaining-validation

Conversation

@einar-oplabs

@einar-oplabs einar-oplabs commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Why

kona-node's SafeDB RPCs need acceptance coverage on both consensus clients, and these cases used to skip kona-node.

What

  • Runs SafeDB persistence across restart and resync, Cannon execution-step proofs, and proposer and dispute-monitor consumers of SafeDB answers against both kona-node and op-node, on op-reth.
  • Adds the DSL waits and SafeDB lookups these tests need: WaitForSuperGameFrom, AwaitSuperRootMatching, SafeHeadRecordAtL1Block and L1ELNode.BlockRefByHash.
  • Documents the historical availability and error behavior of kona-node's SafeDB RPCs.

Closes #22265. Closes #22279. Advances #23473. CI on 68cf64ea2d passed (54 passed, 6 skipped). Re-squash into one signed commit before merge.

Breakdown

Area Added Removed Net changes
Implementation 93 12 +81
Tests 120 19 +101
Docs 215 97 +118

🤖 Generated with Claude Code

@einar-oplabs einar-oplabs self-assigned this Oct 29, 2026
@mintlify

mintlify Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
Optimism-Docs 🟢 Ready View Preview Nov 12, 2026, 4:10 PM

@einar-oplabs

Copy link
Copy Markdown
Contributor Author

GPT-6 (effort unspecified): Decision: keep the SafeDB resync and normal acceptance steps sequential in #23787.

The corrected concurrency experiment completed in CircleCI pipeline #138260, using commit 265daedd8b.

Execution Kona elapsed op-node elapsed Kona peak anonymous RAM op-node peak anonymous RAM
Sequential 12m 44s 13m 54s 4.65 GiB 4.29 GiB
Parallel, 8 normal + 1 resync worker 7m 43s 9m 21s 5.39 GiB 4.54 GiB
Parallel, 7 normal + 1 resync worker 8m 46s 9m 30s 4.74 GiB 4.21 GiB

All six jobs passed: 201 top-level test executions, zero failures, zero skips. Every resync phase passed all three required tests with op-geth; the normal phase used op-reth and matched its complete frozen selection. Kona retained its 32 GiB runner and op-node its 40 GiB runner.

There were zero OOM, OOM-kill, memory-high, or memory-max events. Kona's 8+1 run recorded 189 sock_throttled events. Aggregate cgroup peaks ranged from 25.76–39.99 GiB and include non-anonymous memory already present before the trials; they should not be interpreted as application working-set demand. Anonymous peaks are shown separately above.

Parallel execution saved approximately 4–5 minutes on shard zero, but other shards already determine overall CI completion. Successful reference runs had Kona shard zero at 11m 21s versus a slowest other shard of 16m 21s, and op-node at 11m 45s versus 21m 08s. These shard-zero savings therefore do not establish a full-pipeline speedup.

This was one cold-build-cache trial per configuration, including compilation, with identical source and a fixed client-specific test selection. Warm-cache savings and general stability are not established. Memory was sampled every second; brief peaks may be missed. The first experiment (#138258) is excluded because a child-shell environment reset caused two required resync tests to skip; the corrected harness prevents that reset and rejects any resync skips.

We will retain sequential execution for now. Per-job summary JSON, memory CSV, and test logs are available in the corrected pipeline's artifacts.

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments, mostly on the new CircleCI op-geth SafeDB step. I checked the new docs and the kona README against the kona RPC code: field names, error codes and messages, the --safedb.path flag, and how the genesis case is handled all match.

One more thing outside the diff: in safeheaddb_clsync/safeheaddb_test.go, the comment at L47 in the function this PR touches still says the restarted CL "will use EL sync to resync the chain". The test configures sync.CLSync for both nodes and asserts the DB is preserved, so the comment looks copied from the ELSync test.

Comment thread .circleci/continue/main.yml Resolved
Comment thread .circleci/continue/main.yml Resolved
Comment thread .circleci/continue/main.yml Resolved
Comment thread .circleci/continue/main.yml Resolved
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Fable 5.1 (effort unspecified): The CL-sync comment in safeheaddb_clsync/safeheaddb_test.go now describes CL sync re-deriving from L1 and keeping the database, instead of the EL-sync text (301c29c).

@einar-oplabs

einar-oplabs commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Opus 5.5 (high): #23827 affects the op-geth resync step discussed in #23787 (comment):

  1. It removes the reason for the step. The separate op-geth invocation exists because TestTruncateDatabaseOnELResync and TestPreserveDatabaseOnCLResync SkipOnOpReth, and every acceptance job runs op-reth. op-devstack,op-acceptance-tests: run EL-resync tests on op-reth #23827 adds an explicit Wipe() to the EL, removes both skips, and runs the tests in the normal op-reth jobs. Once it lands, an existing acceptance job is enough.
  2. Until op-devstack,op-acceptance-tests: run EL-resync tests on op-reth #23827 lands, the step is the only coverage for these tests. This PR pins both tests to sysgo.WithJovianAtGenesis, so the op-geth Karst skip in op-devstack/sysgo/deployer.go does not apply, and the step fails on any skip. The two tests do run and must pass under op-geth, so the step stays for now.
  3. The two PRs overlap. Both edit safeheaddb_clsync/safeheaddb_test.go and safeheaddb_elsync/safeheaddb_test.go. This PR already adds WaitForOnline() before PeerWith in the CL-sync test and corrects its "EL sync" comment, and op-devstack,op-acceptance-tests: run EL-resync tests on op-reth #23827 touches the same lines, so whichever lands second needs a rebase.

Proposed order: land #23827, then remove the op-geth step from this PR (or a follow-up), together with the WithJovianAtGenesis pin so the tests run on the default chain. Close #23824 as superseded at that point.

Edited: an earlier version said the op-geth step would pass with every test skipped. That was wrong; the Jovian pin and the skip guard in this PR prevent it.

…consumers

Run the SafeDB acceptance cases that skipped kona-node against both
consensus clients: SafeDB history across restart and resync, Cannon
execution-step proofs through game resolution with time travel, and
proposer and dispute-monitor consumers of super and output roots.

The two SafeDB resync cases skip on op-reth, so a shard-0 CI step runs
them on op-geth, pinned to Jovian at genesis, after the sharded suite.
The step fails when either case skips or does not pass.

Document the kona-node SafeDB RPCs' historical availability and the
SafeDB RPC error contract.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
einar-oplabs and others added 3 commits November 11, 2026 16:55
… checks

The op-geth resync step accepts a test's "--- PASS"/"--- SKIP" output line
when test2json drops the result event, and sets no_output_timeout above
go's -timeout so a hang ends in a goroutine dump.

The proposer test checks the super root's output root against the node's
output at that block, and that verified_required_l1 is the first SafeDB
record with that block safe. The dispute-monitor agreement test checks the
SafeDB record at the game's L1 head directly, since dispute-mon also agrees
when SafeDB is unreadable. The invalid-root case, which never reads
SafeDB, is removed. The DSL gains SuperRootQuerier.AwaitSuperRootAtTimestamp
and L1ELNode.BlockRefByHash for these checks.

Only the ELSync resync test pins Jovian at genesis, so the restart test
on the same preset runs on the default fork. The SafeDB RPC error contract
is documented only in the kona-node RPC docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…enesis

The first super-root game can commit to the L2 genesis block, whose
required L1 is fixed at 0 without a SafeDB read. The proposer test waits
for the first game at or after the second block's timestamp, using the
new DisputeGameFactory.WaitForGameFrom.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ot checks

DisputeGameFactory gains WaitForSuperGameFrom, which retries transient
reads and checks the super game type; it and WaitForGame share one
polling loop. SuperRootQuerier.AwaitSuperRootAtTimestamp reports the
last query error on timeout, and AwaitSuperRootMatching asserts the root
claim on the awaited response. L2CLNode.SafeHeadRecordAtL1Block returns
nil when the SafeDB has no record, so tests need no op-node import.

TestTruncateDatabaseOnELResync asserts that the SafeDB floor moves past
its pre-resync value. The CL-resync test's comment states what it can
observe, and its package is named safeheaddb_clsync.

The op-geth EL step runs both packages in parallel with a 20m go timeout
and names the CL kind in its failure messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sebastianst

Copy link
Copy Markdown
Member

🤖 Fable says: A note from the op-geth decoupling work (#20257). This PR adds a CI step that runs the SafeDB resync tests with DEVSTACK_L2EL_KIND=op-geth, plus WithJovianAtGenesis pins for it.

op-geth is EOL as of Karst, and #21744 removes the in-process op-geth L2 EL from op-devstack/sysgo (l2_el_opgeth.go, MixedL2ELOpGeth, DEVSTACK_L2EL_KIND). It cannot exist after the go.mod flip to upstream go-ethereum (#20266): the same code would silently run upstream geth as the L2 EL. So please don't add new test or CI dependencies on op-geth; they would have to be removed again before the flip.

#23827 makes these two tests run on op-reth through an explicit L2ELNode.Wipe(). Proposed order: #23827 first, then this PR without the op-geth step and the Jovian pins.

…nc test

Wiping the EL makes the CL reset to genesis, which leaves a genesis record,
so the SafeDB floor drops to 0 instead of moving past its pre-resync value
on both op-node and kona-node. The floor does not show whether EL sync
truncated the database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

So please don't add new test or CI dependencies on op-geth; they would have to be removed again before the flip.

Thanks, good call out.

einar-oplabs and others added 2 commits November 12, 2026 17:09
…ning-validation

Resolve the SafeDB resync tests to develop's versions from #23827 (explicit
EL wipe, no op-reth skips, no Jovian pins), keeping this branch's
safeheaddb_clsync package name and client-neutral restart comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestTruncateDatabaseOnELResync and TestPreserveDatabaseOnCLResync run in
the normal op-reth acceptance jobs since #23827, and op-geth is being
removed from op-devstack (#20257, #21744).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einar-oplabs

Copy link
Copy Markdown
Contributor Author

Proposed order: #23827 first, then this PR without the op-geth step and the Jovian pins.

Opus 5.5 (high): Done, in that order. After #23827 merged, develop was merged into this branch (ba8ad6e407), and 68cf64ea2d removes the op-geth CI step. The two resync tests now take #23827's versions: explicit Wipe(), no op-reth skip, no Jovian pin. They run in the normal op-reth acceptance jobs for both consensus clients. #23824, which would have moved the op-geth step into dedicated jobs, is closed. This PR no longer adds any dependency on op-geth.

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. All earlier comments are addressed. The op-geth resync step is now moot because #23827 runs the EL-resync tests on op-reth. I re-checked the kona SafeDB and superroot error strings in the docs against the code, and they match.

@einar-oplabs
einar-oplabs added this pull request to the merge queue Nov 12, 2026
Merged via the queue into develop with commit 5b3854d Nov 12, 2026
69 checks passed
@einar-oplabs
einar-oplabs deleted the einar/safedb-remaining-validation branch November 12, 2026 23:28

This branch was successfully deployed

1 active deployment
staging - docs/public-docs — 68cf64ea Deployed Nov 12, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kona-node,safedb,op-acceptance-tests: validate Cannon execution-step proofs kona-node,safedb,op-acceptance-tests: validate restart persistence

3 participants