Repository navigation
Description
Goal
kona-node records its safe head history in kona-safedb and serves it over RPC, so that op-challenger, op-proposer and op-dispute-mon can run against kona-node verifiers without an op-node in the loop.
Current status
The kona-safedb crate landed in #21522. Safe-head recording, reset/EL-sync truncation and clearing, and fail-fast persistence handling merged in #23583 on 2026-10-23, completing #23474. Core integration merged in #23590 on 2026-10-27: the SafeDB flag, optimism_safeHeadAtL1Block, pre-interop superroot_atTimestamp, and direct op-node/Kona parity coverage.
Super-root withdrawal coverage and SafeDB configuration through normal Kona startup merged in #23704 on 2026-10-28. The direct mixed-node CL-resync and wrong-key coverage merged in #23818 on 2026-10-31, the failed-reset fix in #23815 on 2026-11-04, and the acceptance/proof-consumer tests and public RPC documentation in #23787 (5b3854d301) on 2026-11-12. Since #23827, the two SafeDB resync suites run on op-reth in the normal acceptance jobs, with no op-geth dependency. Every sub-issue is closed. On develop (CircleCI pipeline 139914, 7f7abd37fd), safedb_parity, both safeheaddb resync suites, safedb_proof_consumers, cannon_kona and dispute-mon pass with both kona-node and op-node.
Remaining before this issue closes: fix #24130 (high) and #24131 (medium), then a passing safedb-reviewer re-run on the final integration (first run failed). The ethereum-optimism/protocol-team#283 proposer/challenger rollout is unblocked and tracked there.
Soak progress — 2026-10-29
The Sepolia-backed kona-safedb-0 network was created by devnets-private#2227. Its initial rollout failed because the non-root Kona process could not write /admin/safedb. devnets-private#2296 set fsGroup: 10001; the original 1 GiB PVC was retained. Netchef then completed successfully and devnets-private#2301 marked the network online. Kona is pinned to the merged #23590 build cfec650e57ecedf04ad878891ecc79eaca414365.
- Completed window: 2026-10-28 19:50:49 UTC through 2026-10-29 20:05:25 UTC, approximately 24h15m.
- 24-hour soak passed on 2026-10-29: 7,454/7,454 SafeDB queries matched op-node: 7,422 consecutive queries across L1 12,014,303–12,021,724 plus 32 older queries, with zero RPC errors/mismatches and all 5,758 prior answers unchanged. The upper bound was safe on both nodes at 19:51:48 UTC, beyond the 19:50:49 UTC soak endpoint. Kona retained its original PVC and had zero container restarts after the controlled pod recreation. Final evidence.
- The controlled Kona-only pod recreation retained the EL and original PVC. All 160 restart baselines matched after rederivation; the initial temporary rewind remains documented.
- Final validation confirmed Ready, unchanged pod/image/PVC/bound volume, zero container restarts after the controlled recreation,
fsGroup: 10001, and/admin/safedb. - Full-period log/restart review is complete: 122 Kona warnings and 10 peer-manager errors were confined to restart/catch-up; later entries were 1,111 op-node possible-L1-reorg warnings and 25 batcher warnings, with subsequent confirmed publication. All 1,268 matching lines were retained below the query cap. Detailed final report.
- Checks were manual. The persistence soak is complete; kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23479 is closed. The remaining acceptance and consumer requirements stay open here and in kona-node,safedb,op-acceptance-tests: validate restart persistence #22265/kona-node,safedb,op-acceptance-tests: validate Cannon execution-step proofs #22279.
Remaining scope and deferred work
All six mixed-node parity cases, the wrong-key negative control, the failed-reset fix and the consumer/documentation coverage are merged (#23818, #23815, #23787). The 24-hour persistence soak is complete.
Long-term resume-from-tip recovery (#23658), interval-based gap detection (#23659), node-side metrics and gap-risk warnings (#23478), the history-dump RPC (#23584), dashboard panels/alerts (#23481), and the write-once-per-L1-block optimization (#23480, inherited from #22528 and #23112) are tracked under ethereum-optimism/protocol-team#442 and deferred until after the EoY milestone. They are not completion gates for this issue.
op-challenger clamps its honest output root using optimism_safeHeadAtL1Block; the super-root proposer/challenger paths read superroot_atTimestamp; op-dispute-mon uses safe-head history to judge L1 backing. These consumers require the SafeDB flag and recorded history. Acceptance/proofs-stack validation and the soak remain the rollout dependency for adding a proposer and challenger to ethereum-optimism/protocol-team#283.
Supersedes #22528 and the placeholder #20278.
Components
Each is a sub-issue of this ticket:
- kona-node: record safe heads into kona-safedb #23474 — Record safe heads into
kona-safedb(merged in kona-node: record the safe head history into kona-safedb #23583). - kona-node,safedb: serve optimism_safeHeadAtL1Block #23475 — Serve
optimism_safeHeadAtL1Block(merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590). - kona-node,safedb: serve superroot_atTimestamp before interop #23476 — Serve
superroot_atTimestamppre-interop, withRequiredL1from the database (RPC merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590; withdrawal coverage merged in op-acceptance-tests,op-devstack: enable super-root withdrawals on kona-node #23704). - kona-node,safedb: add configuration, disabled-state errors, and documentation #23477 — Config flag and disabled-state error shape (merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590); public RPC-reference documentation remains unchecked below.
- kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23479 — Closed after the successful 24-hour soak; remaining acceptance work is retained in kona-node,safedb,op-acceptance-tests: validate restart persistence #22265, kona-node,safedb,op-acceptance-tests: validate Cannon execution-step proofs #22279, and this parent.
- kona-node,safedb,op-acceptance-tests: verify parity with op-node #23585 — Direct op-node/Kona SafeDB parity coverage (merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590); all six implemented cases now pass in op-acceptance-tests,safedb: validate parity after Kona resync #23818, whose follow-up merge remains pending.
Definition of done
Core implementation checkmarks are backed by merged #23583, #23590, #23704, #23815, #23818 and #23787. Live restart/deployment checks are scoped to the soak evidence. All six mixed-node parity cases are merged (#23818) and pass on develop, and the deliberate wrong-key red/green control is recorded there. The safedb-reviewer pass on the final integration remains open. The merged persistence policy is fail-fast, without write retries. Observability follow-up (#23478) is tracked under ethereum-optimism/protocol-team#442 and is not required to complete this issue.
Behavior:
- Confirmed locally derived safe heads are persisted with the derivation source L1 block as the key; adopted heads are not recorded (kona-node: record the safe head history into kona-safedb #23583).
- A kona-node started with the safedb flag answers
optimism_safeHeadAtL1Blockfor every L1 block from the first recorded entry to the current L1 head, and the answer matches op-node on the same devstack network for the same L1 block (op-devstack/dsl/safedb.gocheckSafeHeadConsistent). - All six mixed-node parity cases pass on op-acceptance-tests,safedb: validate parity after Kona resync #23818 commit
53f6a13500in native CI job 5812786: steady state, L1 reorg, CL restart, CL+EL restart, wiped-EL resync, and disabled SafeDB. The separate canonical-tip EL-sync recovery policy remains deferred to kona-node,safedb,gap-detection: resume derivation from the canonical SafeDB tip after EL sync #23658. -
superroot_atTimestampon kona-node returnsRequiredL1andVerifiedRequiredL1consistent with op-node for the same timestamp, and the response obeys the contract inop-service/eth/superroot_at_timestamp.go. - Without the flag,
optimism_safeHeadAtL1Blockreturns the op-node-compatible disabled-state error (-32000, containingsafe head database not enabled); a missing record returnsnot found, which the Go rollup client maps tosafedb.ErrNotFound(kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590). - Engine resets and EL-sync completion truncate history to the resume point; resuming above the persisted tip clears history to avoid a silent gap (kona-node: record the safe head history into kona-safedb #23583).
- End-to-end reset and L1-reorg validation confirms that history is re-recorded and no query answers with a later L1 block than the truth. The six invariants in
docs/ai/safedb-review.mdhold and thesafedb-revieweragent passes on the final integration diff. The end-to-end part is covered by the mergedsafedb_parityL1-reorg, restart and resync cases, which pass ondevelopfor both clients. Thesafedb-reviewerpass on7f7abd37fdfailed: F1 (kona-engine: consolidate safe attributes only on top of the local-safe head #24130) can leave an unrecorded range, and F2 (kona-engine: keep SafeDB history after an outage longer than the sequencing window #24131) clears history after a long outage. - A controlled CL restart with the EL and PVC retained preserves historical RPC answers after rederivation: 160 pre-restart answers matched op-node again in the live soak (kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23479). The kona-node,safedb,op-acceptance-tests: validate restart persistence #22265 skip removal merged in kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787.
- A database write failure stops the actor immediately rather than leaving a silent gap, and finalization is queued only after successful recording (kona-node: record the safe head history into kona-safedb #23583).
Tests:
- Both SafeDB restart/resync suites run against Kona without the obsolete Kona skips (kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787) and, since op-devstack,op-acceptance-tests: run EL-resync tests on op-reth #23827, on op-reth in the normal acceptance jobs; all three cases pass on
developfor both consensus clients. -
TestExecuteStep_CannonKonaruns with Kona and op-node in native CI and requires an execution step plus defender victory at game resolution (kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787, merged). - Super-root proposer/challenger withdrawal acceptance runs against Kona in CI, including finalization (op-acceptance-tests,op-devstack: enable super-root withdrawals on kona-node #23704).
- op-dispute-mon runs against kona-node in merged devstack acceptance coverage (kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787): it agrees with a root that is safe at the game's L1 head (checked against the SafeDB record) and rejects one that was not yet safe, for both clients. The invalid-root case already exists in
interop/dispute-monand was not duplicated. - Unit tests cover the persistence/write path, reset and EL-sync truncation/clearing, and fail-fast error handling (kona-node: record the safe head history into kona-safedb #23583; 203 SafeDB and service tests reported passing).
- Unit tests cover both RPCs, including the disabled state, as part of the final integration (kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590).
Operations:
-
Deploy a persistent SafeDB soak network and verify successful startup (devnets-private#2227, devnets-private#2296, devnets-private#2301).
-
Final contiguous SafeDB parity scan through shared safe L1 block 12,021,724 covers the 24-hour endpoint: 7,454 queries match op-node, including 32 older baseline queries; all 5,758 previous answers are unchanged (kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23479).
-
Review errors/warnings and restart history across the full soak; final pod/PVC identity is unchanged with zero container restarts after the controlled recreation. Findings are recorded above.
-
A kona-node on a devnet runs for 24 hours with the flag on and a contiguity check (safedb-scan or an in-tree equivalent) reports no gap.
-
The kona-node flag is documented and the
MethodNotFoundstub/comment is gone (kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590). -
The two RPCs are documented in the public RPC reference (kona-node,safedb: add configuration, disabled-state errors, and documentation #23477, kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787), live at https://docs.optimism.io/node-operators/kona-node/rpc/rollup.
-
ethereum-optimism/protocol-team#283 can add a proposer and challenger pointed at kona verifiers: the code prerequisites are merged, and the rollout is handed off to and tracked in publish new provider #283 (handoff).
Current completion checklist — 2026-11-13
- Existing five-case parity suite passes in native CI, including steady state, L1 reorg, CL restart, CL+EL restart with EL retained, and disabled SafeDB (job 5811528).
- Proposer and dispute-monitor acceptance cases pass with both consensus clients in kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787, including valid, invalid, and historically unsafe roots.
- Public RPC reference and crate README changes are authored/reviewed in kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787; publication awaits merge.
- kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787 merged as
5b3854d301(GitHub Verified) after joshklop approved. - kona-node: stop derivation when pipeline signals fail #23815's temporary/critical actor and service-supervisor/RPC shutdown regressions fail without error propagation and pass with it; all 252 service tests, targeted clippy, and pinned-nightly formatting pass locally. Static Rust/SafeDB reviews have no findings.
- Add and locally compile/lint/review the mixed-node resync test in ready-for-review op-acceptance-tests,safedb: validate parity after Kona resync #23818.
- All 121 checks pass for kona-node: stop derivation when pipeline signals fail #23815's added service/RPC shutdown tests (
55202491e4); no failures or pending checks. - op-acceptance-tests,safedb: validate parity after Kona resync #23818's new mixed-node resync test passes in native CI job 5812896; the wrong-key mutation produces the expected L1-block-5 mismatch and both positive tests pass after exact writer restoration.
- op-acceptance-tests,safedb: validate parity after Kona resync #23818 normal PR CI is complete: 52 checks passed and 3 skipped, with zero failures/pending (
53f6a13500). - Obtain human approvals and merge kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787, kona-node: stop derivation when pipeline signals fail #23815, and op-acceptance-tests,safedb: validate parity after Kona resync #23818.
- Complete the combined merged-integration gate: fix kona-engine: consolidate safe attributes only on top of the local-safe head #24130 and kona-engine: keep SafeDB history after an outage longer than the sequencing window #24131, then get a passing
safedb-reviewerre-run. The first run failed. The end-to-end reset/reorg evidence is the mergedsafedb_paritysuite, green ondevelop. - Confirm ethereum-optimism/protocol-team#283 can roll out proposer/challenger consumers against Kona verifiers: it is unblocked, and the rollout is tracked there (handoff).
#22265, #22279 and #23479 are closed. Canonical-tip recovery (#23658), interval gap representation (#23659), metrics (#23478), dashboards (#23481), dump RPC (#23584), and write buffering (#23480) remain deferred under ethereum-optimism/protocol-team#442.
Runtime evidence: #23818 (comment)
Activity
5 remaining items
[-]kona-node: Kona-SafeDB integration[/-][+]kona-node,safedb: integrate persistence, RPCs, and proof consumers[/+]einar-oplabs commentedon Oct 29, 2026
The 24-hour SafeDB persistence soak passed on
kona-safedb-0.cfec650e57ecedf04ad878891ecc79eaca414365, with one op-node sequencer and op-node/Kona reference verifiers on op-reth.12,021,724was safe on both nodes, with hash0xfb8d20ae3a16d5fdef2510e178252807ae8d5d0ec8eb199eb030a31af9eeb4c8and timestamp 2026-10-29 19:51:48 UTC, beyond the 24-hour endpoint at 19:50:49 UTC. Both nodes had processed beyond it. This is an RPC-level contiguous comparison, not a new storage-interval/gap-detection implementation.fsGroup: 10001and/admin/safedbwere verified.Final Slack checkpoint. Complete JSON responses, pod/PVC snapshots, log review, and the final report are retained locally under
kona-safedb-soak-evidence/2026-10-28/2026-10-29T200437Z-final. All checks were manual; no automatic hourly schedule was enabled.Closing #23479 as requested; #23473 remains open for the remaining integration requirements. This closure does not claim that the outstanding acceptance changes are merged or passing CI. Remaining SafeDB EL/CL-resync acceptance work stays open in #22265; Cannon execution-step and output-root game-resolution validation stays open in #22279 and #23473; merged dispute-monitor acceptance, the final reset/reorg parity-suite rerun, and public RPC documentation stay tracked in #23473. Super-root withdrawal consumer CI is already merged in #23704. Long-term recovery, interval detection, metrics, dashboards, dump RPC, and write buffering remain open under ethereum-optimism/protocol-team#442.
einar-oplabs commentedon Nov 13, 2026
Opus 5.5 (high):
safedb-reviewerpass on the merged kona-node SafeDB integration,developat7f7abd37fd. It followed.claude/agents/safedb-reviewer.mdanddocs/ai/safedb-review.md, read-only, with op-node as the reference. Gate: FAIL, because of F1.Invariants
local_safe_head(), and no engine path sets safe without local-safe.derived_fromfor a confirmed matching block. The delegate actor gets no database, and the CLI refuses--safedb.pathtogether with delegation.safedb.go, and the error strings matchapi.go.Findings
superroot_atTimestamp, and a safe head that is too low fromoptimism_safeHeadAtL1Block. → kona-engine: consolidate safe attributes only on top of the local-safe head #24130derived_from, so settled history is identical. The difference shows only in answers for the L1 block still being processed (the I5 window).docs/ai/safedb-review.md:30still says kona's SafeDB is not wired, andsafedb/src/traits.rs:28-31states a resume rule kona no longer follows. To be fixed with F1 in kona-engine: consolidate safe attributes only on top of the local-safe head #24130.RollupNodeBuilderaccepts asafe_dbtogether with delegation and would serve a database that nothing writes to. Only the CLI guard (bin/node/src/commands/node/command.rs:333-339) prevents that, so the shipped binary cannot reach it.Test gaps: F1 needs a test that delivers attributes derived before a reset after it, with history asserted below the tip. F2 needs a test with an outage longer than the sequencing window, then a restart, checking that history is kept.
The gate item above stays unchecked until #24130 is fixed and a re-run passes.