Skip to content

kona-node,safedb: integrate persistence, RPCs, and proof consumers #23473

Open
9 of 11 issues completed
Open

Description

@einar-oplabs

Goal

kona-node records its safe head history in kona-safedb and serves it over RPC, so that op-challenger, op-proposer and op-dispute-mon can run against kona-node verifiers without an op-node in the loop.

Current status

The kona-safedb crate landed in #21522. Safe-head recording, reset/EL-sync truncation and clearing, and fail-fast persistence handling merged in #23583 on 2026-10-23, completing #23474. Core integration merged in #23590 on 2026-10-27: the SafeDB flag, optimism_safeHeadAtL1Block, pre-interop superroot_atTimestamp, and direct op-node/Kona parity coverage.

Super-root withdrawal coverage and SafeDB configuration through normal Kona startup merged in #23704 on 2026-10-28. The direct mixed-node CL-resync and wrong-key coverage merged in #23818 on 2026-10-31, the failed-reset fix in #23815 on 2026-11-04, and the acceptance/proof-consumer tests and public RPC documentation in #23787 (5b3854d301) on 2026-11-12. Since #23827, the two SafeDB resync suites run on op-reth in the normal acceptance jobs, with no op-geth dependency. Every sub-issue is closed. On develop (CircleCI pipeline 139914, 7f7abd37fd), safedb_parity, both safeheaddb resync suites, safedb_proof_consumers, cannon_kona and dispute-mon pass with both kona-node and op-node.

Remaining before this issue closes: fix #24130 (high) and #24131 (medium), then a passing safedb-reviewer re-run on the final integration (first run failed). The ethereum-optimism/protocol-team#283 proposer/challenger rollout is unblocked and tracked there.

Soak progress — 2026-10-29

The Sepolia-backed kona-safedb-0 network was created by devnets-private#2227. Its initial rollout failed because the non-root Kona process could not write /admin/safedb. devnets-private#2296 set fsGroup: 10001; the original 1 GiB PVC was retained. Netchef then completed successfully and devnets-private#2301 marked the network online. Kona is pinned to the merged #23590 build cfec650e57ecedf04ad878891ecc79eaca414365.

  • Completed window: 2026-10-28 19:50:49 UTC through 2026-10-29 20:05:25 UTC, approximately 24h15m.
  • 24-hour soak passed on 2026-10-29: 7,454/7,454 SafeDB queries matched op-node: 7,422 consecutive queries across L1 12,014,303–12,021,724 plus 32 older queries, with zero RPC errors/mismatches and all 5,758 prior answers unchanged. The upper bound was safe on both nodes at 19:51:48 UTC, beyond the 19:50:49 UTC soak endpoint. Kona retained its original PVC and had zero container restarts after the controlled pod recreation. Final evidence.
  • The controlled Kona-only pod recreation retained the EL and original PVC. All 160 restart baselines matched after rederivation; the initial temporary rewind remains documented.
  • Final validation confirmed Ready, unchanged pod/image/PVC/bound volume, zero container restarts after the controlled recreation, fsGroup: 10001, and /admin/safedb.
  • Full-period log/restart review is complete: 122 Kona warnings and 10 peer-manager errors were confined to restart/catch-up; later entries were 1,111 op-node possible-L1-reorg warnings and 25 batcher warnings, with subsequent confirmed publication. All 1,268 matching lines were retained below the query cap. Detailed final report.
  • Checks were manual. The persistence soak is complete; kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23479 is closed. The remaining acceptance and consumer requirements stay open here and in kona-node,safedb,op-acceptance-tests: validate restart persistence #22265/kona-node,safedb,op-acceptance-tests: validate Cannon execution-step proofs #22279.

Remaining scope and deferred work

All six mixed-node parity cases, the wrong-key negative control, the failed-reset fix and the consumer/documentation coverage are merged (#23818, #23815, #23787). The 24-hour persistence soak is complete.

Long-term resume-from-tip recovery (#23658), interval-based gap detection (#23659), node-side metrics and gap-risk warnings (#23478), the history-dump RPC (#23584), dashboard panels/alerts (#23481), and the write-once-per-L1-block optimization (#23480, inherited from #22528 and #23112) are tracked under ethereum-optimism/protocol-team#442 and deferred until after the EoY milestone. They are not completion gates for this issue.

op-challenger clamps its honest output root using optimism_safeHeadAtL1Block; the super-root proposer/challenger paths read superroot_atTimestamp; op-dispute-mon uses safe-head history to judge L1 backing. These consumers require the SafeDB flag and recorded history. Acceptance/proofs-stack validation and the soak remain the rollout dependency for adding a proposer and challenger to ethereum-optimism/protocol-team#283.

Supersedes #22528 and the placeholder #20278.

Components

Each is a sub-issue of this ticket:

  1. kona-node: record safe heads into kona-safedb #23474 — Record safe heads into kona-safedb (merged in kona-node: record the safe head history into kona-safedb #23583).
  2. kona-node,safedb: serve optimism_safeHeadAtL1Block #23475 — Serve optimism_safeHeadAtL1Block (merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590).
  3. kona-node,safedb: serve superroot_atTimestamp before interop #23476 — Serve superroot_atTimestamp pre-interop, with RequiredL1 from the database (RPC merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590; withdrawal coverage merged in op-acceptance-tests,op-devstack: enable super-root withdrawals on kona-node #23704).
  4. kona-node,safedb: add configuration, disabled-state errors, and documentation #23477 — Config flag and disabled-state error shape (merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590); public RPC-reference documentation remains unchecked below.
  5. kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23479 — Closed after the successful 24-hour soak; remaining acceptance work is retained in kona-node,safedb,op-acceptance-tests: validate restart persistence #22265, kona-node,safedb,op-acceptance-tests: validate Cannon execution-step proofs #22279, and this parent.
  6. kona-node,safedb,op-acceptance-tests: verify parity with op-node #23585 — Direct op-node/Kona SafeDB parity coverage (merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590); all six implemented cases now pass in op-acceptance-tests,safedb: validate parity after Kona resync #23818, whose follow-up merge remains pending.

Definition of done

Core implementation checkmarks are backed by merged #23583, #23590, #23704, #23815, #23818 and #23787. Live restart/deployment checks are scoped to the soak evidence. All six mixed-node parity cases are merged (#23818) and pass on develop, and the deliberate wrong-key red/green control is recorded there. The safedb-reviewer pass on the final integration remains open. The merged persistence policy is fail-fast, without write retries. Observability follow-up (#23478) is tracked under ethereum-optimism/protocol-team#442 and is not required to complete this issue.

Behavior:

Tests:

Operations:

Current completion checklist — 2026-11-13

#22265, #22279 and #23479 are closed. Canonical-tip recovery (#23658), interval gap representation (#23659), metrics (#23478), dashboards (#23481), dump RPC (#23584), and write buffering (#23480) remain deferred under ethereum-optimism/protocol-team#442.

Runtime evidence: #23818 (comment)

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. 5 remaining items

  3. changed the title [-]kona-node: Kona-SafeDB integration[/-] [+]kona-node,safedb: integrate persistence, RPCs, and proof consumers[/+] on Oct 26, 2026
  4. einar-oplabs commented on Oct 29, 2026

    @einar-oplabs
    ContributorAuthor

    The 24-hour SafeDB persistence soak passed on kona-safedb-0.

    • Window: 2026-10-28 19:50:49 UTC → 2026-10-29 20:05:25 UTC (about 24h15m). Kona image: cfec650e57ecedf04ad878891ecc79eaca414365, with one op-node sequencer and op-node/Kona reference verifiers on op-reth.
    • Final scan: 7,454/7,454 queries match op-node, with zero RPC errors, missing/duplicate responses, later-than-requested L1 answers, or parity mismatches. Every L1 block 12,014,303–12,021,724 was queried (7,422 consecutive requests), plus 32 older baseline requests. All 5,758 previously saved answers are unchanged.
    • Endpoint coverage: L1 12,021,724 was safe on both nodes, with hash 0xfb8d20ae3a16d5fdef2510e178252807ae8d5d0ec8eb199eb030a31af9eeb4c8 and timestamp 2026-10-29 19:51:48 UTC, beyond the 24-hour endpoint at 19:50:49 UTC. Both nodes had processed beyond it. This is an RPC-level contiguous comparison, not a new storage-interval/gap-detection implementation.
    • Persistence/restart: the controlled Kona-only pod recreation retained the EL and original PVC. All 160 restart baselines matched after rederivation; answers initially rewound during catch-up and that intermediate evidence is retained. At final validation Kona was Ready, with zero container restarts after that recreation and unchanged pod UID, image, PVC UID, and bound volume. fsGroup: 10001 and /admin/safedb were verified.
    • Full-period warning/error review: 1,268 matching lines, below the 5,000 cap. The 122 Kona warnings and 10 peer-manager errors were confined to the controlled restart/catch-up at 19:52–19:54 UTC on October 28. Later entries were 1,111 op-node possible-L1-reorg warnings and 25 batcher warnings: 23 blob-fee retries, one canceled fee request, and one skipped publish signal. No later Kona warning/error lines were returned. The batcher confirmed a transaction at 20:03:55 UTC on October 29. This was not an error-free log stream; the reviewed warnings did not produce a lasting historical RPC mismatch.

    Final Slack checkpoint. Complete JSON responses, pod/PVC snapshots, log review, and the final report are retained locally under kona-safedb-soak-evidence/2026-10-28/2026-10-29T200437Z-final. All checks were manual; no automatic hourly schedule was enabled.

    Closing #23479 as requested; #23473 remains open for the remaining integration requirements. This closure does not claim that the outstanding acceptance changes are merged or passing CI. Remaining SafeDB EL/CL-resync acceptance work stays open in #22265; Cannon execution-step and output-root game-resolution validation stays open in #22279 and #23473; merged dispute-monitor acceptance, the final reset/reorg parity-suite rerun, and public RPC documentation stay tracked in #23473. Super-root withdrawal consumer CI is already merged in #23704. Long-term recovery, interval detection, metrics, dashboards, dump RPC, and write buffering remain open under ethereum-optimism/protocol-team#442.

  5. einar-oplabs commented on Nov 13, 2026

    @einar-oplabs
    ContributorAuthor

    Opus 5.5 (high): safedb-reviewer pass on the merged kona-node SafeDB integration, develop at 7f7abd37fd. It followed .claude/agents/safedb-reviewer.md and docs/ai/safedb-review.md, read-only, with op-node as the reference. Gate: FAIL, because of F1.

    Invariants

    • I1 (local-safe only): holds. Recorded heads come from local_safe_head(), and no engine path sets safe without local-safe.
    • I2 (no gaps): violated by F1, which is inferred from code. The reset and EL-sync paths themselves cannot leave a gap.
    • I3 (truncation point = resume point): holds at both truncation sites (engine reset and EL-sync completion). F1 breaks it afterwards.
    • I4 (L1-derived only): holds. The only write is derived_from for a confirmed matching block. The delegate actor gets no database, and the CLI refuses --safedb.path together with delegation.
    • I5 (stable after L1 passed): holds, as for op-node.
    • I6 (exact): holds. The queries and miss kinds match op-node's safedb.go, and the error strings match api.go.

    Findings

    • F1 (high): attributes derived before an engine reset can consolidate after it. Local-safe then jumps over blocks that derivation never confirmed, and the SafeDB never records them. Later answers are confidently wrong: a later required L1 from superroot_atTimestamp, and a safe head that is too low from optimism_safeHeadAtL1Block. → kona-engine: consolidate safe attributes only on top of the local-safe head #24130
    • F2 (medium): after an outage longer than a sequencing window, the reset walk-back is not capped at the EL safe head. The reset head lands above the tip and the whole history is cleared, where op-node keeps it. Answers go missing; none are wrong. → kona-engine: keep SafeDB history after an outage longer than the sequencing window #24131
    • F3 (low, no action): kona records blocks in the middle of a span batch; op-node moves local-safe only when the span concludes. Every block in a span shares one derived_from, so settled history is identical. The difference shows only in answers for the L1 block still being processed (the I5 window).
    • F4 (low): two docs are out of date. docs/ai/safedb-review.md:30 still says kona's SafeDB is not wired, and safedb/src/traits.rs:28-31 states a resume rule kona no longer follows. To be fixed with F1 in kona-engine: consolidate safe attributes only on top of the local-safe head #24130.
    • F5 (low, no action): RollupNodeBuilder accepts a safe_db together with delegation and would serve a database that nothing writes to. Only the CLI guard (bin/node/src/commands/node/command.rs:333-339) prevents that, so the shipped binary cannot reach it.

    Test gaps: F1 needs a test that delivers attributes derived before a reset after it, with history asserted below the tip. F2 needs a test with an outage longer than the sequencing window, then a restart, checking that history is kept.

    The gate item above stays unchecked until #24130 is fixed and a re-run passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions