Skip to content

kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23479

Closed

Description

@einar-oplabs

Part of #23473. Closed after the successful 24-hour persistence soak. Unmerged acceptance work remains open in #22265, #22279, and #23473; closure does not mark those requirements complete.

Original scope and remaining tracking

Prove the integration end to end against kona-node, not just in unit tests.

Acceptance suites:

Proofs stack:

  • An output-root fault dispute game is played to resolution by an op-challenger whose only rollup node is kona-node.
  • op-proposer in super-root mode and op-dispute-mon run against kona-node only.

Soak:

  • A kona-node on a devnet runs 24 hours with the flag on and a contiguity check (safedb-scan from infrastructure-services, or an in-tree equivalent) reports no gap. Record restarts during the soak and confirm the history survives them.

Current progress — 2026-10-29

Merged coverage

Historical local acceptance work — 2026-10-26

The local branch einar/23479-kona-safedb-acceptance at a676f931a6 contains additional SafeDB, Cannon, and proof-consumer acceptance work. Saved runs from 2026-10-26 passed the SafeDB EL/CL-resync suites with Kona/op-geth and op-node/op-geth, the Kona/op-reth restart case, TestExecuteStep_CannonKona with both node implementations, and proposer/dispute-monitor checks with both node implementations. The execution-step test waits for game resolution; dispute-monitor cases cover valid/invalid roots and a root that was not safe at the game's L1 head. A disabled-SafeDB negative control failed as expected.

These are local development results, not evidence that the changes are merged or passing CI on the final integration. That original branch remains local; its remaining work is now carried onto current develop in #23787. The remaining suite/step/dispute-monitor work is carried by the open tickets listed below; it is not completed by this closure.

Live persistence soak

The Sepolia-backed kona-safedb-0 topology has one op-node sequencer, one op-node reference verifier, and one Kona verifier, all on op-reth. devnets-private#2227 created the network. The initial rollout failed because Kona's non-root user could not create /admin/safedb. devnets-private#2296 fixed PVC permissions with fsGroup: 10001; the existing 1 GiB PVC was retained. Netchef completed and devnets-private#2301 marked the network online. Kona runs the merged #23590 build cfec650e57ecedf04ad878891ecc79eaca414365.

  • Window: 2026-10-28 19:50:49 UTC through 2026-10-29 20:05:25 UTC, approximately 24h15m.
  • 24-hour soak passed on 2026-10-29: 7,454/7,454 SafeDB queries matched op-node: 7,422 consecutive queries across L1 12,014,303–12,021,724 plus 32 older queries, with zero RPC errors/mismatches and all 5,758 prior answers unchanged. The upper bound was safe on both nodes at 19:51:48 UTC, beyond the 19:50:49 UTC soak endpoint. Kona retained its original PVC and had zero container restarts after the controlled pod recreation. Final evidence.
  • Controlled restart: only the Kona pod was recreated, retaining the EL and original PVC. All 160 pre-restart historical answers matched after rederivation. Initial temporary rewinds during catch-up are retained in the evidence; uninterrupted parity during rederivation is not claimed.
  • Final health: Ready; pod UID, image, PVC UID and bound volume unchanged; zero container restarts after the controlled recreation; fsGroup: 10001 and /admin/safedb verified.
  • Full-period logs reviewed: 1,268 warning/error lines, below the 5,000 cap. The 122 Kona warnings and 10 peer-manager errors occurred only during the controlled restart/catch-up at 19:52–19:54 UTC on October 28. Later entries were 1,111 op-node possible-L1-reorg warnings and 25 batcher warnings (23 blob-fee retries, one canceled fee request, one skipped publish signal). The batcher confirmed a transaction at 20:03:55 UTC on October 29. These findings are recorded, not treated as an error-free log stream.
  • Monitoring: manual checks; no automatic hourly schedule was enabled.
  • Result on the parent: Final report on #23473.

Done

  • Super-root withdrawal consumer coverage passes against Kona in merged CI (op-acceptance-tests,op-devstack: enable super-root withdrawals on kona-node #23704).
  • The persistent soak network is online with SafeDB enabled.
  • A controlled Kona-only restart retains the PVC and the tested historical answers match again after rederivation.
  • The intermediate full-range scan through finalized L1 block 12,019,356 passes: 5,086 queries match op-node, including the 32 older baseline queries; all 575 previously saved answers are unchanged (2026-10-29 12:15 UTC).
  • Errors/warnings since the restart verification and pod restart/PVC state are reviewed through the 2026-10-29 12:15 UTC checkpoint; findings are recorded above.
  • At least 24 hours of soak operation are complete and a full-range check reports no gap or parity mismatch.
  • The full soak's errors and restarts are reviewed and recorded.
  • Final soak result is posted on kona-node,safedb: integrate persistence, RPCs, and proof consumers #23473.

Remaining work carried by open tickets

The 24-hour scan proves contiguous RPC parity over the observed range. It does not implement interval-based storage/gap detection or validate true EL-data-directory wipe/resync recovery; those remain deferred in #23659 and #23658.

Follow-up: ready-for-review #23787 carries the acceptance and RPC documentation work with green CI. Ready-for-review #23815 fixes failed-reset propagation, and #23818 supplies the mixed-node resync and wrong-key controls; both also have green CI. This soak ticket stays closed; #22265, #22279, and #23473 remain open until their own requirements are met.

Follow-up validation — 2026-10-30

#23787, #23815, and #23818 are ready for review with green CI. All six mixed-node parity cases pass; the intentional wrong-key control fails with the expected mismatch, and both positive cases pass after exact source restoration. The failed-reset fix also has actor and service/RPC shutdown red/green regressions, with all 252 service tests passing locally. This soak ticket remains closed; human review/merge and the combined integration/rollout gates remain in #23473.

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. changed the title [-]op-acceptance-tests: integration tests for kona-node safedb[/-] [+]kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers[/+] on Oct 26, 2026
  3. einar-oplabs commented on Oct 29, 2026

    @einar-oplabs
    ContributorAuthor

    SafeDB soak checkpoint — October 29, 12:15 UTC

    5,086/5,086 historical queries match op-node, with zero RPC errors and zero mismatches: all 5,054 consecutive L1 queries from 12,014,303–12,019,356, plus 32 older baseline queries. All 575 previously saved answers are unchanged. Both nodes have processed beyond the finalized upper bound.
    Kona is Ready with zero container restarts since the controlled restart. Pod, image, PVC, and bound volume are unchanged; fsGroup is 10001 and SafeDB is at /admin/safedb. Safe and finalized L2 heads match between nodes.
    Log review since October 28 at 19:56 UTC returned no Kona warning/error lines. There were 23 batcher blob-fee retries, one canceled blob-fee request, one skipped publish signal, and 703 op-node possible-L1-reorg warnings (728 matching lines, below the 5,000 limit). The batcher subsequently published and confirmed a transaction at 12:07 UTC. These warnings are retained in the evidence.
    The controlled restart previously passed 160 baseline comparisons after rederivation. Recent answers initially rewound during catch-up; that evidence is retained.

    About 16h24m elapsed, with 7h36m remaining until the 24-hour minimum: October 29 at 19:50:49 UTC / 12:50 PDT. The final full-duration scan and full-period log/restart review remain pending. #23473 and #23479 remain open.

  4. einar-oplabs commented on Oct 29, 2026

    @einar-oplabs
    ContributorAuthor

    The 24-hour SafeDB persistence soak passed on kona-safedb-0.

    • Window: 2026-10-28 19:50:49 UTC → 2026-10-29 20:05:25 UTC (about 24h15m). Kona image: cfec650e57ecedf04ad878891ecc79eaca414365, with one op-node sequencer and op-node/Kona reference verifiers on op-reth.
    • Final scan: 7,454/7,454 queries match op-node, with zero RPC errors, missing/duplicate responses, later-than-requested L1 answers, or parity mismatches. Every L1 block 12,014,303–12,021,724 was queried (7,422 consecutive requests), plus 32 older baseline requests. All 5,758 previously saved answers are unchanged.
    • Endpoint coverage: L1 12,021,724 was safe on both nodes, with hash 0xfb8d20ae3a16d5fdef2510e178252807ae8d5d0ec8eb199eb030a31af9eeb4c8 and timestamp 2026-10-29 19:51:48 UTC, beyond the 24-hour endpoint at 19:50:49 UTC. Both nodes had processed beyond it. This is an RPC-level contiguous comparison, not a new storage-interval/gap-detection implementation.
    • Persistence/restart: the controlled Kona-only pod recreation retained the EL and original PVC. All 160 restart baselines matched after rederivation; answers initially rewound during catch-up and that intermediate evidence is retained. At final validation Kona was Ready, with zero container restarts after that recreation and unchanged pod UID, image, PVC UID, and bound volume. fsGroup: 10001 and /admin/safedb were verified.
    • Full-period warning/error review: 1,268 matching lines, below the 5,000 cap. The 122 Kona warnings and 10 peer-manager errors were confined to the controlled restart/catch-up at 19:52–19:54 UTC on October 28. Later entries were 1,111 op-node possible-L1-reorg warnings and 25 batcher warnings: 23 blob-fee retries, one canceled fee request, and one skipped publish signal. No later Kona warning/error lines were returned. The batcher confirmed a transaction at 20:03:55 UTC on October 29. This was not an error-free log stream; the reviewed warnings did not produce a lasting historical RPC mismatch.

    Final Slack checkpoint. Complete JSON responses, pod/PVC snapshots, log review, and the final report are retained locally under kona-safedb-soak-evidence/2026-10-28/2026-10-29T200437Z-final. All checks were manual; no automatic hourly schedule was enabled.

    Closing #23479 as requested; #23473 remains open for the remaining integration requirements. This closure does not claim that the outstanding acceptance changes are merged or passing CI. Remaining SafeDB EL/CL-resync acceptance work stays open in #22265; Cannon execution-step and output-root game-resolution validation stays open in #22279 and #23473; merged dispute-monitor acceptance, the final reset/reorg parity-suite rerun, and public RPC documentation stay tracked in #23473. Super-root withdrawal consumer CI is already merged in #23704. Long-term recovery, interval detection, metrics, dashboards, dump RPC, and write buffering remain open under ethereum-optimism/protocol-team#442.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions