Repository navigation
Description
Part of #23473. Closed after the successful 24-hour persistence soak. Unmerged acceptance work remains open in #22265, #22279, and #23473; closure does not mark those requirements complete.
Original scope and remaining tracking
Prove the integration end to end against kona-node, not just in unit tests.
Acceptance suites:
- Remove
SkipOnKonaNodefromop-acceptance-tests/tests/safeheaddb_elsyncand runsafeheaddb_clsyncagainst kona-node. Closes kona-node,safedb,op-acceptance-tests: validate restart persistence #22265. - Run
TestExecuteStep_CannonKonaagainst kona-node. Closes kona-node,safedb,op-acceptance-tests: validate Cannon execution-step proofs #22279. - Direct op-node versus kona-node comparison on one chain is its own sub-issue, kona-node,safedb,op-acceptance-tests: verify parity with op-node #23585; coverage merged in kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590. The final full-suite rerun remains tracked in kona-node,safedb: integrate persistence, RPCs, and proof consumers #23473.
Proofs stack:
- An output-root fault dispute game is played to resolution by an op-challenger whose only rollup node is kona-node.
- op-proposer in super-root mode and op-dispute-mon run against kona-node only.
Soak:
- A kona-node on a devnet runs 24 hours with the flag on and a contiguity check (safedb-scan from infrastructure-services, or an in-tree equivalent) reports no gap. Record restarts during the soak and confirm the history survives them.
Current progress — 2026-10-29
Merged coverage
- kona-node: record the safe head history into kona-safedb #23583 merged safe-head persistence and reset/EL-sync handling.
- kona-node,op-acceptance-tests: serve the safe head history and compare it with op-node #23590 merged the SafeDB flag, both RPCs, and direct parity coverage.
- op-acceptance-tests,op-devstack: enable super-root withdrawals on kona-node #23704 merged normal-startup SafeDB configuration and enabled super-permissioned and SuperCannonKona withdrawal acceptance against Kona. Both withdrawals pass through finalization on Kona and op-node; all 51 CI checks passed. This supplies merged super-root proposer/challenger consumer coverage.
Historical local acceptance work — 2026-10-26
The local branch einar/23479-kona-safedb-acceptance at a676f931a6 contains additional SafeDB, Cannon, and proof-consumer acceptance work. Saved runs from 2026-10-26 passed the SafeDB EL/CL-resync suites with Kona/op-geth and op-node/op-geth, the Kona/op-reth restart case, TestExecuteStep_CannonKona with both node implementations, and proposer/dispute-monitor checks with both node implementations. The execution-step test waits for game resolution; dispute-monitor cases cover valid/invalid roots and a root that was not safe at the game's L1 head. A disabled-SafeDB negative control failed as expected.
These are local development results, not evidence that the changes are merged or passing CI on the final integration. That original branch remains local; its remaining work is now carried onto current develop in #23787. The remaining suite/step/dispute-monitor work is carried by the open tickets listed below; it is not completed by this closure.
Live persistence soak
The Sepolia-backed kona-safedb-0 topology has one op-node sequencer, one op-node reference verifier, and one Kona verifier, all on op-reth. devnets-private#2227 created the network. The initial rollout failed because Kona's non-root user could not create /admin/safedb. devnets-private#2296 fixed PVC permissions with fsGroup: 10001; the existing 1 GiB PVC was retained. Netchef completed and devnets-private#2301 marked the network online. Kona runs the merged #23590 build cfec650e57ecedf04ad878891ecc79eaca414365.
- Window: 2026-10-28 19:50:49 UTC through 2026-10-29 20:05:25 UTC, approximately 24h15m.
- 24-hour soak passed on 2026-10-29: 7,454/7,454 SafeDB queries matched op-node: 7,422 consecutive queries across L1 12,014,303–12,021,724 plus 32 older queries, with zero RPC errors/mismatches and all 5,758 prior answers unchanged. The upper bound was safe on both nodes at 19:51:48 UTC, beyond the 19:50:49 UTC soak endpoint. Kona retained its original PVC and had zero container restarts after the controlled pod recreation. Final evidence.
- Controlled restart: only the Kona pod was recreated, retaining the EL and original PVC. All 160 pre-restart historical answers matched after rederivation. Initial temporary rewinds during catch-up are retained in the evidence; uninterrupted parity during rederivation is not claimed.
- Final health: Ready; pod UID, image, PVC UID and bound volume unchanged; zero container restarts after the controlled recreation;
fsGroup: 10001and/admin/safedbverified. - Full-period logs reviewed: 1,268 warning/error lines, below the 5,000 cap. The 122 Kona warnings and 10 peer-manager errors occurred only during the controlled restart/catch-up at 19:52–19:54 UTC on October 28. Later entries were 1,111 op-node possible-L1-reorg warnings and 25 batcher warnings (23 blob-fee retries, one canceled fee request, one skipped publish signal). The batcher confirmed a transaction at 20:03:55 UTC on October 29. These findings are recorded, not treated as an error-free log stream.
- Monitoring: manual checks; no automatic hourly schedule was enabled.
- Result on the parent: Final report on #23473.
Done
- Super-root withdrawal consumer coverage passes against Kona in merged CI (op-acceptance-tests,op-devstack: enable super-root withdrawals on kona-node #23704).
- The persistent soak network is online with SafeDB enabled.
- A controlled Kona-only restart retains the PVC and the tested historical answers match again after rederivation.
- The intermediate full-range scan through finalized L1 block 12,019,356 passes: 5,086 queries match op-node, including the 32 older baseline queries; all 575 previously saved answers are unchanged (2026-10-29 12:15 UTC).
- Errors/warnings since the restart verification and pod restart/PVC state are reviewed through the 2026-10-29 12:15 UTC checkpoint; findings are recorded above.
- At least 24 hours of soak operation are complete and a full-range check reports no gap or parity mismatch.
- The full soak's errors and restarts are reviewed and recorded.
- Final soak result is posted on kona-node,safedb: integrate persistence, RPCs, and proof consumers #23473.
Remaining work carried by open tickets
- kona-node,safedb,op-acceptance-tests: validate restart persistence #22265: remove the Kona SafeDB acceptance skips and obtain merged CI coverage for the EL/CL-resync suites; the live restart result does not replace these suites.
- kona-node,safedb,op-acceptance-tests: validate Cannon execution-step proofs #22279: remove the Cannon execution-step skip and obtain merged CI evidence. The output-root challenger/game-resolution requirement is also retained in kona-node,safedb: integrate persistence, RPCs, and proof consumers #23473.
- kona-node,safedb: integrate persistence, RPCs, and proof consumers #23473: merged dispute-monitor acceptance, final reset/reorg/CL-resync parity-suite rerun, and public RPC documentation remain completion requirements. kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers #23787 carries this acceptance work onto current develop and its signed commit has 158 passing checks and one skipped check. Native SafeDB, Cannon, and consumer cases pass; merge remains pending.
The 24-hour scan proves contiguous RPC parity over the observed range. It does not implement interval-based storage/gap detection or validate true EL-data-directory wipe/resync recovery; those remain deferred in #23659 and #23658.
Follow-up: ready-for-review #23787 carries the acceptance and RPC documentation work with green CI. Ready-for-review #23815 fixes failed-reset propagation, and #23818 supplies the mixed-node resync and wrong-key controls; both also have green CI. This soak ticket stays closed; #22265, #22279, and #23473 remain open until their own requirements are met.
Follow-up validation — 2026-10-30
#23787, #23815, and #23818 are ready for review with green CI. All six mixed-node parity cases pass; the intentional wrong-key control fails with the expected mismatch, and both positive cases pass after exact source restoration. The failed-reset fix also has actor and service/RPC shutdown red/green regressions, with all 252 service tests passing locally. This soak ticket remains closed; human review/merge and the combined integration/rollout gates remain in #23473.
Activity
[-]op-acceptance-tests: integration tests for kona-node safedb[/-][+]kona-node,safedb,op-acceptance-tests: validate persistence and proof consumers[/+]einar-oplabs commentedon Oct 29, 2026
einar-oplabs commentedon Oct 29, 2026
The 24-hour SafeDB persistence soak passed on
kona-safedb-0.cfec650e57ecedf04ad878891ecc79eaca414365, with one op-node sequencer and op-node/Kona reference verifiers on op-reth.12,021,724was safe on both nodes, with hash0xfb8d20ae3a16d5fdef2510e178252807ae8d5d0ec8eb199eb030a31af9eeb4c8and timestamp 2026-10-29 19:51:48 UTC, beyond the 24-hour endpoint at 19:50:49 UTC. Both nodes had processed beyond it. This is an RPC-level contiguous comparison, not a new storage-interval/gap-detection implementation.fsGroup: 10001and/admin/safedbwere verified.Final Slack checkpoint. Complete JSON responses, pod/PVC snapshots, log review, and the final report are retained locally under
kona-safedb-soak-evidence/2026-10-28/2026-10-29T200437Z-final. All checks were manual; no automatic hourly schedule was enabled.Closing #23479 as requested; #23473 remains open for the remaining integration requirements. This closure does not claim that the outstanding acceptance changes are merged or passing CI. Remaining SafeDB EL/CL-resync acceptance work stays open in #22265; Cannon execution-step and output-root game-resolution validation stays open in #22279 and #23473; merged dispute-monitor acceptance, the final reset/reorg parity-suite rerun, and public RPC documentation stay tracked in #23473. Super-root withdrawal consumer CI is already merged in #23704. Long-term recovery, interval detection, metrics, dashboards, dump RPC, and write buffering remain open under ethereum-optimism/protocol-team#442.