Repository navigation
Description
Part of ethereum-optimism/protocol-team#442 (long-term Kona improvements); related to #23473. Produces the node-side signals consumed by #23481; dashboards and alert rules are implemented there.
Scope
Expose SafeDB health at the node/service layer, consistent with the review direction on #23583 and ethereum-optimism/protocol-team#405. Keep the storage library free of duplicate operation logging.
- Persisted-tip L1/L2 gauges, restored from existing records on startup and updated only after successful persistence. Distinguish disabled, empty, and unreadable databases.
- Counters for successful writes, persistence failures, successful truncations, and successful history clears. Label reset/EL-sync operations by reason, with bounded label values.
- A warning and an observable event when the persisted tip is below the resume head and history must be cleared, including reset versus EL-sync context (ethereum-optimism/protocol-team#402).
- Operation logs at database startup, confirmed derived-head writes, reset/EL-sync truncation and clearing, and SafeDB RPC handling. Include attempts, outcomes, errors, and relevant L1/L2 context without duplicating lower-layer messages.
The merged policy is fail-fast: database write failures stop the actor immediately, and finalization is queued only after successful recording. There are no write retries; do not add retry metrics. A fatal persistence failure must remain observable through an error log and the existing process/actor failure signal, since the node may exit before its final counter increment is scraped.
This issue exposes known gap-risk conditions; interval-based detection of missing historical coverage is separate long-term work in #23659.
Done and testing
- Metrics are exported under the kona-node metrics prefix and documented, including disabled/empty/unreadable states.
- Startup restores persisted-tip gauges; failed operations never advance those gauges or successful-operation counters.
- Reset-below-tip warnings/events distinguish reset from EL sync and can be consumed by kona-node,safedb,grafana: add dashboard panels and alerts #23481.
- Fatal persistence errors include operation and block context and remain observable when the node stops before a metrics scrape.
- Unit tests use the SafeDb mock seam for successful/failed writes, reset/clear failures, empty/unreadable tips, disabled configuration, and reset/EL-sync variants, asserting metric and event behavior.
- Dashboard panels and alert rules consuming these signals are tracked in kona-node,safedb,grafana: add dashboard panels and alerts #23481.
Status — 2026-10-29
The current integration passed its 24-hour persistence soak and #23479 is closed. Final evidence. Health and logs were inspected manually through Kubernetes/Grafana. This does not implement the proposed SafeDB metrics, startup gauges, or gap-risk events. This issue remains open as deferred long-term work under ethereum-optimism/protocol-team#442; none of its implementation checkboxes are completed by the soak.
Activity
einar-oplabs commentedon Oct 23, 2026
Follow-up from Josh’s review of #23583: keep observability at the highest layer that owns the operation, rather than inside the
kona-safedbstorage library. Review context: #23583 (comment)The library’s operation logs and its
tracingdependency are being removed from #23583. The existing derivation actor logs remain; a broader observability pass is deferred to this issue.SafeDatabasewill hold an open RocksDB handle until dropped, with no explicitclose()or internal closed state.Work to pick up here:
settle_derived_attributes), reset and EL-sync truncation/clearing (truncate_safe_db), database startup, and eventual RPC handlers. Cover attempts, successes, failures, and relevant L1/L2 block context without duplicate library messages.SafeDbmock seam to test successful/failed writes, reset/clear failures, empty or unreadable tips, disabled configuration, and reset/EL-sync variants. Assert that failed operations do not advance persisted-tip metrics or success counters and that the gap warning is observable.This keeps the implementation and review of the observability policy out of #23583 while preserving existing operational logging.
[-]kona-node: safedb metrics and gap visibility[/-][+]kona-node,safedb: expose metrics and gap-risk warnings[/+][-]kona-node,safedb: expose metrics and gap-risk warnings[/-][+]kona-node,safedb,gap-detection: expose metrics and gap-risk warnings[/+]