Skip to content

kona-node,safedb,gap-detection: expose metrics and gap-risk warnings #23478

Open

Description

@einar-oplabs

Part of ethereum-optimism/protocol-team#442 (long-term Kona improvements); related to #23473. Produces the node-side signals consumed by #23481; dashboards and alert rules are implemented there.

Scope

Expose SafeDB health at the node/service layer, consistent with the review direction on #23583 and ethereum-optimism/protocol-team#405. Keep the storage library free of duplicate operation logging.

  • Persisted-tip L1/L2 gauges, restored from existing records on startup and updated only after successful persistence. Distinguish disabled, empty, and unreadable databases.
  • Counters for successful writes, persistence failures, successful truncations, and successful history clears. Label reset/EL-sync operations by reason, with bounded label values.
  • A warning and an observable event when the persisted tip is below the resume head and history must be cleared, including reset versus EL-sync context (ethereum-optimism/protocol-team#402).
  • Operation logs at database startup, confirmed derived-head writes, reset/EL-sync truncation and clearing, and SafeDB RPC handling. Include attempts, outcomes, errors, and relevant L1/L2 context without duplicating lower-layer messages.

The merged policy is fail-fast: database write failures stop the actor immediately, and finalization is queued only after successful recording. There are no write retries; do not add retry metrics. A fatal persistence failure must remain observable through an error log and the existing process/actor failure signal, since the node may exit before its final counter increment is scraped.

This issue exposes known gap-risk conditions; interval-based detection of missing historical coverage is separate long-term work in #23659.

Done and testing

  • Metrics are exported under the kona-node metrics prefix and documented, including disabled/empty/unreadable states.
  • Startup restores persisted-tip gauges; failed operations never advance those gauges or successful-operation counters.
  • Reset-below-tip warnings/events distinguish reset from EL sync and can be consumed by kona-node,safedb,grafana: add dashboard panels and alerts #23481.
  • Fatal persistence errors include operation and block context and remain observable when the node stops before a metrics scrape.
  • Unit tests use the SafeDb mock seam for successful/failed writes, reset/clear failures, empty/unreadable tips, disabled configuration, and reset/EL-sync variants, asserting metric and event behavior.
  • Dashboard panels and alert rules consuming these signals are tracked in kona-node,safedb,grafana: add dashboard panels and alerts #23481.

Status — 2026-10-29

The current integration passed its 24-hour persistence soak and #23479 is closed. Final evidence. Health and logs were inspected manually through Kubernetes/Grafana. This does not implement the proposed SafeDB metrics, startup gauges, or gap-risk events. This issue remains open as deferred long-term work under ethereum-optimism/protocol-team#442; none of its implementation checkboxes are completed by the soak.

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. einar-oplabs commented on Oct 23, 2026

    @einar-oplabs
    ContributorAuthor

    Follow-up from Josh’s review of #23583: keep observability at the highest layer that owns the operation, rather than inside the kona-safedb storage library. Review context: #23583 (comment)

    The library’s operation logs and its tracing dependency are being removed from #23583. The existing derivation actor logs remain; a broader observability pass is deferred to this issue. SafeDatabase will hold an open RocksDB handle until dropped, with no explicit close() or internal closed state.

    Work to pick up here:

    • Audit logging, metrics, and tracing at the node/service boundary: confirmed derived-head writes (settle_derived_attributes), reset and EL-sync truncation/clearing (truncate_safe_db), database startup, and eventual RPC handlers. Cover attempts, successes, failures, and relevant L1/L2 block context without duplicate library messages.
    • Add persisted-tip L1/L2 gauges and write, failure, truncation, and history-clear counters. Restore tip gauges from existing records on startup; update them only after successful persistence. Keep disabled, empty, and unreadable databases distinguishable.
    • Preserve the warning when the persisted tip is below the resume head and history must be cleared. Include reset versus EL-sync context, and expose this condition for kona-node,safedb,grafana: add dashboard panels and alerts #23481’s dashboards and alerts.
    • The current policy is fail-fast: DB write errors propagate immediately and stop the actor; finalization is queued only after successful recording. There is no retry mechanism, so the original retry-counter/alert requirements need revisiting rather than instrumenting nonexistent retries.
    • Use the SafeDb mock seam to test successful/failed writes, reset/clear failures, empty or unreadable tips, disabled configuration, and reset/EL-sync variants. Assert that failed operations do not advance persisted-tip metrics or success counters and that the gap warning is observable.

    This keeps the implementation and review of the observability policy out of #23583 while preserving existing operational logging.

  3. changed the title [-]kona-node: safedb metrics and gap visibility[/-] [+]kona-node,safedb: expose metrics and gap-risk warnings[/+] on Oct 26, 2026
  4. changed the title [-]kona-node,safedb: expose metrics and gap-risk warnings[/-] [+]kona-node,safedb,gap-detection: expose metrics and gap-risk warnings[/+] on Oct 26, 2026
  5. removed their assignment
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions