Skip to content

kona-node,safedb,grafana: add dashboard panels and alerts #23481

Open

Description

@einar-oplabs

Part of ethereum-optimism/protocol-team#442 (long-term Kona improvements). Related to #23473; dashboards and alerts are not a completion requirement for the current SafeDB integration. Depends on the node-side metrics, warnings, and failure signals in #23478. Dashboard JSON and alert rules land in ethereum-optimism/k8s.

Scope

Make Kona SafeDB health visible on the kona-node Grafana dashboard and route actionable alerts to team=protocol. Reuse op-node SafeDB panels where available so both clients read consistently.

Panels (one row, lowercase side: metric (window)):

  • safedb: tip l1 block beside the L1 head, with lag as a stat.
  • safedb: tip l2 block beside local-safe, with lag as a stat.
  • safedb: writes (5m) and safedb: persistence failures (5m).
  • safedb: truncations (1h) and safedb: history clears (1h), split by reason (reset, EL-sync).
  • safedb: tip below resume point, showing reset-below-tip events from kona-node,safedb,gap-detection: expose metrics and gap-risk warnings #23478.
  • Distinguish disabled, empty, unreadable, and unavailable/scrape-missing states from a healthy zero value.

Alerts:

  • SafeDB tip L1 lag above a documented threshold for 15 minutes while derivation is advancing; validate the rule against legitimate periods with no safe-head advancement.
  • Any tip-below-resume-point event (the gap-risk condition from ethereum-optimism/protocol-team#402).
  • A fatal SafeDB persistence failure, using the error/process-failure signals from kona-node,safedb,gap-detection: expose metrics and gap-risk warnings #23478 and counters where available. The alert must work if the node exits before the failure counter is scraped.

The persistence policy is fail-fast, with no write retries. There is no retry-rate panel or exhausted-retry alert. Interval-based historical gap detection remains separate long-term work in #23659.

Done and testing

  • Panels render on the kona-node sequencer devnet (ethereum-optimism/protocol-team#283), including disabled and unavailable states.
  • All three alerts route to protocol and include recovery guidance, linking to ethereum-optimism/protocol-team#404 when available.
  • Thresholds and their reasoning are recorded here; normal periods without safe-head advancement do not produce false stall alerts.
  • Alert-rule tests cover normal operation, lag, reset-below-tip, and fatal persistence failure before the next metrics scrape.

Status — 2026-10-29

The current integration passed its 24-hour persistence soak and #23479 is closed. Final evidence. Grafana log access supported a manual review. This does not mean the proposed SafeDB panels or alert rules were implemented or deliberately tested. This issue remains open as deferred long-term work under ethereum-optimism/protocol-team#442; none of its implementation checkboxes are completed by the soak.

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. changed the title [-]kona-node: SafeDB dashboard panels and metrics[/-] [+]kona-node: SafeDB dashboard panels and alerts[/+] on Oct 26, 2026
  3. changed the title [-]kona-node: SafeDB dashboard panels and alerts[/-] [+]kona-node,safedb: add dashboard panels and alerts[/+] on Oct 26, 2026
  4. changed the title [-]kona-node,safedb: add dashboard panels and alerts[/-] [+]kona-node,safedb,grafana: add dashboard panels and alerts[/+] on Oct 26, 2026
  5. removed their assignment
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions