Skip to content

kona-node: safedb dashboard and alerts #23573

Closed as not planned
Closed as not planned

Description

@einar-oplabs

Superseded by #23481 (dashboard panels and alerts), with node-side signals in #23478, under ethereum-optimism/protocol-team#442. This closed ticket is historical tracking; its state does not establish implemented or tested dashboards.

The merged persistence policy is fail-fast with no write retries; the older retry-rate/exhausted-retry requirements below are historical and superseded by #23481's fatal-persistence-failure signals.

Part of #23473. Depends on the metrics from #23478.

Scope

Make the kona-node safedb observable on the devnets where kona-node runs, and page when it is unhealthy. The implementation lands in the k8s repo (Grafana dashboards and alert rules), following the existing k8s alert conventions (team=protocol routes to the protocol on-call; P3 and below is informational).

Dashboard, on the kona-node dashboard:

  • safedb tip L1 block against the node's current L1 head, so lag is visible at a glance;
  • safedb tip L2 block against local-safe;
  • write, retry and truncation rates;
  • a table of recent reset-below-tip warnings.

Alerts:

  • safedb tip lagging the L1 head by more than a channel timeout for longer than 15 minutes while derivation is otherwise progressing (a gap is forming);
  • any reset-below-tip warning (the condition that produced the hole in ethereum-optimism/protocol-team#402);
  • write retries exhausted (the node stops, so this is a crash alert with a specific cause).

Threshold choices and a one-paragraph runbook per alert go into the alert annotations.

Done

  • Dashboard panels merged and visible for every kona-node on the kona-node sequencer devnet (ethereum-optimism/protocol-team#283).
  • The three alerts are merged, routed to protocol, and each has a runbook annotation.
  • Each alert has fired at least once in a deliberate test (for example by stopping writes or restoring a stale snapshot) and the on-call received it.

Current integration status: the 24-hour soak passed (evidence); manual Grafana log review does not complete #23481.

Activity

  1. self-assigned this
    on Oct 21, 2026
  2. einar-oplabs commented on Oct 21, 2026

    @einar-oplabs
    ContributorAuthor

    Duplicate of #23481, which already covers the safedb dashboard panels and alerts. Closing in its favor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions