You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Superseded by #23481 (dashboard panels and alerts), with node-side signals in #23478, under ethereum-optimism/protocol-team#442. This closed ticket is historical tracking; its state does not establish implemented or tested dashboards.
The merged persistence policy is fail-fast with no write retries; the older retry-rate/exhausted-retry requirements below are historical and superseded by #23481's fatal-persistence-failure signals.
Part of #23473. Depends on the metrics from #23478.
Scope
Make the kona-node safedb observable on the devnets where kona-node runs, and page when it is unhealthy. The implementation lands in the k8s repo (Grafana dashboards and alert rules), following the existing k8s alert conventions (team=protocol routes to the protocol on-call; P3 and below is informational).
Dashboard, on the kona-node dashboard:
safedb tip L1 block against the node's current L1 head, so lag is visible at a glance;
safedb tip L2 block against local-safe;
write, retry and truncation rates;
a table of recent reset-below-tip warnings.
Alerts:
safedb tip lagging the L1 head by more than a channel timeout for longer than 15 minutes while derivation is otherwise progressing (a gap is forming);
any reset-below-tip warning (the condition that produced the hole in ethereum-optimism/protocol-team#402);
write retries exhausted (the node stops, so this is a crash alert with a specific cause).
Threshold choices and a one-paragraph runbook per alert go into the alert annotations.
Done
Dashboard panels merged and visible for every kona-node on the kona-node sequencer devnet (ethereum-optimism/protocol-team#283).
The three alerts are merged, routed to protocol, and each has a runbook annotation.
Each alert has fired at least once in a deliberate test (for example by stopping writes or restoring a stale snapshot) and the on-call received it.
Current integration status: the 24-hour soak passed (evidence); manual Grafana log review does not complete #23481.
Activity
einar-oplabs commentedon Oct 21, 2026
Duplicate of #23481, which already covers the safedb dashboard panels and alerts. Closing in its favor.