Skip to content

alerting: bench coverage rules (Railway prom) #1002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 1 commit into from
Jul 8, 2026
Merged

alerting: bench coverage rules (Railway prom) #1002

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions infrastructure/monitoring/prometheus/alert_rules.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,50 @@ groups:
annotations:
summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Latency monitor service is down'
description: "The aggregator latency monitor has been down for 2 minutes. No metrics are being collected."

- name: bench_coverage_alerts
interval: 60s
rules:
# RPC provider error-rate (no-key + keyed): the 1RPC/dRPC per-IP
# throttling incident (2026-07-08) ran for days unalerted. Fires
# when a (provider, chain) cell errors on >50% of calls for 1h.
- alert: RpcProviderErrorRate
expr: |
(
sum by (provider, chain) (rate(rpc_call_total{result!="ok"}[30m]))
/ sum by (provider, chain) (rate(rpc_call_total[30m]))
) > 0.5
for: 1h
labels:
severity: warning
alert_type: rpc_error_rate
app: rpc_benches
annotations:
summary: '{{ $labels.provider }} erroring on {{ $labels.chain }} ({{ $value | humanizePercentage }})'
description: "More than half of probes to {{ $labels.provider }} on {{ $labels.chain }} failed over the last 30m, sustained 1h. Rate limit, dead endpoint, or our own footprint (check probe cadence before blaming the provider)."

# Keyed free-tier quota guard approaching: the guard pauses at 90%,
# this warns at 85% so a human can react before data gaps appear.
- alert: KeyedQuotaNearExhaustion
expr: rpc_keyed_quota_used_ratio > 0.85
for: 10m
labels:
severity: warning
alert_type: quota
app: rpc_benches
annotations:
summary: 'Keyed RPC quota {{ $labels.provider }} at {{ $value | humanizePercentage }} of monthly budget'
description: "The 90% guard will pause probing soon; leaderboard gaps follow. Consider a cadence reduction or a bigger plan."

# Aggregator WS silently down: connection-state gauge from the
# reconnect instrumentation (PR #950).
- alert: AggregatorWSDisconnected
expr: ws_connected == 0
for: 10m
labels:
severity: warning
alert_type: ws_state
app: aggregator_latency_monitor
annotations:
summary: '{{ $labels.aggregator }} WebSocket disconnected ({{ $labels.region }})'
description: "ws_connected has been 0 for 10 minutes; reconnect loop is running but not succeeding. Check JWT/cookie/proxy for codex, API key for mobula."
Loading