Repository navigation
epic: alerting system — notifications for network events, health changes, and anomalies #775
Description
Summary
CoreScope sees everything — node health, topology changes, RF anomalies, clock drift — but operators have to manually check to notice problems. When a bridge node goes offline and the mesh splits at 2am, nobody knows until morning.
This epic adds a configurable alerting system that watches for meaningful events and notifies operators through their preferred channels.
Origin: Identified during the neighbor graph epic (#773) — "when a bridge node goes offline and the graph splits, I need a notification."
Alert Categories
Network Topology
| Alert | Trigger |
|---|---|
| Mesh split/merge | Cluster count changed |
| Bridge node offline | Articulation point went stale |
| New node appeared | Previously unseen public key advertised |
| Node went offline | No adverts for configurable threshold (default: 24h for repeaters) |
RF Health
| Alert | Trigger |
|---|---|
| Noise floor spike | Observer reports noise above threshold |
| Observer offline | Observer stopped reporting to MQTT |
| SNR degradation | Link SNR dropped below threshold |
| Packet loss | Loss rate exceeded threshold on a path |
Repeater Health
| Alert | Trigger |
|---|---|
| Clock skew | Exceeded threshold (ref #690) |
| Battery voltage | Dropping below configured level (ref #663) |
| Relay traffic zero | Repeater forwarding nothing (ref #662) |
Operational
| Alert | Trigger |
|---|---|
| Foreign adverts | Unknown network adverts detected (ref #730) |
| Hash collision | New prefix collision introduced |
| Coverage gap | Observer stopped hearing a previously-covered repeater |
Security
| Alert | Trigger |
|---|---|
| Unauthorized signatures | Advert signature validation failure (ref #686) |
| Unexpected node | Node appeared that doesn't match known network |
Delivery Mechanisms
| Channel | Notes |
|---|---|
| Webhook | POST to URL — works with Discord, Slack, IFTTT, custom integrations |
| SMTP configuration | |
| In-app banner | Notification area in CoreScope UI with alert history |
| MQTT publish | Dedicated alerting topic on the MQTT bus |
| Log output | Structured log lines for operators who monitor logs |
Each alert rule maps to one or more delivery channels. Operators configure which alerts go where.
Design Considerations
- Configurable thresholds per alert type (e.g., "stale after 12h" vs "stale after 48h")
- Alert fatigue management — rate limiting, hysteresis (don't fire/resolve/fire/resolve), cooldown periods
- Alert history — what fired, when, auto-resolved or still active, with a UI to browse past alerts
- Silencing/snoozing — maintenance windows, temporary mute per alert or per node
- Per-node overrides — "this repeater is solar and goes offline every winter night, don't alert on it"
- Escalation — if an alert isn't acknowledged within X minutes, re-notify or escalate to a different channel
Milestones
M1: Alert Engine + In-App Notifications
Core alert evaluation loop in the Go backend. Runs on each ingest cycle. Starts with topology alerts (node offline, mesh split) since these require no new data — just existing advert aging and neighbor graph. In-app notification banner + alert history page.
Alerts: Node offline, mesh split/merge, bridge node offline, new node appeared.
M2: Webhook Delivery
Configurable webhook endpoints. JSON payload with alert type, affected node(s), timestamp, severity. Enables Discord/Slack integration immediately — highest operator demand.
M3: RF + Repeater Health Alerts
Noise floor, SNR degradation, observer offline, packet loss, clock skew, battery voltage, relay traffic zero. These require threshold configuration UI in the customizer.
M4: Alert Management UX
Silencing/snoozing UI, per-node overrides, maintenance windows, alert fatigue controls (hysteresis, cooldown). Escalation rules.
M5: Operational + Security Alerts
Foreign adverts, hash collisions, coverage gaps, unauthorized signatures, unexpected nodes. These are lower frequency but high value for security-conscious deployments.
M6: Email + MQTT Delivery
SMTP integration, MQTT alerting topic. These serve operators who don't use webhooks.
Non-Goals (for now)
- SMS/phone call delivery (use webhook → Twilio/IFTTT bridge instead)
- Auto-remediation (reboot nodes, etc.) — alerting only, not control
- Third-party monitoring integration (Prometheus/Grafana export) — separate epic if needed
Related Issues
- epic: neighbor graph analytics — from toy to proper network topology tool #773 (neighbor graph — topology data that enables split detection)
- Feature: Clock skew detection for nodes & observers #690 (clock skew detection)
- feat: battery voltage trend tracking and low-battery alerts #663 (battery voltage tracking)
- feat: repeater liveness — detect if repeater is actively relaying traffic #662 (relay traffic monitoring)
- feat: foreign advert detection — alert when out-of-region nodes appear in local mesh #730 (foreign advert detection)
- BYOP Advert signature validation #686 (advert signature validation)
Activity
Kpa-clawbot commentedon Jun 17, 2026
Severity: P2 — operator-visibility gap (no automated notifications on mesh events).
Claim: confirmed scope. 6-milestone epic, multi-month, cross-cutting (backend engine, delivery, UI, config).
Code today: no alert engine exists; thresholds scattered across
aging.go,liveness.go. No notifications subsystem.Findings:
Fix path: epic, no single fix path. Recommend split: M1+M2 (engine + webhook + silencing) as first PR series, decide on remaining milestones after operator feedback.
Dup check: #730, #686, #690, #663, #662 referenced as related.
Triage verdict: needs-operator-input (milestone prioritization + delivery channel ranking).
efiten commentedon Sep 13, 2026
There is no alert rule or webhook code on master, and the child issues this epic proposed were never filed.
Closing as not planned during backlog triage (see #1922). Nobody has picked this up since it was filed and no PR ever referenced it. Closed issues stay searchable: if you want to build this, reopen it or open a fresh issue with the current state of the code.
feat: mail notifications for watched nodes (user management part E) (#…
efiten commentedon Oct 8, 2026
For the record, the per-user mail part of this epic landed in #2140. A logged-in user watches nodes and gets one mail when a watched node goes offline, comes back or reports a low battery. Admins can add a new foreign node and an observer going offline. The detection is separate from delivery, so another channel such as a webhook can be added later.
The topology, RF and anomaly alerts listed here (mesh split, SNR degradation, noise spikes, clock drift) are not built.