diff --git a/ONBOARDING.md b/ONBOARDING.md index 8e8bff3b..2f5ded14 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -296,5 +296,5 @@ branch, open a PR, get it green, then merge. - [`docs/pipeline.md`](docs/pipeline.md) — detection → tracker → geolocator → aircraft JSON. - [`docs/arc-display.md`](docs/arc-display.md) — how bistatic uncertainty arcs are drawn. - [`docs/runbook.md`](docs/runbook.md) — production operations, server access, incident response. -- [`docs/alerting.md`](docs/alerting.md) — monitoring, alerts, and the dead-man's-switch. +- [`docs/alerting.md`](docs/alerting.md) — monitoring, alerts and the outside-in probes. - [`docs/simulation.md`](docs/simulation.md) — running the fleet simulator. diff --git a/backend/.env.example b/backend/.env.example index 3d903318..7402de03 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -75,8 +75,9 @@ ALERT_WEBHOOK_FORMAT= # Labels each alert's "environment" field (production/staging/test). The deployed # stacks set this in their compose overlay; a bare local run is not expected to. ALERT_ENVIRONMENT= -# Dead-man's-switch: ping this URL periodically (e.g. a Healthchecks.io check). -# The external service alerts when pings stop. See docs/alerting.md. +# Dormant: pings this URL on HEARTBEAT_INTERVAL_S when set, and no environment sets it. +# Outside-in probing is DigitalOcean Uptime's job +# (claude-shared/docs/runbooks/uptime-monitoring.md). HEARTBEAT_URL= # Health-check thresholds. Both fall back to their default on a value that diff --git a/backend/services/tasks/health_monitor.py b/backend/services/tasks/health_monitor.py index 6a7db705..037bcdf4 100644 --- a/backend/services/tasks/health_monitor.py +++ b/backend/services/tasks/health_monitor.py @@ -7,8 +7,9 @@ - emit a "resolved" alert when a previously-firing condition clears. This decouples alerting from the request path: the server alerts on its own -schedule, and a wedged endpoint no longer means silent degradation (the -heartbeat/dead-man's-switch covers a fully-down process — see heartbeat.py). +schedule, and a wedged endpoint no longer means silent degradation (a +fully-down process is caught from outside by the DigitalOcean uptime checks; +see claude-shared/docs/runbooks/uptime-monitoring.md). """ import asyncio diff --git a/backend/services/tasks/heartbeat.py b/backend/services/tasks/heartbeat.py index 307e4063..b4387cf2 100644 --- a/backend/services/tasks/heartbeat.py +++ b/backend/services/tasks/heartbeat.py @@ -1,11 +1,8 @@ -"""Dead-man's-switch heartbeat. - -If HEARTBEAT_URL is set, pings it on a fixed interval. Point it at a free -external check (e.g. Healthchecks.io): the external service alerts when pings -*stop*, which is the one failure mode in-process alerting can't catch — a -crashed process, a dead host, or the disk-full deploy death-spiral. No -infrastructure to run on our side; the SaaS owns the timeout + notification. +"""Optional heartbeat ping. +If HEARTBEAT_URL is set, pings it on a fixed interval. No environment sets +it: outside-in probing is done by DigitalOcean Uptime (see +claude-shared/docs/runbooks/uptime-monitoring.md), so this stays dormant. Disabled (no-op) when HEARTBEAT_URL is unset. """ diff --git a/docs/alerting.md b/docs/alerting.md index 359c3e9c..fcb15727 100644 --- a/docs/alerting.md +++ b/docs/alerting.md @@ -1,8 +1,9 @@ # Alerting & monitoring How RETINA detects problems and notifies operators. The goal is pre-launch -coverage with **no infrastructure we have to run ourselves** — alerting is -in-process plus a free external dead-man's-switch. +coverage with **no infrastructure we have to run ourselves**: alerting is +in-process, plus outside-in probes and notifications from the platforms +already in use (layer 3). ## How it works @@ -90,11 +91,19 @@ Three layers, in order of what they catch: of them keep the `Alert webhook returned ` wording, which is what the droplet logs are grepped for when counting delivery failures. -3. **Dead-man's-switch (external).** `services/tasks/heartbeat.py` pings - `HEARTBEAT_URL` every `HEARTBEAT_INTERVAL_S` (default 60s). Point it at a - free [Healthchecks.io](https://healthchecks.io) check. The external service - alerts when pings **stop** — the one failure mode in-process alerting can't - catch: a crashed process, a dead host, or the disk-full deploy death-spiral. +3. **Outside-in probes (external).** DigitalOcean Uptime polls `/api/health` on `api` + (this backend) and on `towers` (tower-finder-service's own edge) for prod and + staging, every minute from four regions, and emails when every region has seen the + origin down for two minutes. Together with the DigitalOcean resource alerts and + Cloudflare's origin and certificate notifications, this is the layer that catches a + crashed process, a dead host or a broken edge path, which in-process alerting cannot. + What is configured, where to look and how to rebuild it: + `claude-shared/docs/runbooks/uptime-monitoring.md`. + + `services/tasks/heartbeat.py` remains and is dormant: it pings `HEARTBEAT_URL` when + that is set, and nothing sets it. An outside probe covers what a heartbeat would, + except a dead alert loop behind a live HTTP server, which `health_monitor_task`'s + per-cycle exception handling makes unlikely. ## Severity @@ -135,9 +144,9 @@ own history is tracked in ClickUp 86cb81gkn. - Default: always **200**. Body `{"status": "ok"}` or `{"status": "degraded"}`. Used as the Docker container **liveness** check — it must not flip to non-200 on transient degradation or the container would restart-loop. -- `?strict=1`: **readiness** probe — returns **503** when degraded. Point an - external uptime monitor (UptimeRobot/BetterStack free tier) at this for an - independent outside-in alert. +- `?strict=1`: **readiness** probe — returns **503** when degraded. Not what the outside + probes use: they assert liveness, because the strict form also trips on warnings that + are still being calibrated (ClickUp 86cb5c8dq). Details are intentionally **not** exposed on this unauthenticated endpoint — they're in the logs and the webhook payloads. @@ -151,10 +160,9 @@ they're in the logs and the webhook payloads. to it, set `ALERT_WEBHOOK_FORMAT=clickup_chat`, and set `ALERT_WEBHOOK_URL` to `https://api.clickup.com/api/v3/workspaces/{workspace_id}/chat/channels/{channel_id}/messages`. -2. Create a free Healthchecks.io check (period 1m, grace ~2m) → set - `HEARTBEAT_URL` to its ping URL. Configure its notification channel. -3. (Optional) Add an UptimeRobot/BetterStack monitor on - `https:///api/health?strict=1`. +2. Outside-in probes, resource alerts and Cloudflare notifications are account-level + configuration, not environment variables: see + `claude-shared/docs/runbooks/uptime-monitoring.md`. | Env var | Default | Purpose | | --- | --- | --- | @@ -163,7 +171,7 @@ they're in the logs and the webhook payloads. | `ALERT_WEBHOOK_AUTH` | _(unset)_ | Sent verbatim as the `Authorization` header when set | | `ALERT_WEBHOOK_FORMAT` | `raw` | Payload shape: `raw` or `clickup_chat` | | `ALERT_ENVIRONMENT` | _(unset → `unknown`)_ | Labels each alert's `environment` field | -| `HEARTBEAT_URL` | _(unset → disabled)_ | External dead-man's-switch ping target | +| `HEARTBEAT_URL` | _(unset → disabled)_ | Dormant: set by no environment (see layer 3) | | `HEARTBEAT_INTERVAL_S` | `60` | Heartbeat ping period | | `HEALTH_MONITOR_INTERVAL_S` | `30` | Health evaluation period | | `NODE_DROPOUT_THRESHOLD` | `0.8` | Active/peak node ratio below which dropout fires | @@ -171,7 +179,8 @@ they're in the logs and the webhook payloads. ## Deferred (needs real infrastructure) -Metrics history and dashboards (Prometheus + Grafana, Loki for logs, Sentry for -exceptions) are **not** required for launch — the webhook + heartbeat cover -"something is wrong, tell a human." Add them later if you want trend graphs or -exception aggregation; they require standing up and maintaining services. +Metrics history and dashboards (VictoriaMetrics + Grafana, Loki for logs, Sentry for +exceptions) are the third monitoring sub-project in +`claude-shared/docs/decisions/2026-09-15-uptime-monitoring.md` and are not yet ticketed. +The admin dashboard's Infrastructure page shows DigitalOcean's uptime and droplet metrics +meanwhile. diff --git a/docs/runbook.md b/docs/runbook.md index 85ae9b59..f33db861 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -250,8 +250,9 @@ the server's own schedule, independent of who polls `/api/health` — see [`alerting.md`](alerting.md). `/api/health` itself stays **200** (liveness, used by the Docker healthcheck); -`/api/health?strict=1` returns **503** when degraded (readiness, for an external -uptime monitor). Details are never exposed on the endpoint — read them from logs: +`/api/health?strict=1` returns **503** when degraded (readiness; the outside-in +probes use the plain form, see `claude-shared/docs/runbooks/uptime-monitoring.md`). +Details are never exposed on the endpoint — read them from logs: ```bash docker compose logs --tail=200 | grep "Health check degraded"