From de200a9691fe16e9e5bdef93948b37a7c463b2eb Mon Sep 17 00:00:00 2001 From: Babissimo Date: Tue, 15 Sep 2026 13:41:17 +0100 Subject: [PATCH] Point the alerting docs at the uptime runbook docs/alerting.md described a dead-man's switch nobody had wired and told readers to add an UptimeRobot or Healthchecks.io monitor as an optional extra. The outside-in layer now exists, as DigitalOcean Uptime checks and notifications recorded in claude-shared, so the doc names that instead, marks the heartbeat task as dormant rather than implying it runs, and stops pointing an external probe at the strict readiness form, which trips on warnings that are still being calibrated. ClickUp 123zgec2z7a. Co-Authored-By: Claude Sonnet 5 --- ONBOARDING.md | 2 +- backend/.env.example | 5 ++- backend/services/tasks/health_monitor.py | 5 ++- backend/services/tasks/heartbeat.py | 11 ++---- docs/alerting.md | 47 ++++++++++++++---------- docs/runbook.md | 5 ++- 6 files changed, 42 insertions(+), 33 deletions(-) diff --git a/ONBOARDING.md b/ONBOARDING.md index 8e8bff3b..2f5ded14 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -296,5 +296,5 @@ branch, open a PR, get it green, then merge. - [`docs/pipeline.md`](docs/pipeline.md) — detection → tracker → geolocator → aircraft JSON. - [`docs/arc-display.md`](docs/arc-display.md) — how bistatic uncertainty arcs are drawn. - [`docs/runbook.md`](docs/runbook.md) — production operations, server access, incident response. -- [`docs/alerting.md`](docs/alerting.md) — monitoring, alerts, and the dead-man's-switch. +- [`docs/alerting.md`](docs/alerting.md) — monitoring, alerts and the outside-in probes. - [`docs/simulation.md`](docs/simulation.md) — running the fleet simulator. diff --git a/backend/.env.example b/backend/.env.example index 3d903318..7402de03 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -75,8 +75,9 @@ ALERT_WEBHOOK_FORMAT= # Labels each alert's "environment" field (production/staging/test). The deployed # stacks set this in their compose overlay; a bare local run is not expected to. ALERT_ENVIRONMENT= -# Dead-man's-switch: ping this URL periodically (e.g. a Healthchecks.io check). -# The external service alerts when pings stop. See docs/alerting.md. +# Dormant: pings this URL on HEARTBEAT_INTERVAL_S when set, and no environment sets it. +# Outside-in probing is DigitalOcean Uptime's job +# (claude-shared/docs/runbooks/uptime-monitoring.md). HEARTBEAT_URL= # Health-check thresholds. Both fall back to their default on a value that diff --git a/backend/services/tasks/health_monitor.py b/backend/services/tasks/health_monitor.py index 6a7db705..037bcdf4 100644 --- a/backend/services/tasks/health_monitor.py +++ b/backend/services/tasks/health_monitor.py @@ -7,8 +7,9 @@ - emit a "resolved" alert when a previously-firing condition clears. This decouples alerting from the request path: the server alerts on its own -schedule, and a wedged endpoint no longer means silent degradation (the -heartbeat/dead-man's-switch covers a fully-down process — see heartbeat.py). +schedule, and a wedged endpoint no longer means silent degradation (a +fully-down process is caught from outside by the DigitalOcean uptime checks; +see claude-shared/docs/runbooks/uptime-monitoring.md). """ import asyncio diff --git a/backend/services/tasks/heartbeat.py b/backend/services/tasks/heartbeat.py index 307e4063..b4387cf2 100644 --- a/backend/services/tasks/heartbeat.py +++ b/backend/services/tasks/heartbeat.py @@ -1,11 +1,8 @@ -"""Dead-man's-switch heartbeat. - -If HEARTBEAT_URL is set, pings it on a fixed interval. Point it at a free -external check (e.g. Healthchecks.io): the external service alerts when pings -*stop*, which is the one failure mode in-process alerting can't catch — a -crashed process, a dead host, or the disk-full deploy death-spiral. No -infrastructure to run on our side; the SaaS owns the timeout + notification. +"""Optional heartbeat ping. +If HEARTBEAT_URL is set, pings it on a fixed interval. No environment sets +it: outside-in probing is done by DigitalOcean Uptime (see +claude-shared/docs/runbooks/uptime-monitoring.md), so this stays dormant. Disabled (no-op) when HEARTBEAT_URL is unset. """ diff --git a/docs/alerting.md b/docs/alerting.md index 359c3e9c..fcb15727 100644 --- a/docs/alerting.md +++ b/docs/alerting.md @@ -1,8 +1,9 @@ # Alerting & monitoring How RETINA detects problems and notifies operators. The goal is pre-launch -coverage with **no infrastructure we have to run ourselves** — alerting is -in-process plus a free external dead-man's-switch. +coverage with **no infrastructure we have to run ourselves**: alerting is +in-process, plus outside-in probes and notifications from the platforms +already in use (layer 3). ## How it works @@ -90,11 +91,19 @@ Three layers, in order of what they catch: of them keep the `Alert webhook returned ` wording, which is what the droplet logs are grepped for when counting delivery failures. -3. **Dead-man's-switch (external).** `services/tasks/heartbeat.py` pings - `HEARTBEAT_URL` every `HEARTBEAT_INTERVAL_S` (default 60s). Point it at a - free [Healthchecks.io](https://healthchecks.io) check. The external service - alerts when pings **stop** — the one failure mode in-process alerting can't - catch: a crashed process, a dead host, or the disk-full deploy death-spiral. +3. **Outside-in probes (external).** DigitalOcean Uptime polls `/api/health` on `api` + (this backend) and on `towers` (tower-finder-service's own edge) for prod and + staging, every minute from four regions, and emails when every region has seen the + origin down for two minutes. Together with the DigitalOcean resource alerts and + Cloudflare's origin and certificate notifications, this is the layer that catches a + crashed process, a dead host or a broken edge path, which in-process alerting cannot. + What is configured, where to look and how to rebuild it: + `claude-shared/docs/runbooks/uptime-monitoring.md`. + + `services/tasks/heartbeat.py` remains and is dormant: it pings `HEARTBEAT_URL` when + that is set, and nothing sets it. An outside probe covers what a heartbeat would, + except a dead alert loop behind a live HTTP server, which `health_monitor_task`'s + per-cycle exception handling makes unlikely. ## Severity @@ -135,9 +144,9 @@ own history is tracked in ClickUp 86cb81gkn. - Default: always **200**. Body `{"status": "ok"}` or `{"status": "degraded"}`. Used as the Docker container **liveness** check — it must not flip to non-200 on transient degradation or the container would restart-loop. -- `?strict=1`: **readiness** probe — returns **503** when degraded. Point an - external uptime monitor (UptimeRobot/BetterStack free tier) at this for an - independent outside-in alert. +- `?strict=1`: **readiness** probe — returns **503** when degraded. Not what the outside + probes use: they assert liveness, because the strict form also trips on warnings that + are still being calibrated (ClickUp 86cb5c8dq). Details are intentionally **not** exposed on this unauthenticated endpoint — they're in the logs and the webhook payloads. @@ -151,10 +160,9 @@ they're in the logs and the webhook payloads. to it, set `ALERT_WEBHOOK_FORMAT=clickup_chat`, and set `ALERT_WEBHOOK_URL` to `https://api.clickup.com/api/v3/workspaces/{workspace_id}/chat/channels/{channel_id}/messages`. -2. Create a free Healthchecks.io check (period 1m, grace ~2m) → set - `HEARTBEAT_URL` to its ping URL. Configure its notification channel. -3. (Optional) Add an UptimeRobot/BetterStack monitor on - `https:///api/health?strict=1`. +2. Outside-in probes, resource alerts and Cloudflare notifications are account-level + configuration, not environment variables: see + `claude-shared/docs/runbooks/uptime-monitoring.md`. | Env var | Default | Purpose | | --- | --- | --- | @@ -163,7 +171,7 @@ they're in the logs and the webhook payloads. | `ALERT_WEBHOOK_AUTH` | _(unset)_ | Sent verbatim as the `Authorization` header when set | | `ALERT_WEBHOOK_FORMAT` | `raw` | Payload shape: `raw` or `clickup_chat` | | `ALERT_ENVIRONMENT` | _(unset → `unknown`)_ | Labels each alert's `environment` field | -| `HEARTBEAT_URL` | _(unset → disabled)_ | External dead-man's-switch ping target | +| `HEARTBEAT_URL` | _(unset → disabled)_ | Dormant: set by no environment (see layer 3) | | `HEARTBEAT_INTERVAL_S` | `60` | Heartbeat ping period | | `HEALTH_MONITOR_INTERVAL_S` | `30` | Health evaluation period | | `NODE_DROPOUT_THRESHOLD` | `0.8` | Active/peak node ratio below which dropout fires | @@ -171,7 +179,8 @@ they're in the logs and the webhook payloads. ## Deferred (needs real infrastructure) -Metrics history and dashboards (Prometheus + Grafana, Loki for logs, Sentry for -exceptions) are **not** required for launch — the webhook + heartbeat cover -"something is wrong, tell a human." Add them later if you want trend graphs or -exception aggregation; they require standing up and maintaining services. +Metrics history and dashboards (VictoriaMetrics + Grafana, Loki for logs, Sentry for +exceptions) are the third monitoring sub-project in +`claude-shared/docs/decisions/2026-09-15-uptime-monitoring.md` and are not yet ticketed. +The admin dashboard's Infrastructure page shows DigitalOcean's uptime and droplet metrics +meanwhile. diff --git a/docs/runbook.md b/docs/runbook.md index 85ae9b59..f33db861 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -250,8 +250,9 @@ the server's own schedule, independent of who polls `/api/health` — see [`alerting.md`](alerting.md). `/api/health` itself stays **200** (liveness, used by the Docker healthcheck); -`/api/health?strict=1` returns **503** when degraded (readiness, for an external -uptime monitor). Details are never exposed on the endpoint — read them from logs: +`/api/health?strict=1` returns **503** when degraded (readiness; the outside-in +probes use the plain form, see `claude-shared/docs/runbooks/uptime-monitoring.md`). +Details are never exposed on the endpoint — read them from logs: ```bash docker compose logs --tail=200 | grep "Health check degraded"