Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ONBOARDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,5 +296,5 @@ branch, open a PR, get it green, then merge.
- [`docs/pipeline.md`](docs/pipeline.md) — detection → tracker → geolocator → aircraft JSON.
- [`docs/arc-display.md`](docs/arc-display.md) — how bistatic uncertainty arcs are drawn.
- [`docs/runbook.md`](docs/runbook.md) — production operations, server access, incident response.
- [`docs/alerting.md`](docs/alerting.md) — monitoring, alerts, and the dead-man's-switch.
- [`docs/alerting.md`](docs/alerting.md) — monitoring, alerts and the outside-in probes.
- [`docs/simulation.md`](docs/simulation.md) — running the fleet simulator.
5 changes: 3 additions & 2 deletions backend/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,9 @@ ALERT_WEBHOOK_FORMAT=
# Labels each alert's "environment" field (production/staging/test). The deployed
# stacks set this in their compose overlay; a bare local run is not expected to.
ALERT_ENVIRONMENT=
# Dead-man's-switch: ping this URL periodically (e.g. a Healthchecks.io check).
# The external service alerts when pings stop. See docs/alerting.md.
# Dormant: pings this URL on HEARTBEAT_INTERVAL_S when set, and no environment sets it.
# Outside-in probing is DigitalOcean Uptime's job
# (claude-shared/docs/runbooks/uptime-monitoring.md).
HEARTBEAT_URL=

# Health-check thresholds. Both fall back to their default on a value that
Expand Down
5 changes: 3 additions & 2 deletions backend/services/tasks/health_monitor.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@
- emit a "resolved" alert when a previously-firing condition clears.

This decouples alerting from the request path: the server alerts on its own
schedule, and a wedged endpoint no longer means silent degradation (the
heartbeat/dead-man's-switch covers a fully-down process — see heartbeat.py).
schedule, and a wedged endpoint no longer means silent degradation (a
fully-down process is caught from outside by the DigitalOcean uptime checks;
see claude-shared/docs/runbooks/uptime-monitoring.md).
"""

import asyncio
Expand Down
11 changes: 4 additions & 7 deletions backend/services/tasks/heartbeat.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
"""Dead-man's-switch heartbeat.

If HEARTBEAT_URL is set, pings it on a fixed interval. Point it at a free
external check (e.g. Healthchecks.io): the external service alerts when pings
*stop*, which is the one failure mode in-process alerting can't catch — a
crashed process, a dead host, or the disk-full deploy death-spiral. No
infrastructure to run on our side; the SaaS owns the timeout + notification.
"""Optional heartbeat ping.

If HEARTBEAT_URL is set, pings it on a fixed interval. No environment sets
it: outside-in probing is done by DigitalOcean Uptime (see
claude-shared/docs/runbooks/uptime-monitoring.md), so this stays dormant.
Disabled (no-op) when HEARTBEAT_URL is unset.
"""

Expand Down
47 changes: 28 additions & 19 deletions docs/alerting.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Alerting & monitoring

How RETINA detects problems and notifies operators. The goal is pre-launch
coverage with **no infrastructure we have to run ourselves** — alerting is
in-process plus a free external dead-man's-switch.
coverage with **no infrastructure we have to run ourselves**: alerting is
in-process, plus outside-in probes and notifications from the platforms
already in use (layer 3).

## How it works

Expand Down Expand Up @@ -90,11 +91,19 @@ Three layers, in order of what they catch:
of them keep the `Alert webhook returned <code>` wording, which is what
the droplet logs are grepped for when counting delivery failures.

3. **Dead-man's-switch (external).** `services/tasks/heartbeat.py` pings
`HEARTBEAT_URL` every `HEARTBEAT_INTERVAL_S` (default 60s). Point it at a
free [Healthchecks.io](https://healthchecks.io) check. The external service
alerts when pings **stop** — the one failure mode in-process alerting can't
catch: a crashed process, a dead host, or the disk-full deploy death-spiral.
3. **Outside-in probes (external).** DigitalOcean Uptime polls `/api/health` on `api`
(this backend) and on `towers` (tower-finder-service's own edge) for prod and
staging, every minute from four regions, and emails when every region has seen the
origin down for two minutes. Together with the DigitalOcean resource alerts and
Cloudflare's origin and certificate notifications, this is the layer that catches a
crashed process, a dead host or a broken edge path, which in-process alerting cannot.
What is configured, where to look and how to rebuild it:
`claude-shared/docs/runbooks/uptime-monitoring.md`.

`services/tasks/heartbeat.py` remains and is dormant: it pings `HEARTBEAT_URL` when
that is set, and nothing sets it. An outside probe covers what a heartbeat would,
except a dead alert loop behind a live HTTP server, which `health_monitor_task`'s
per-cycle exception handling makes unlikely.

## Severity

Expand Down Expand Up @@ -135,9 +144,9 @@ own history is tracked in ClickUp 86cb81gkn.
- Default: always **200**. Body `{"status": "ok"}` or `{"status":
"degraded"}`. Used as the Docker container **liveness** check — it must not
flip to non-200 on transient degradation or the container would restart-loop.
- `?strict=1`: **readiness** probe — returns **503** when degraded. Point an
external uptime monitor (UptimeRobot/BetterStack free tier) at this for an
independent outside-in alert.
- `?strict=1`: **readiness** probe — returns **503** when degraded. Not what the outside
probes use: they assert liveness, because the strict form also trips on warnings that
are still being calibrated (ClickUp 86cb5c8dq).

Details are intentionally **not** exposed on this unauthenticated endpoint —
they're in the logs and the webhook payloads.
Expand All @@ -151,10 +160,9 @@ they're in the logs and the webhook payloads.
to it, set `ALERT_WEBHOOK_FORMAT=clickup_chat`, and set
`ALERT_WEBHOOK_URL` to
`https://api.clickup.com/api/v3/workspaces/{workspace_id}/chat/channels/{channel_id}/messages`.
2. Create a free Healthchecks.io check (period 1m, grace ~2m) → set
`HEARTBEAT_URL` to its ping URL. Configure its notification channel.
3. (Optional) Add an UptimeRobot/BetterStack monitor on
`https://<host>/api/health?strict=1`.
2. Outside-in probes, resource alerts and Cloudflare notifications are account-level
configuration, not environment variables: see
`claude-shared/docs/runbooks/uptime-monitoring.md`.

| Env var | Default | Purpose |
| --- | --- | --- |
Expand All @@ -163,15 +171,16 @@ they're in the logs and the webhook payloads.
| `ALERT_WEBHOOK_AUTH` | _(unset)_ | Sent verbatim as the `Authorization` header when set |
| `ALERT_WEBHOOK_FORMAT` | `raw` | Payload shape: `raw` or `clickup_chat` |
| `ALERT_ENVIRONMENT` | _(unset → `unknown`)_ | Labels each alert's `environment` field |
| `HEARTBEAT_URL` | _(unset → disabled)_ | External dead-man's-switch ping target |
| `HEARTBEAT_URL` | _(unset → disabled)_ | Dormant: set by no environment (see layer 3) |
| `HEARTBEAT_INTERVAL_S` | `60` | Heartbeat ping period |
| `HEALTH_MONITOR_INTERVAL_S` | `30` | Health evaluation period |
| `NODE_DROPOUT_THRESHOLD` | `0.8` | Active/peak node ratio below which dropout fires |
| `HIGH_MISS_RATE_THRESHOLD` | `0.98` | Fleet-average miss rate above which `high_miss_rate` fires |

## Deferred (needs real infrastructure)

Metrics history and dashboards (Prometheus + Grafana, Loki for logs, Sentry for
exceptions) are **not** required for launch — the webhook + heartbeat cover
"something is wrong, tell a human." Add them later if you want trend graphs or
exception aggregation; they require standing up and maintaining services.
Metrics history and dashboards (VictoriaMetrics + Grafana, Loki for logs, Sentry for
exceptions) are the third monitoring sub-project in
`claude-shared/docs/decisions/2026-09-15-uptime-monitoring.md` and are not yet ticketed.
The admin dashboard's Infrastructure page shows DigitalOcean's uptime and droplet metrics
meanwhile.
5 changes: 3 additions & 2 deletions docs/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,8 +250,9 @@ the server's own schedule, independent of who polls `/api/health` — see
[`alerting.md`](alerting.md).

`/api/health` itself stays **200** (liveness, used by the Docker healthcheck);
`/api/health?strict=1` returns **503** when degraded (readiness, for an external
uptime monitor). Details are never exposed on the endpoint — read them from logs:
`/api/health?strict=1` returns **503** when degraded (readiness; the outside-in
probes use the plain form, see `claude-shared/docs/runbooks/uptime-monitoring.md`).
Details are never exposed on the endpoint — read them from logs:

```bash
docker compose logs --tail=200 | grep "Health check degraded"
Expand Down
Loading