P2: Operations runbook — SLOs, capacity, incidents, failover (#180)

[dkijania]

What & why

Part of the production-readiness epic (#163). Closes #180.

There was one benchmark data point but no runbook, SLOs, or documented failure-mode response.

Adds docs/runbook.md

• SLOs — starting targets for availability, p50/p99 latency, error rate.
• What to watch — the /metrics RED signals, readiness, structured logs, and suggested alert thresholds.
• Scaling & capacity — stateless horizontal scaling, Postgres as the real ceiling, and replicas × PG_MAX_CONNECTIONS vs DB max_connections math.
• Multi-host Postgres failover — documented semantics and recovery expectations (with an honest "validate for your topology" caveat).
• Common incidents — a symptom → cause → action table.
• Deploys & rollback — tied to graceful shutdown + readiness gating.

Linked from the README. References observability/config features delivered by the sibling PRs. An automated replica-failover test is noted as a follow-up (needs a multi-host DB harness), so the doc is the deliverable here.

Testing

Docs only. prettier --debug-check . clean. No application code changed.

🤖 Generated with Claude Code

[SanabriaRusso]

Great capstone for the epic — the symptom→cause→action table and the honest "validate for your topology" failover caveat are exactly what on-call wants. One correctness heads-up before merge: almost everything the runbook tells operators to observe or tune ships in sibling PRs that aren't on main yet, and the doc presents them as current with no status note. On today's main:

• /metrics + every http_* metric → don't exist (P1: Expose Prometheus metrics at /metrics (#173) #191)
• /readiness — the 503-vs-200 signal the incident table and most alerts lean on → doesn't exist; server only serves / and /healthcheck (P1: Add readiness probe distinct from liveness (#169) #187)
• PG_STATEMENT_TIMEOUT / PG_MAX_CONNECTIONS → not wired; the client is a bare postgres(connectionString) with no pool/timeout options (P0: Configure Postgres pool limits + statement_timeout (#165) #182)
• RATE_LIMIT_MAX / X-Forwarded-For handling → not present (P0: Add per-IP request rate limiting (#166) #185)
• graceful drain on SIGTERM + terminationGracePeriodSeconds + trace flush → src/index.ts currently does server.close() then an immediate process.exit(0), no drain (P1: Graceful shutdown — drain, flush traces, uncaught handlers (#170) #188)
• the docs/security.md (P0: Add security & deployment hardening guide (#168) #186) and deploy/kubernetes.yaml (P2: Reference deployment manifests — k8s + prod Compose (#179) #196) links → those files aren't on main yet, so both 404 today

Docs PRs tend to merge ahead of feature PRs, and if this one lands first the runbook misdirects on-call mid-incident (curling /readiness → 404, "confirm PG_STATEMENT_TIMEOUT is set" → no-op, relying on a drain that doesn't happen). Two clean options: gate this merge behind those PRs, or add a short status banner up top, e.g.

Status: Some capabilities below ship in in-flight PRs and are not yet on main: /metrics (#191), /readiness (#187), Postgres pool/timeout knobs (#182), rate limiting (#185), graceful shutdown (#188), config validation (#193), plus the linked deploy/ manifests (#196) and docs/security.md (#186). Sections depending on them are marked (pending).

One smaller note on Scaling: "Add read replicas and point PG_CONN at them before scaling the API further" reads as added read capacity, but the postgres client treats multi-host PG_CONN as failover to the first reachable host, not load-balanced fan-out — so extra hosts buy redundancy, not throughput. Worth a word so operators don't expect horizontal DB read scaling from it. (Your dedicated Failover section already gets this exactly right.)

Nice work overall — just want the doc to be safe to follow the day it merges.

[dkijania]

Thanks @SanabriaRusso — fixed in b6dc43a.

The "describes unshipped behavior" problem. Agreed this was the important one: a runbook that misdirects on-call mid-incident is worse than no runbook. Of your two options I took a third — scoping by version rather than by in-flight PR numbers:

Applies to 1.0.0 and later. Much of what follows — /readiness, /metrics, PG_STATEMENT_TIMEOUT / PG_MAX_CONNECTIONS, RATE_LIMIT_MAX, and the graceful drain on SIGTERM — does not exist on 0.0.x images…

Reasoning: the merge plan already lands this doc after #182/#185/#187/#188/#191, so a PR-status banner would be stale on arrival and need deleting. The version framing stays true afterwards and is arguably the more useful warning anyway — the published image today is 0.0.6, so "is this on main?" is the wrong question for an operator mid-incident; "what am I actually running?" is the right one. Same approach applied in #186 and #196.

The Scaling claim — you were right, and I can now say why. I checked the driver rather than take it on faith: postgres.js declares hostIndex inside function Connection(...) (src/connection.js:89), so it's per-connection state reset to 0 for every pooled connection. Every connection starts at host[0] and only advances when that connection's attempt fails. Failover, not fan-out — extra hosts buy redundancy, exactly as you said. Both the Scaling bullet and the failover section now say so, and point at a balancer (PgBouncer/HAProxy/managed reader endpoint) for real read capacity.

Worth flagging that this same claim was already on main's README ("the server fans queries across them") — so it wasn't just this doc. Fixed there via #186, and in deploy/README.md via #196.

One addition from the #185 fix: I split the Many 429s incident row. Now that TRUST_PROXY bounds X-Forwarded-For trust, the most likely cause of mass 429s across unrelated clients is TRUST_PROXY being unset behind a gateway — every client collapses into one bucket. That's a different diagnosis and fix from a single client exceeding the limit, so it gets its own row (and the app now warns about it at startup).