feat: expose graph-node stats and health-check it

[creatornader]

Closes the observability gap the 2026-07-29 outage exposed. Follows #599 (the fix) and #601 (canary ordering + heap watchdog).

The gap

graph-node had no health, stats, or metrics endpoint, and no Fly health check — [[http_service.checks]] was absent and flyctl status returned an empty CHECKS column. During the crash loop Fly had no liveness signal whatsoever: it kept routing to a dead instance and returning 502s. The only thing that noticed was a post-deploy canary in a different job.

Meanwhile the one number that would have predicted the outage — heap headroom — lived only in Fly logs.

1. GET /v1/stats

{"service":"atrib-graph-node","record_count":120000,"context_count":900,
 "heap":{"used_mb":360,"limit_mb":828,"used_pct":43.5},
 "uptime_s":23,"replay":{"ms":5462,"records":120000}}

Shape mirrors log-node's /v1/stats. Deliberately not in the spec — I checked, and /v1/stats appears nowhere in atrib-spec.md, so log-node's is operational rather than normative too. §3.4 is the graph query interface; this must never start returning graph facts (§3.6 fact/policy separation).

replay is omitted, not null, when no archive is configured, so callers can distinguish "no archive" from "archive replayed zero records".

Kept O(1) and there's a test asserting it does not scan the store. That's a real constraint, not a nicety: the Fly check polls it on an interval, so if it ever walked the store it would become a periodic full scan of the thing it exists to protect — close to the original bug.

2. Fly health check

timeout = 10s, far above the ~0.15s a healthy response takes, because the failure worth catching is a wedged process rather than a slow one. During the outage requests took 17–25s while the process was still listening and spending 65–75% of wall-clock in GC.

grace_period = 60s because Fly clamps it there. flyctl config validate on my first attempt returned:

Service HTTP check has a grace period greater than 1 minute (2m0s); this will be lowered to 1 minute

I had written 120s. It would have silently become 60s — protection I thought I had and didn't.

That clamp bounds the design, not just this file. main.ts replays the whole archive before binding the port, so nothing listens until replay finishes. At the measured ~20.6k records/sec, 60s of grace covers ~1.24M records, and machine sizing hits that before it hits a memory wall:

machine	heap	capacity @85%	replay
1GB	768MB	214k	~10s
2GB	1536MB	427k	~21s
4GB	3072MB	854k	~41s
8GB	6144MB	1.7M	~83s — past the 60s cap

So "just keep raising memory" expires somewhere past the 4GB tier for startup reasons, not heap reasons. Added to P059.

3. check-graph-smoke.mjs in the graph-canary job

Runs before the canary: no dependencies, and it fails in seconds with a specific message where the canary would burn its full 90s deadline discovering the same thing less clearly.

Kept separate from check-log-smoke.mjs on purpose — folding graph checks into it would re-couple a log-node deploy to graph-node's health, the exact coupling #601 removed.

Fails on contract violations, on a slow response, and on heap ≥95%. That threshold is high enough not to block the very deploy that fixes a memory problem, and GRAPH_SMOKE_FAIL_HEAP_PCT overrides it for remediation.

Verification

Every path exercised against a real graph-node at production scale (120k records under a 780MB ceiling — 360MB/828MB @ 43.5%, 3.07 KB/record, versus production's 345/780 @ 44.2%, 3.13 KB/record):

case	exit
happy path	0
heap over fail threshold	1
latency budget exceeded	1
unreachable origin	1
non-JSON body	1
record_count: 0 (archive did not replay)	1
invalid env value	1

Exit codes captured directly, not through a pipe — tail masked them on my first pass.

One thing I fixed mid-verification: the first version reported per record 59.39 KB on a 500-record store, because dividing total heap by record count attributes Node's ~25MB baseline to records. Runway is now reported in MB always, and in records only above 10k where the baseline is noise.

• services/graph-node: 132/132 tests (124 before, +8 for /v1/stats).
• flyctl config validate: valid, no warnings.
• pnpm doc-sync: 14 checks pass.

Not done here

Only graph-node gets a health check. log-node has /v1/stats and could take one, but its startup profile is different and I haven't measured it, so setting a grace period there would be guessing. directory-node and archive-node have no stats endpoint at all. Worth a follow-up; I didn't want to ship configuration I couldn't justify with a measurement.