bug(observability): time-to-ready is never recorded, so the deploy timeout is set from anecdote #196

Time-to-ready is the number the staging deploy gate is set against, and nothing records it. Every figure we have was produced by hand, once, because someone asked.

Evidence that this is a real gap, not a nicety

Three measurements of the same thing, all hand-collected over ssh, none stored:

The two dev numbers are the same box with the same config a factor of two apart, because the dataset moved: 903,979 observations on 2026-10-06 against 771,294 on 2026-10-11 (dev expires packets at 7 days, so its load shrinks and grows). A single sample cannot answer "what timeout is safe"; the spread can, and the spread is not kept.

The consequence already happened. In #195 and on PR #194 the 90s gate was replaced with 300s, and the review input offered for that decision was a five-day-old one-off that was wrong by 2x in the pessimistic direction, plus a prod figure from a different box that did not apply at all. The timeout is being set from anecdote.

The data exists and is discarded twice

1. The server knows exactly when it became ready and never says so. readiness is an atomic.Int32 [repo: cmd/server/healthz.go:9-11] that flips once background init completes (store.Load, pickBestObservation, neighbor-graph build). /api/healthz reports whether it is ready and never when it became ready or how long it took. There is uptime elsewhere [repo: cmd/server/types.go:211] but no startupMs / readyAt anywhere.

2. CI computes it on every staging deploy and throws it away. The wait loop in PR #194 tracks elapsed and prints ready after ${elapsed}s to the job log. That is the measurement, generated on every deploy, landing in a log that nobody aggregates and that ages out.

Proposed

Small, and in the two places the number already passes through:

1. Expose it. Add elapsed-to-ready to /api/healthz (e.g. startupMs, plus readyAt), set when readiness flips. Cheap, and it makes the figure available to anything that polls — including the deploy gate that already does.
2. Record it. Have the deploy step emit the value somewhere durable rather than only printing it: a workflow summary line, a step output, or a metric. Enough to build a distribution over time.

Then vars.HEALTH_TIMEOUT becomes a number set from data and revisited when the data moves, instead of a guess defended with whatever the last person remembers.

Worth deciding alongside, not assumed

• Whether this belongs in the existing monitoring work rather than standing alone — there is Netdata/Uptime-Kuma on these boxes, and a startup-duration metric may have a natural home there.
• Whether prod should be tracked too. Prod's ~260s is not comparable to dev's and the two should not be conflated (that mistake was made in this thread), but prod's own trend matters for the deploy-prod window, which fix(deploy-prod): healthcheck window too short for prod cold-start on 8GB DB #127/fix(deploy-prod): widen healthcheck window for prod 8GB cold-start (#127) #128 set to 300s with auto-rollback.

Related

#181 is why dev's start is memory-pressured at all (mem_limit: 2g with GOMEMLIMIT unset); if it lands, these numbers change and the lack of tracking is exactly why nobody would notice.

Refs #195, #194, #181, #127, #128