Skip to content

chore(release): 0.9.0 - health timings, instance id, bounded facts - #19

Merged
vreshch merged 1 commit into
masterfrom
feature/health-timing
Aug 10, 2026
Merged

chore(release): 0.9.0 - health timings, instance id, bounded facts#19
vreshch merged 1 commit into
masterfrom
feature/health-timing

Conversation

@vreshch

@vreshch vreshch commented Aug 10, 2026

Copy link
Copy Markdown
Member

Why

A health response says "db": "ok" and nothing else. That word cannot tell you whether the check ran a real query or returned a memoized value that has been reporting ok since the dependency died, whether the payload was computed just now or served from a cache, or whether the 197ms the admin console reports is the service or the edge.

Four additive fields make the envelope interpretable, and one bug fix closes a latent restart-loop found while adding them.

What

Field Answers
timings Per check, plus facts. Near 0 is memoized; milliseconds is a real round trip.
durationMs Total server-side cost, so a probe subtracts it and attributes the rest to the wire.
checkedAt The cache detector. Frozen across two probes means something is serving a copy.
instance Random per process. Changing between probes means a different replica, not a cache and not a restart.
reasons Only when a check is not ok. A timeout and an instant ECONNREFUSED both read down alone.
{
  "status": "degraded", "service": "catalog-backend", "instance": "b82fc8d3",
  "checkedAt": "2026-08-10T01:04:40.586Z", "durationMs": 60.9,
  "checks":  { "db": "ok", "cache": "ok", "search": "degraded" },
  "timings": { "db": 21.5, "cache": 0.1, "search": 60.7, "facts": 3.2 },
  "reasons": { "search": "timed out after 60ms" },
  "facts":   { "servers": 20804 }
}

cache: 0.1 next to db: 21.5 is the whole point: one of those checks measured something.

Same numbers go out as Server-Timing: health;dur=60.9, db;dur=21.5, ..., so the split shows in devtools without parsing the body.

Bug fixed: the facts producer had no timeout

resolveHealth raced every check against a timer and left the facts producer unbounded. A fact off a wedged database outlives the container HEALTHCHECK --timeout=5s, three retries, Swarm kills a task that was only ever slow to count rows. Same shape as the store-mount rollback. It has not bitten because catalog-backend, the only service returning a DB-backed fact, hand-rolls its own withTimeout - the author had to supply the guarantee the kit should own. Now bounded by factsTimeoutMs (1s default).

Also

startedAt derives from process.uptime() instead of module load, so a lazily-imported Next route handler stops reporting a fresh uptime for a container that has been up for hours.

Findings that motivated this

  • catalog-web runs at least two replicas and the admin console probes one at random: eight samples of catalog.agentage.io/health returned two distinct startedAt values, interleaved. A single dead replica would flicker green/red across polls rather than reading red. instance makes that visible; a follow-up in mcp-catalog decides whether to probe both or drop to one.
  • Nothing in the estate is cached today - cache-control: no-store verified on all sampled endpoints. checkedAt is there so that stays true observably rather than by assumption.

Compatibility

Additive only. checks stays a flat Record<string, CheckState>, so admin's derive.ts, the e2e smokes and the deploy gates keep working untouched - a nested {state, ms} would have broken all three. Services pick the fields up on their next kit bump. runChecks keeps its published signature; the detail is available via the new runCheckOutcomes.

Verification

npm run verify green: type-check, lint, format, 100 tests (49 in health, 26 of them new), build, dist smoke. New coverage includes instance stability and non-hostname-ness, checkedAt advancing, startedAt tracking process.uptime(), memoized-vs-real timing separation, timeout-vs-refusal reasons, reason flattening and bounding, and that a hanging facts producer neither hangs the probe nor downs a service whose checks all passed.

Adds the four fields that let a probe interpret a health response instead of
just reading its status word, and closes a latent restart-loop in the facts
path.

- `timings` (per check, plus `facts`) and `durationMs`: a check near 0ms is a
  memoized value, not a measurement. `"db": "ok"` cannot distinguish a live
  round trip from a cached one that keeps reporting ok after the dependency
  dies. `durationMs` also lets a probe subtract the server from its own round
  trip: the admin console reports 197ms for catalog-backend where curl sees
  55ms including TLS, and nothing in the payload could attribute the gap.
- `checkedAt`: the cache detector. Frozen across two probes means something in
  front of the service is serving a copy. Nothing in the estate is cached
  today - `no-store` verified on all 21 - but it is undetectable if that
  changes.
- `instance`: random per process, so a value that changes between probes means
  a different replica answered. catalog-web runs at least two (distinct
  startedAt, interleaved across 8 samples), so its uptime bounces in a way
  otherwise indistinguishable from a crash loop. Not the hostname: /health is
  public and must not leak internal topology. Stashed on globalThis because
  Next can load one module into several bundle contexts per process.
- `reasons`: a timeout and an instant ECONNREFUSED both read `down` alone.
  Flattened and bounded to 200 chars - a driver message is not a stack trace
  to paste onto a public endpoint.

Fixes: the facts producer had no timeout while every check had one, so a fact
off a wedged DB could outlive the container HEALTHCHECK and have Swarm kill a
healthy task. Same shape as the store-mount rollback. catalog-backend, the one
service returning a DB-backed fact, had to hand-roll its own guard.

Also: `startedAt` now derives from `process.uptime()` rather than module load,
so a lazily-imported Next route handler stops reporting a fresh uptime for a
container that has been up for hours.

Additive only. `checks` stays a flat string map, so admin's derive, the e2e
smokes and the deploy gates keep working untouched; services pick the fields up
on their next kit bump. `runChecks` keeps its signature, with the detail
available via the new `runCheckOutcomes`.
@vreshch
vreshch merged commit 35ec111 into master Aug 10, 2026
1 check passed
@vreshch
vreshch deleted the feature/health-timing branch August 10, 2026 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant