chore(release): 0.9.0 - health timings, instance id, bounded facts - #19
Merged
Conversation
Adds the four fields that let a probe interpret a health response instead of just reading its status word, and closes a latent restart-loop in the facts path. - `timings` (per check, plus `facts`) and `durationMs`: a check near 0ms is a memoized value, not a measurement. `"db": "ok"` cannot distinguish a live round trip from a cached one that keeps reporting ok after the dependency dies. `durationMs` also lets a probe subtract the server from its own round trip: the admin console reports 197ms for catalog-backend where curl sees 55ms including TLS, and nothing in the payload could attribute the gap. - `checkedAt`: the cache detector. Frozen across two probes means something in front of the service is serving a copy. Nothing in the estate is cached today - `no-store` verified on all 21 - but it is undetectable if that changes. - `instance`: random per process, so a value that changes between probes means a different replica answered. catalog-web runs at least two (distinct startedAt, interleaved across 8 samples), so its uptime bounces in a way otherwise indistinguishable from a crash loop. Not the hostname: /health is public and must not leak internal topology. Stashed on globalThis because Next can load one module into several bundle contexts per process. - `reasons`: a timeout and an instant ECONNREFUSED both read `down` alone. Flattened and bounded to 200 chars - a driver message is not a stack trace to paste onto a public endpoint. Fixes: the facts producer had no timeout while every check had one, so a fact off a wedged DB could outlive the container HEALTHCHECK and have Swarm kill a healthy task. Same shape as the store-mount rollback. catalog-backend, the one service returning a DB-backed fact, had to hand-roll its own guard. Also: `startedAt` now derives from `process.uptime()` rather than module load, so a lazily-imported Next route handler stops reporting a fresh uptime for a container that has been up for hours. Additive only. `checks` stays a flat string map, so admin's derive, the e2e smokes and the deploy gates keep working untouched; services pick the fields up on their next kit bump. `runChecks` keeps its signature, with the detail available via the new `runCheckOutcomes`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A health response says
"db": "ok"and nothing else. That word cannot tell you whether the check ran a real query or returned a memoized value that has been reportingoksince the dependency died, whether the payload was computed just now or served from a cache, or whether the 197ms the admin console reports is the service or the edge.Four additive fields make the envelope interpretable, and one bug fix closes a latent restart-loop found while adding them.
What
timingsfacts. Near0is memoized; milliseconds is a real round trip.durationMscheckedAtinstancereasonsECONNREFUSEDboth readdownalone.{ "status": "degraded", "service": "catalog-backend", "instance": "b82fc8d3", "checkedAt": "2026-08-10T01:04:40.586Z", "durationMs": 60.9, "checks": { "db": "ok", "cache": "ok", "search": "degraded" }, "timings": { "db": 21.5, "cache": 0.1, "search": 60.7, "facts": 3.2 }, "reasons": { "search": "timed out after 60ms" }, "facts": { "servers": 20804 } }cache: 0.1next todb: 21.5is the whole point: one of those checks measured something.Same numbers go out as
Server-Timing: health;dur=60.9, db;dur=21.5, ..., so the split shows in devtools without parsing the body.Bug fixed: the facts producer had no timeout
resolveHealthraced every check against a timer and left the facts producer unbounded. A fact off a wedged database outlives the containerHEALTHCHECK --timeout=5s, three retries, Swarm kills a task that was only ever slow to count rows. Same shape as the store-mount rollback. It has not bitten because catalog-backend, the only service returning a DB-backed fact, hand-rolls its ownwithTimeout- the author had to supply the guarantee the kit should own. Now bounded byfactsTimeoutMs(1s default).Also
startedAtderives fromprocess.uptime()instead of module load, so a lazily-imported Next route handler stops reporting a fresh uptime for a container that has been up for hours.Findings that motivated this
catalog.agentage.io/healthreturned two distinctstartedAtvalues, interleaved. A single dead replica would flicker green/red across polls rather than reading red.instancemakes that visible; a follow-up in mcp-catalog decides whether to probe both or drop to one.cache-control: no-storeverified on all sampled endpoints.checkedAtis there so that stays true observably rather than by assumption.Compatibility
Additive only.
checksstays a flatRecord<string, CheckState>, so admin'sderive.ts, the e2e smokes and the deploy gates keep working untouched - a nested{state, ms}would have broken all three. Services pick the fields up on their next kit bump.runCheckskeeps its published signature; the detail is available via the newrunCheckOutcomes.Verification
npm run verifygreen: type-check, lint, format, 100 tests (49 in health, 26 of them new), build, dist smoke. New coverage includes instance stability and non-hostname-ness,checkedAtadvancing,startedAttrackingprocess.uptime(), memoized-vs-real timing separation, timeout-vs-refusal reasons, reason flattening and bounding, and that a hanging facts producer neither hangs the probe nor downs a service whose checks all passed.