bug(ci): Deploy Staging fails on a 90s healthcheck while the deploy actually succeeds #195

Filed late. This was observed and measured on 2026-10-06 during the v0.3.1 release and should have been filed that day. It was not, and #182 carries a comment of mine claiming it had been "filed separately" when it had not. ITblacksheep and Jeymz then spent 2026-10-10/11 rediscovering it on PRs #191 and #192. The measurements below existed the whole time.

PR #194 is the fix and is already open. This issue exists for the record, and to carry the root-cause dependency that #194 does not address.

Observed failure

Deploy Staging fails while the deploy itself succeeds.

corescope-dev failed health check after 90s
##[error]Process completed with exit code 1.

[box: gh run view 37556439970, 2026-10-06] — the master CI/CD run for 6cab7d69 reported failure, but every other job passed: Go Build & Test, Playwright E2E, Docker image, Publish Badges. Only Deploy Staging failed, and only on its healthcheck gate.

The container was fine. [box: corescope-dev on offband, 2026-10-06] dev came up healthy, served edge at commit 6cab7d6, and its bind-mounted config survived the redeploy. The deploy worked; the gate gave up too early.

Why 90s is not enough

[box: docker exec corescope-dev ps / wget, 2026-10-06] — a cold start measured over two minutes of polling:

First response at roughly 100s, and the full start-to-serving window measured about 210s including the pre-poll period. Docker's own healthcheck reported failure for about three minutes, then recovered unaided.

The old gate polled .State.Health.Status for 90 iterations at 1s. It cannot pass.

Root cause of the slowness is #181, which is still open

[box: /opt/corescope-dev/docker-compose.yml + docker exec env, 2026-10-06] — the dev service runs mem_limit: 2g with GOMEMLIMIT unset and GOGC=150. The Go heap sizes itself against the host's 8 GB, not the 2 GiB cgroup, so it climbs to 1.4g steady state against a 2.0g ceiling and the start is slow and memory-pressured. dmesg shows corescope-server OOM-killed at anon-rss:2045524kB on about 2026-09-27.

#194 makes CI tolerate the slow start. #181 is why it is slow. Both are worth doing; neither replaces the other.

What #194 changes

• window 90 → ${{ vars.HEALTH_TIMEOUT || '300' }}
• polls GET /api/healthz for "ready":true instead of Docker's health status
• exits non-zero immediately if the container is exited/dead/missing
• logs state every ~15s, dumps health JSON + 100 log lines on timeout

The endpoint it keys on is correct: /api/healthz is registered at [repo: cmd/server/routes.go:259] and [repo: cmd/server/healthz.go] returns {"ready": false, "reason": "loading"} with 503 until store.Load, pickBestObservation and the neighbor-graph build finish, then {"ready": true, ...} with 200. wget is the right client; that image has no curl.

One note on the new value: 300s against a measured ~210s cold start is about 43% headroom. Adequate, and vars.HEALTH_TIMEOUT makes it tunable without a code change. If #181 is fixed the start should get faster and the margin grows; if dev's dataset grows first, the margin shrinks.

Done when

Deploy Staging passes on a normal dev deploy without a false failure, and the run does not report failure when the deploy actually succeeded.

Refs #194, #181, #182, #127, #128