Skip to content

fix(health): split liveness from readiness and make reachability configurable - #40

Merged
thinkbig1979 merged 1 commit into
mainfrom
fix/health-liveness-readiness
Jul 31, 2026
Merged

fix(health): split liveness from readiness and make reachability configurable#40
thinkbig1979 merged 1 commit into
mainfrom
fix/health-liveness-readiness

Conversation

@thinkbig1979

Copy link
Copy Markdown
Owner

Closes agent-os-69a.

What

/health returned 503 whenever the Docker daemon was unreachable. Capstan is a
separate process from Docker, so that is a readiness signal on a liveness
channel: anything that restarts on failed health checks would bounce Capstan for
an outage a restart cannot fix.

  • GET /health — liveness. 200 whenever the process is serving, no Docker
    call. Body keeps {"status":"healthy"} so existing monitors still match.
  • GET /health/ready — readiness. Reports dependencies, 503 naming what is
    degraded, 2s timeout on the Docker probe. Uses Ping (one round-trip) instead
    of the GetContainerList("") the old handler ran every 30 seconds.
  • Handler moved out of main.go — the last inline route — into
    internal/handlers/health.go, with 12 tests covering every branch.

Two corrections to the issue's premises

1. The HEALTHCHECK never detected the outage in the first place. The issue
assumes a Docker outage marks the container unhealthy. It does not. The check
used wget --spider, which sends HEAD; no HEAD route is registered, so every
HEAD falls through to the SPA catch-all and returns 200. Measured on the
pre-change image with Docker confirmed unreachable:

GET  /health                          -> {"status":"unhealthy"} [503]
HEAD /health                          -> [200]
HEAD /definitely-not-a-route          -> [200]     <- the SPA catch-all
HEALTHCHECK cmd (wget --spider) exit  -> 0
container health over 110s            -> healthy, FailingStreak=0 throughout

So the shipped healthcheck was a bare port probe — it would not have caught a
wedged server returning 500s either. Fixed here (outside the issue's scope, but
it is the mechanism the issue's central claim rests on): it now issues a real
GET, verified to exit 8 against a 503 and 0 against liveness.

2. The HEALTHCHECK URL does not change. The issue asks to point it at
liveness; liveness stays at /health, so only the command form changed. The
image still needed rebuilding because the binary's behaviour changed.

Decision: HEALTH_ALLOWED_NETWORKS, not TRUSTED_NETWORKS

The issue preferred reusing TRUSTED_NETWORKS. That value is simultaneously
Gin's trusted-proxy list and the AUTH_DISABLED bypass list — the README
documents that overloading at line 522. Reusing it would force an operator to
grant an uptime monitor X-Forwarded-For spoofing and authentication bypass just
to let it read a health endpoint. A dedicated setting is least-privilege; only
the CIDR-matching helper is shared (middleware.IsTrustedIP, now exported).

Empty means loopback only, so no upgrade widens exposure silently. The whole
loopback range is accepted, matching the net.IP.IsLoopback the old handler used
rather than narrowing to the literal 127.0.0.1 that IsTrustedIP matches.

Verification

Against a killable socat proxy in front of the real socket, so the host daemon
and its unrelated containers were never touched. The proxy's port was
confirmed closed before measuring — an earlier run assumed the kill worked
and measured a daemon that was still reachable, which produced a false "ready".

Docker reachable     /health 200 healthy   /health/ready 200 ready
Docker unreachable   /health 200 healthy   /health/ready 503
                     {"checks":{"docker":{"error":"Cannot connect to the Docker
                      daemon at tcp://127.0.0.1:12375...","status":"unavailable"}},
                      "degraded":["docker"],"status":"degraded"}
container health     healthy, FailingStreak=0, across 110s / >3 intervals

Reachability over bridge networking, so the caller is genuinely non-loopback
(under --network host everything looks like loopback):

default                                403 on both endpoints
HEALTH_ALLOWED_NETWORKS=172.16.0.0/12  200

That second run is also the proof the new env var is actually read end to end,
rather than being documented-and-defaulted like PORT (agent-os-o6q).

Each of the three behaviours that matter was confirmed by reverting it and
re-running: liveness consulting Docker (3 tests fail), the readiness timeout
removed (the hung-daemon test fails after 3s), and the network gate opened
(the default-denied and no-probe-when-denied tests fail).

Gates

go build ./... && go vet ./...                        OK
go test ./... -count=1                                746 passed, 0 failed  (baseline 727)
go vet -tags=integration ./internal/integrationtest/  OK

No frontend changes. gofmt -l flags internal/middleware/auth.go for import
ordering, pre-existing on mainagent-os-far's territory.

Images removed and docker builder prune -f run afterwards (2.9GB); no leftover
containers or stray processes.

🤖 Generated with Claude Code

…igurable

Capstan is a separate process from Docker, but /health returned 503 whenever the
daemon was unreachable — a readiness signal on a liveness channel. Anything that
restarts on failed health checks would bounce Capstan for an outage a restart
cannot fix.

- /health is liveness: 200 whenever the process is serving, no Docker call. Body
  keeps {"status":"healthy"} so existing monitors still match.
- /health/ready is readiness: reports dependencies, 503 naming what is degraded,
  with a 2s timeout on the Docker probe so a hung daemon cannot pile up
  goroutines across repeated probes. Uses Ping, one round-trip, rather than the
  GetContainerList("") the old handler ran every 30 seconds.
- Handler moved out of main.go into internal/handlers/health.go, the last inline
  route, with 12 tests covering every branch.
- Reachability is HEALTH_ALLOWED_NETWORKS, deliberately NOT TRUSTED_NETWORKS:
  that value is Gin's trusted-proxy list and the AUTH_DISABLED bypass list, so
  reusing it would force an operator to grant an uptime monitor
  X-Forwarded-For spoofing and auth bypass just to read a health endpoint.
  Empty means loopback only, so no upgrade widens exposure silently. The whole
  loopback range is accepted, matching the net.IP.IsLoopback the old handler
  used rather than narrowing to the literal 127.0.0.1.
- DockerService.Ping tolerates a nil receiver and nil client: main leaves
  dockerService nil when the daemon was unreachable at startup, and a nil
  *DockerService in an interface is a non-nil interface value.

The HEALTHCHECK was also fixed, and this was not in the issue. It used
`wget --spider`, which sends HEAD; no HEAD route is registered, so every HEAD
fell through to the SPA catch-all and returned 200. Measured on the pre-change
image with Docker unreachable: GET /health 503, HEAD /health 200, the healthcheck
command exit 0, and the container reporting healthy with FailingStreak=0 through
four intervals. So the outage the issue describes never marked the container
unhealthy in the first place — the check was a bare port probe. It now issues a
real GET, verified to exit 8 against a 503 and 0 against liveness.

Verified on a rebuilt image against a killable socat proxy in front of the real
socket, so the host daemon and its unrelated containers were never touched, with
the proxy's port confirmed closed before measuring:
  Docker reachable    /health 200 healthy   /health/ready 200 ready
  Docker unreachable  /health 200 healthy   /health/ready 503 degraded ["docker"]
  container health    healthy, streak 0, across 110s / >3 intervals
And over bridge networking, so the caller is genuinely non-loopback:
  default                                403 on both endpoints
  HEALTH_ALLOWED_NETWORKS=172.16.0.0/12  200

Closes agent-os-69a

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thinkbig1979
thinkbig1979 merged commit fc7d4c1 into main Jul 31, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant