Skip to content

fix(selfhost): retry the container health-status check, not just /ready#5090

Merged
JSONbored merged 1 commit into
mainfrom
fix/5085-followup-health-status-retry
Jul 11, 2026
Merged

fix(selfhost): retry the container health-status check, not just /ready#5090
JSONbored merged 1 commit into
mainfrom
fix/5085-followup-health-status-retry

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • selfhost-post-update-check.sh false-fails on a normal deploy: single /ready probe, no retry #5085 fixed the `/ready` HTTP probe to retry instead of single-attempt-failing. Deploying that exact fix (live, edge-nl-01, 2026-07-11) immediately surfaced a second, previously-masked instance of the same bug right below it: the Docker container health-status check (`docker inspect ... .State.Health.Status`) is also a single, non-retrying check, and caught the container in Docker's own transient `"starting"` state — Docker's OWN healthcheck re-probes `/ready` on an independent `interval`/`start_period` schedule, so it can still be `"starting"` even after the script's own `/ready` probe already answered 2xx.
  • Before selfhost-post-update-check.sh false-fails on a normal deploy: single /ready probe, no retry #5085 this was masked: the `/ready` probe almost always failed FIRST and immediately, so execution never reached the status check while the container was still genuinely starting.
  • Retries only through `"starting"` — any other non-healthy/running status (`unhealthy`, `exited`, `restarting`, ...) fails immediately rather than burning the full retry budget on something that will not resolve on its own.

Closes #5089

Scope

Validation

  • `git diff --check`
  • `bash -n` + `shellcheck` clean (only the same pre-existing, unrelated SC1091 info note)
  • `npm run typecheck`
  • Ran the 3 existing test files that reference this script — all pass unchanged.
  • Functionally verified both paths: a status sequence of starting/starting/healthy resolves via retry (confirmed 3 checks), and an immediate `unhealthy` status fails in ~0.01s without waiting out the retry budget.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A.)
  • UI changes use live API data or real empty/error/loading states. (N/A.)
  • Visible UI changes include a `UI Evidence` section. (N/A — no UI changed.)
  • Public docs/changelogs are updated where needed. (N/A.)

…dy (#5089)

#5085 fixed the /ready HTTP probe to retry instead of single-attempt-
failing, then immediately surfaced this second, previously-masked
instance of the same bug on the very next deploy: the Docker
container health-status check right below it is also a single,
non-retrying check, and caught the container in Docker's own
transient "starting" state (its healthcheck re-probes /ready on an
independent interval/start_period schedule, so it can still be
"starting" even after /ready itself already answered 2xx).

Retries only through "starting" -- any other non-healthy/running
status (unhealthy, exited, restarting, ...) fails immediately rather
than burning the full retry budget on something that will not
resolve on its own. Verified against both paths: starting -> healthy
resolves via retry, and an immediate unhealthy status fails fast
without waiting.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 11, 2026
@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.20%. Comparing base (0800f23) to head (193a958).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5090   +/-   ##
=======================================
  Coverage   94.20%   94.20%           
=======================================
  Files         470      470           
  Lines       39722    39722           
  Branches    14493    14493           
=======================================
  Hits        37421    37421           
  Misses       1645     1645           
  Partials      656      656           
Flag Coverage Δ
shard-1 46.42% <ø> (ø)
shard-2 33.58% <ø> (ø)
shard-3 31.34% <ø> (ø)
shard-4 32.80% <ø> (ø)
shard-5 33.54% <ø> (ø)
shard-6 44.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored JSONbored merged commit e2bc19b into main Jul 11, 2026
17 checks passed
@JSONbored JSONbored deleted the fix/5085-followup-health-status-retry branch July 11, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

selfhost-post-update-check.sh: the container health-status check still isn't retried (follow-up to #5085)

1 participant