Skip to content

Wait for a slow scheduler start, and fail at once on a crash - #89

Merged
eigmax merged 1 commit into
mainfrom
fix/ctl-health-wait
Nov 8, 2026
Merged

eigmax merged 1 commit into
mainfrom
fix/ctl-health-wait

Conversation

@eigmax

@eigmax eigmax commented Nov 8, 2026

Copy link
Copy Markdown
Member

What happened

The v0.1.10 control-host upgrade ended with fleet-ctl: scheduler is not healthy; run: fleet-ctl.sh scheduler rollback, but the scheduler was fine. Before it listens, a scheduler reads every job and command record it keeps. On a control host with about 45,000 job and 69,000 command records (455 MB), it logged "state loaded" at 12:54:25 and "listening" at 12:55:37: 71 seconds. fleet-ctl.sh gave up after 30 health checks two seconds apart, about a minute. The scheduler was healthy from the next check on, and no rollback was needed.

Change

  • A deadline instead of a count (deploy/fleet-ctl.sh): scheduler upgrade and scheduler rollback wait up to FLEET_CTL_HEALTH_WAIT seconds (default 300) for /healthz.
  • A crash still fails fast: the wait ends at once when the scheduler's container exits or restarts, naming its state and pointing at its logs.
  • The script says it's waiting: once, with how long.
  • Docs (docs/maintenance.md): why the restart can take minutes, and the setting.

Tests

Test Without the change
test_a_scheduler_that_starts_slowly_is_waited_for Gives up after 30 checks; the scheduler answers on the 40th
test_a_scheduler_that_restarts_fails_at_once Keeps checking a restarted container 30 times
test_a_scheduler_that_never_answers_fails_at_the_deadline Ignores the deadline and checks 30 times

python -m pytest -q bridge/tests deploy/tests passes. The container check was also run read-only against the real control host, where it reads running 0.

Not changed

Startup still reads every record. Purging old job history (fleetctl job purge --older-than DAYS) shortens it; loading only what's in flight is a separate change.

fleet-ctl.sh gave the scheduler 30 health checks, two seconds apart,
before it reported "scheduler is not healthy" and suggested a rollback.
A scheduler reads every job and command record it keeps before it
listens: on a control host with about 45,000 job and 69,000 command
records that took 71 seconds, so the v0.1.10 upgrade reported a healthy
scheduler as failed.

The wait is now a deadline, FLEET_CTL_HEALTH_WAIT seconds (default 300),
and it ends at once when the scheduler's container exits or restarts,
so a crash still fails fast. The script says once that it is waiting,
and why. Tests cover a slow start, a restart and the deadline; each
fails on the previous script.
@eigmax
eigmax merged commit 7cdd0ac into main Nov 8, 2026
4 checks passed
@eigmax
eigmax deleted the fix/ctl-health-wait branch November 8, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant