Skip to content

fix: the health probe gets a third answer, and stops holding the lock - #441

Merged
zcsizmadia merged 1 commit into
mainfrom
fix/probe-tristate-and-unlocked
Sep 19, 2026
Merged

zcsizmadia merged 1 commit into
mainfrom
fix/probe-tristate-and-unlocked

Conversation

@zcsizmadia

Copy link
Copy Markdown
Collaborator

Closes #437. #440 bounded the COM call; this is the rest.

Engine.Running returns (bool, error)

Both implementations collapsed a failed probe into false — and the reconciler's response to false is Engine.Start. So a wedged wslservice, a WSL update mid-flight, or any transient probe failure read as "the engine is down" and provoked a start of an engine that was almost certainly running.

That is also why the bound could not simply be added to tick. Bounding a two-valued Running converts "slow" into "down" — the same bug with a timer attached. The third answer had to come first.

(false, nil) now means definitely not running. Anything the probe could not determine is an error, and tick does nothing that round rather than guess.

Provisioner keeps both shapes: EngineRunning for the waitFor polls, where "cannot tell" and "not yet" both correctly mean keep waiting; EngineRunningErr for the caller whose next move is to start something.

The probe runs outside s.mu

tick held the mutex across Engine.Running with the supervisor's process-lifetime context, so a service that stopped answering parked the reconciler inside the lock: every Demand() blocked (docker commands hung rather than failing), LifecycleSnapshot froze so the tray could not show the supervisor was stuck, and the poke loop never ran again.

Now: probe outside, decide inside, 60 s ceiling on the probe. Generous deliberately — a cold wsl.exe --list on a loaded machine is not fast, and this is a ceiling for a probe that has stopped answering, not a latency target.

Moving it out opens a window where Demand can cold-start the engine between the probe and the decision. startGen closes it: tick captures the counter before probing and gives up the round if it moved. Start is idempotent so the stale case was survivable anyway — but "engine is down; starting it" logged about an engine somebody just started is how an operator stops believing the log.

Verification

Three negative controls, each run both ways:

control result
probe error ignored Start called 41 times in 400 ms against a running engine
probe back inside mu Demand blocked behind a probe in flight
(#440) bound removed the COM test times out at 45 s

Plus TestDefiniteDownStillStartsTheEngine, so this cannot have bought safety by quietly disabling the supervisor.

Closes #437. #440 bounded the COM call; this is the rest.

Engine.Running returns (bool, error)
------------------------------------
Both implementations used to collapse a failed probe into false, and
the reconciler's response to false is Engine.Start. So a wedged
wslservice, a WSL update mid-flight or any transient probe failure read
as "the engine is down" and provoked a start of an engine that was
almost certainly running.

That is also why the bound could not simply be added to tick: bounding
a two-valued Running converts "slow" into "down", which is the same bug
with a timer attached. The third answer had to come first.

(false, nil) now means "definitely not running". Anything the probe
could not determine is an error, and tick does nothing at all that
round rather than guess -- lastUp keeps the last reading worth
trusting, and the next tick asks again.

Provisioner keeps both shapes. EngineRunning stays for the waitFor
polls, where "cannot tell" and "not yet" both correctly mean keep
waiting; EngineRunningErr is for the caller whose next move is to start
something. Same probe, the error no longer discarded on the path that
needs it.

The probe runs outside s.mu
---------------------------
tick held the mutex across Engine.Running with the supervisor's
process-lifetime context, so a service that stopped answering parked
the reconciler INSIDE the lock: every Demand() blocked, meaning every
docker connection hung rather than failing; LifecycleSnapshot froze, so
the tray could not even show the supervisor was stuck; and the poke
loop never ran again.

Now: probe outside, decide inside, with a 60s ceiling on the probe
itself. Generous on purpose -- a cold `wsl.exe --list` on a loaded
machine is not fast, and this is a ceiling for a probe that has stopped
answering, not a latency target.

Moving it out opens a window where a Demand can cold-start the engine
between the probe and the decision, leaving tick holding a reading that
no longer describes the machine. startGen closes it: tick captures the
counter before probing and gives up the round if it moved. Start is
idempotent so the stale case was survivable anyway, but "engine is
down; starting it" logged about an engine somebody just started is how
an operator stops believing the log.

Verification
------------
Three negative controls, each run both ways:

  probe error ignored     -> Start called 41 times in 400ms against a
                             RUNNING engine
  probe back inside mu    -> "Demand blocked behind a probe in flight"
  (from #440) bound gone  -> the COM test times out

Plus TestDefiniteDownStillStartsTheEngine, so the fix cannot have
bought safety by quietly disabling the supervisor.
@zcsizmadia
zcsizmadia merged commit f7b2465 into main Sep 19, 2026
5 checks passed
@zcsizmadia
zcsizmadia deleted the fix/probe-tristate-and-unlocked branch September 19, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

supervise: tick() holds s.mu across an unbounded engine probe; one hung COM call wedges the whole bridge

1 participant