supervise: tick() holds s.mu across an unbounded engine probe; one hung COM call wedges the whole bridge

[zcsizmadia]

internal/supervise/supervise.go holds s.mu across Engine.Running, and nothing on that path has a deadline.

Path: tick → engineAdapter.Running → Provisioner.engineRunning → Fast.List → comSession.do(ctx). The ctx is the supervisor's process-lifetime context — there is no per-tick timeout anywhere on it.

If EnumerateDistributions hangs — a service restart, wsl --update, a wedged VM — do blocks on <-done until shutdown while holding s.mu, and then:

• every Demand() blocks, so every docker connection through demandDialer hangs rather than failing — a hang is worse than an error here, because the CLI just sits there
• LifecycleSnapshot blocks, so the stats file freezes and the tray cannot even show that the supervisor is stuck
• the poke loop never runs again (same goroutine)

The COM rewrite amplified this

Before #380, COM callers were independent processes shelling out to wsl.exe. Now every caller — List, Terminate, doctor — queues behind one unbuffered channel and one OS thread (the apartment goroutine). A single hung call stalls callers that used to be unable to affect each other.

Related, same file: a panic on the apartment thread has no recover, so it takes the whole supervisor down — bridge and reconciler included. unsafe.Slice(arr, count) panics if the service ever returns count > 0 with a nil array, and hr == 0 is the only guard.

Fix

• A per-tick deadline on the engine probe, sized so a slow-but-alive wsl call still succeeds. The supervisor's own ctx is the wrong bound for a single tick.
• Do not hold s.mu across it. The probe's result is what needs the lock, not the probe.
• recover on the apartment goroutine, and restart the loop rather than leaving it dead — a dead loop with s.stop unclosed turns every later do into a block until its ctx fires, which on this path is never.

Found in the pre-0.6.0 independent concurrency review.

[zcsizmadia]

Half of this is done as of #440: the COM call is bounded and a panicking call no longer takes the supervisor down.

Done

• comSession.do derives a 20s bound, so a wedged wslservice no longer parks a caller for the life of the process. Deriving keeps the caller''s own cancellation winning when sooner, which Fast.List relies on to tell "backend unresponsive" (demote, fall back to wsl.exe) from "caller cancelled" (pass through).
• A panicking call is recovered per call, reported as an error, and the apartment loop survives. All three parts were needed: no recover kills the supervisor; a recover reporting success would make list return no distros and no error; a loop-level recover would leave the loop dead.

Negative controls: bound removed → the test times out at 45s. Panic reported as success → the test says so.

Still open, and the reason it was not done in the same change

tick still holds s.mu across the probe. Two obstacles, and the first one is a trap worth recording:

1. A tick-level deadline is actively dangerous as the code stands. Engine.Running returns a bool. A timeout therefore reads as "the engine is down", and tick''s very next move is:

case desired == DesiredRunning && !up:
    ...
    s.log().Warn("engine is down; starting it", ...)
    s.Engine.Start(ctx)

So bounding the probe naively converts "the probe was slow" into "start an engine that is probably already running". The prerequisite is a third answer — Running needs to be able to say unknown, and tick needs to do nothing when it hears it. That is an interface change with several implementations behind it.

2. Moving the probe outside the mutex changes the reconciler''s invariants. Demand takes the same lock, so it could run between the probe and the decision, and tick would then act on a stale reading — deciding the engine is down and starting it just after Demand already did. Whatever shape this takes needs to re-check under the lock, or make the decision idempotent.

The COM bound was separable because a failed list degrades to the wsl.exe path rather than to a wrong answer. These two do not have that property, which is why they want their own change.