fix(worker): rebuild the Hypha client when the server evicts it

[nilsmechtel]

The worker's own Hypha client never recovers from a server-side eviction: Hypha drops the client registration while the socket stays open from the container's side, so hypha-rpc never sees a disconnect, never reconnects, and <workspace>/<client-id>:bioengine-worker stops resolving forever. The process stays alive, kill -0 1 passes, the liveness probe passes, and nothing is logged. Observed on 2026-09-01 while validating #167, reproduced with a docker pause of 150–260 s.

The monitoring loop's existing echo("ping") cannot see this. It kept succeeding for ten minutes against a server that had already dropped the registration — an open socket answers regardless of what the registry holds. So the loop now also asks whether Hypha still serves the worker's own service id, once a minute, and rebuilds the connection when it does not. This is the same decision PR #167 made for ProxyDeployment; in the incident that motivated this issue the two halves of a single process disagreed, with the worker client MISSING and the app proxy RESOLVED because the proxy's maintenance loop noticed within one 60 s probe and rebuilt in ~5 s.

The probe rebuilds; it never condemns

This is the part worth reviewing, because the tempting version is wrong.

There is a second, unrelated failure mode on deNBI: four dropouts in 47.4 h in which Hypha briefly served nothing — worker, model-runner and cellpose3-runner all gone together, two different pods, both restartCount=0, no Killing/Started events. Those episodes self-heal. cute-mare's paired registry sampler puts them at 7 consecutive failed probes per episode (1×500 then 6×404), a failing span of ≥191–192 s, an outage window ≤252 s, and a duration MLE of 222 s (band 206–238 s). Meanwhile /health/liveness returned 200 on every 30 s sample throughout all four.

So a 3-consecutive-miss rule at a 30 s interval would condemn at ~90 s and would have killed a recovering worker four times, and neither endpoint alone yields a correct condemn decision. The discriminator between the two failure modes is recovery, not any single sample: this issue never self-heals, the sibling always does at a fixed ~200 s.

_check_service_registration therefore never raises and never reports its outcome upwards — in particular it never touches _monitor_consecutive_errors, which is what flips get_status to not-ready and lets the k8s liveness probe cycle the pod. A redundant rebuild during a recoverable outage costs one reconnect. A condemn kills a worker that was about to come back. Two tests pin this, one behavioural and one on the source of the method itself.

The bounded disconnect

_connect_to_server closed the previous connection with an unbounded await self.server.disconnect(). That was harmless while the only caller was startup and an echo-detected dead socket; it is not harmless now, because the close runs on the rebuild path, where the transport being closed is by definition the one suspected of being wedged. It is now bounded at 5 s so a hung close cannot stall the monitoring loop on exactly the socket that prompted the rebuild.

One correction to the issue's framing while I was in here. The issue expected the worker client to carry "the worker service AND every app service registered through it", making a stranded registration worse than on the proxy. It does not: register_service appears in exactly two places, and app services are registered by each ProxyDeployment on its own client. The worker client carries one service plus the artifact-manager and app-builder handles derived from it, and complete_initialization re-points those at the new connection on every rebuild. The blast radius is smaller than assumed, which is why bounding the close was enough rather than needing a lock or a generation counter.

Not folded in

The proxy's _RECONNECT_GRACE_S / _REREGISTER_BACKOFF_S pair is not lifted. The proxy needs a grace window because it rebuilds from the on_disconnected hook, which fires for drops hypha-rpc goes on to repair itself — its first version rebuilt three times in 30 s during a real wobble and crashed. The worker has no such hook: the 60 s probe interval is itself the rate limit, at most one rebuild attempt per minute, so a second backoff constant would be dead weight. If review prefers the constants shared between the two rather than chosen per site, that is a reasonable call to make now rather than later.

Tests

11 new tests in tests/worker/test_worker_registration_probe.py. 216 passed on the branch against 205 on origin/main under the same invocation, with the same 55 pre-existing --noconftest collection errors.

Five positive controls, each failing exactly the tests that name the behaviour and no others:

control	fails
probe with echo("ping") instead of the registration lookup	5, including test_a_live_socket_with_a_dropped_registration_still_rebuilds
raise after a failed rebuild	3, including test_the_probe_is_not_wired_into_the_readiness_backstop
drop the per-interval gate	test_the_probe_costs_one_round_trip_per_interval
restore the unbounded disconnect()	test_a_hung_disconnect_does_not_stall_the_rebuild
remove the probe from the monitoring loop	test_the_monitoring_loop_runs_the_registration_probe

Not yet validated on a live cluster — this is bioengine/**, so it needs a dev image and a docker pause reproduction before it is marked ready. It is the sixth PR queued on that same validation call, after #175, #176, #177, #178 and #179.