Proxy deployment restarts itself when a pinned service lookup 404s

[nilsmechtel]

Version: bioengine 0.11.19 (BIOENGINE_REF=4e73d9d), hypha_rpc 0.21.46, Ray 2.57.0.

ProxyDeployment's health check resolves its own service by the pinned id
<workspace>/<client_id>:<service_id>@<app_id>. That lookup intermittently 404s
while the client-agnostic form <workspace>/*:<service_id> resolves fine at the
same moment. On failure the check nulls websocket_service_id and raises, so Ray
restarts the replica, it re-registers under a new client id, and every handle a
caller is holding goes stale mid-request.

Why this is more than a transient error

The asymmetry is the tell. In apps/proxy_deployment.py the ping immediately
above the lookup tolerates _MAX_CONSECUTIVE_PING_FAILURES before declaring the
replica unhealthy. The lookup right below it gets no tolerance at all, so a single
404 is escalated to a replica restart.

For us the cascade shows up as a federated training round failing partway through:
the orchestrator holds a handle to the trainer proxy, the proxy restarts under a
new client id, and the round's next call finds nothing. Three separate layers
(orchestrator retry, UI polling tolerance, proxy health) got resilience work before
the shared cause was found, which is the main reason for writing this up.

proxy_deployment.py:262 sets self.client_id = f"{worker_client_id}-{app_hash}",
which is deterministic and stable across restarts, so the pinned id is not
changing under the lookup. The 404 appears to come from the resolution side.

Smaller thing in the same area

The "not found" log line is an uninterpolated f-string, so it prints the literal
placeholder rather than the id it failed on:

"Service not found: {service_id}@{app_id}")

That cost us a while, since the message looks like a deliberate template.

What we did locally

Patched in our container image, as a stopgap only:

• Count consecutive lookup failures and stay healthy until the same threshold the
  ping already uses.
• Reset that counter on a successful check, so unrelated blips hours apart cannot
  accumulate into a restart.

This hides the 404 rather than fixing it. The two candidate real fixes we can see
are to make the lookup resolve reliably, or to have the health check use the
client-agnostic form, which is what callers have to use anyway to survive a
restart.