Version: bioengine 0.11.19 (4e73d9d)
File: bioengine/apps/proxy_deployment.py
Two independent defects in the same health-check path, reported together because a
fix for either one alone does not help. Part 1 is the inner probe of the entry
deployment. Part 2 is Ray Serve's outer window, which fires before the inner probe
runs at all.
Part 1: the entry probe deregisters Hypha services after a single 3 s timeout
check_health probes the entry deployment with a 3 second timeout and treats any
exception as terminal: it deregisters the application's Hypha services and raises.
There is no tolerance for a single slow reply, so any entry deployment that is
briefly busy loses its service registration and every peer holding its service id
starts getting 404s.
The Hypha ping a dozen lines further down in the same method already tolerates
_MAX_CONSECUTIVE_PING_FAILURES before drawing a conclusion. The entry check,
whose failure is strictly more destructive, tolerates nothing.
Code
else:
try:
await asyncio.wait_for(
self.entry_deployment_handle.check_health.remote(),
timeout=3.0,
)
except Exception as e:
logger.error(
f"❌ Entry deployment unhealthy for '{self.application_id}': {e}. "
f"Deregistering Hypha service."
)
await self._deregister_services()
raise RuntimeError(f"Entry deployment is unhealthy: {e}") from eObserved failure
A single-worker federated training run (one orchestrator app, one trainer app, both
Ray Serve deployments on the same worker). During round 0 the orchestrator was
aggregating model weights while the trainer's fit saturated the container's six
CPUs. The orchestrator's entry deployment missed one 3 s window:
11:41:13 ERROR ❌ Entry deployment unhealthy for 'crimson-heart-2759': . Deregistering Hypha service.
11:41:13 INFO 🔌 Unregistered WebSocket service 'chiron-platform/…:crimson-heart-2759'
11:41:13 INFO 🔌 Unregistered WebRTC service 'chiron-platform/…:crimson-heart-2759-rtc'
11:41:13 WARNING controller -- Health check for Replica(id='gww0lvl4', deployment='ProxyDeployment',
app='crimson-heart-2759') failed: RuntimeError: Entry deployment is unhealthy:
11:41:16 WARNING controller -- Didn't receive health check response for replica
Replica(id='lv4wll2q', deployment='ProxyDeployment', app='fragrant-waterfall-9016')
after 5.0s, marking it unhealthy.
Both proxy replicas were marked unhealthy within three seconds of each other, which
points at node-wide CPU contention rather than at an application fault. From that
moment the trainer's ping to the orchestrator and the browser's status poll both
failed against a service that was alive throughout:
KeyError: 'Service not found: chiron-platform/…:crimson-heart-2759@*'
The run ended there.
Three separate problems
1. No tolerance. A busy entry deployment is not an unhealthy one. Ray Serve's
health check asks whether the replica is alive. Answering it with a blocking RPC
into application code means any application slow enough to matter reports itself
dead. Aggregating a foundation model's state_dict exceeds 3 s routinely.
2. Deregistration takes the app off the network to report that it is busy.
The service does come back: once the entry deployment answers again the proxy
re-registers under the same id, so client handles recover on their own. But for
the whole gap every caller gets a hard transport failure rather than a slow
response. Measured on one run: deregistered at 12:16:07, re-registered at
12:17:24, so 77 seconds during which the browser saw net::ERR_ABORTED on every
call and the trainer's orchestrator ping burned retries. The health signal that
prompted this was a slow reply, and the response to a slow reply was to remove
the only route to the app. Nothing is gained by it, since a caller that could
have waited is now forced to fail.
3. The error message is empty. asyncio.TimeoutError stringifies to "", so
the log line renders as Entry deployment unhealthy for 'x': . Deregistering Hypha service. and RuntimeError: Entry deployment is unhealthy:. Neither names a
cause, and a timeout is indistinguishable from a dead actor or a serialization
error.
Suggested fix
Give this check the policy the ping below it already has:
- count consecutive failures, keep the services registered until the threshold
- make the timeout configurable, with a default that reflects what entry
deployments actually do
- include
type(e).__name__ in the log lines
A stronger version would decouple liveness from availability: track the last time
the entry deployment was seen alive and only deregister once that goes stale,
rather than probing synchronously on every tick.
Workaround in use
Patched locally as bioengine-proxy-entry-health-tolerance: consecutive-failure
tolerance reusing _MAX_CONSECUTIVE_PING_FAILURES, timeout from
CHIRON_ENTRY_HEALTH_TIMEOUT (default 30 s), and the exception type in both log
lines.
Part 2: the outer health-check window is shorter than a normal transfer
ProxyDeployment is declared with:
@deployment(
num_replicas=1,
max_ongoing_requests=10,
health_check_period_s=10,
health_check_timeout_s=5,
...
)health_check_timeout_s is not a budget inside check_health. It is Ray Serve's
wait for the actor to return from it. If the replica's event loop is not
scheduled within five seconds, the window closes before the method body executes.
This is why Part 1 cannot be fixed on its own. Adding tolerance inside
check_health is unreachable when the method never runs, and a longer inner
timeout is actively incoherent, because an inner wait cannot complete inside a
shorter outer window.
Observed failure
Same run, one round later, with the Part 1 workaround already in place. The
services stayed registered, but the trainer's proxy replica was restarted anyway
and the orchestrator was left holding a stale handle mid-round:
Didn't receive health check response for replica Replica(id='1w2ll1eo',
deployment='ProxyDeployment', app='noisy-frost-1098') after 5.0s,
marking it unhealthy.
What blocks the loop
Host load during the failures was 3.02 across 16 cores, so this is not blanket CPU
starvation. The misses cluster in the window where a round's weights move, and
they alternate between the two proxies in sender-then-receiver order:
11:50:48 start_fit OK
11:51:12 miss noisy-frost-1098 (trainer proxy, sends weights)
11:51:23 miss solitary-sea-2114 (orchestrator proxy, receives them)
11:51:31 miss noisy-frost-1098
11:51:32 miss solitary-sea-2114
Our working explanation is that aiortc's SCTP and DTLS processing of a large
parameter blob runs on the same asyncio loop that answers health checks, so a
transfer starves the check for as long as it takes. The fit occupies the same
window, so this is consistent rather than proven. Either way, a five second
scheduling guarantee is a strong assumption for a replica that also carries
application traffic.
Observed instance: 194 seconds of silence while the app kept working
A Geneformer run on 2026-08-22 gives a cleaner trace than the health-check
misses above, because the entry replica went completely quiet rather than
merely slow.
| Time (CEST) |
Event |
| 12:12:52 |
orchestrator calls start_fit on the trainer, round 1 |
| 12:14:10 |
last log line of any kind from the FederatedTrainingOrchestrator replica |
| 12:14:16 |
Ray Serve router begins reporting Failed to route request |
| 12:15:06, 12:15:36, 12:16:07 |
entry probe fails 3 times, Hypha services deregistered |
| 12:17:05 |
orchestrator calls start_fit for round 2 |
| 12:17:24 |
replica answers RPC again, proxy re-registers, controller reports it passed |
The replica emitted no log line at all for 194 seconds, and Ray's controller
never marked the entry deployment unhealthy or restarted it. The application was
not wedged: at 12:17:05, while still unreachable over RPC and with its Hypha
service already deregistered, it drove the federation forward by starting round
2 on the trainer. Only the request-serving path was starved.
This matters for the fix because it rules out the two readings that would make
the current behaviour correct. The replica was not dead, so deregistering it lost
a working service. And the stall was not caused by the health check itself, so
tightening the probe cannot help. Whatever occupies the loop during the round
boundary, the correct response is to wait for it, not to withdraw the app from
the network.
Suggested fix
Make both values configurable rather than fixed in the decorator, and pick
defaults that assume the proxy is doing real work. More fundamentally, the
liveness check should not be answerable only by the same loop that serves bulk
data. A cached "last seen alive" timestamp updated out of band would let the check
return immediately regardless of what the application is doing.
Workaround in use
Patched locally as bioengine-proxy-health-check-window: both values read from
the environment, defaulting to a 30 s period and a 60 s timeout, chosen to stay
above the Part 1 inner timeout so the two compose.
Related
The same method's WebSocket service lookup has the identical no-tolerance shape and
is reported separately, together with the pinned-id lookup that makes it fire.
Version: bioengine 0.11.19 (
4e73d9d)File:
bioengine/apps/proxy_deployment.pyTwo independent defects in the same health-check path, reported together because a
fix for either one alone does not help. Part 1 is the inner probe of the entry
deployment. Part 2 is Ray Serve's outer window, which fires before the inner probe
runs at all.
Part 1: the entry probe deregisters Hypha services after a single 3 s timeout
check_healthprobes the entry deployment with a 3 second timeout and treats anyexception as terminal: it deregisters the application's Hypha services and raises.
There is no tolerance for a single slow reply, so any entry deployment that is
briefly busy loses its service registration and every peer holding its service id
starts getting 404s.
The Hypha ping a dozen lines further down in the same method already tolerates
_MAX_CONSECUTIVE_PING_FAILURESbefore drawing a conclusion. The entry check,whose failure is strictly more destructive, tolerates nothing.
Code
Observed failure
A single-worker federated training run (one orchestrator app, one trainer app, both
Ray Serve deployments on the same worker). During round 0 the orchestrator was
aggregating model weights while the trainer's fit saturated the container's six
CPUs. The orchestrator's entry deployment missed one 3 s window:
Both proxy replicas were marked unhealthy within three seconds of each other, which
points at node-wide CPU contention rather than at an application fault. From that
moment the trainer's ping to the orchestrator and the browser's status poll both
failed against a service that was alive throughout:
The run ended there.
Three separate problems
1. No tolerance. A busy entry deployment is not an unhealthy one. Ray Serve's
health check asks whether the replica is alive. Answering it with a blocking RPC
into application code means any application slow enough to matter reports itself
dead. Aggregating a foundation model's
state_dictexceeds 3 s routinely.2. Deregistration takes the app off the network to report that it is busy.
The service does come back: once the entry deployment answers again the proxy
re-registers under the same id, so client handles recover on their own. But for
the whole gap every caller gets a hard transport failure rather than a slow
response. Measured on one run: deregistered at 12:16:07, re-registered at
12:17:24, so 77 seconds during which the browser saw
net::ERR_ABORTEDon everycall and the trainer's orchestrator ping burned retries. The health signal that
prompted this was a slow reply, and the response to a slow reply was to remove
the only route to the app. Nothing is gained by it, since a caller that could
have waited is now forced to fail.
3. The error message is empty.
asyncio.TimeoutErrorstringifies to"", sothe log line renders as
Entry deployment unhealthy for 'x': . Deregistering Hypha service.andRuntimeError: Entry deployment is unhealthy:. Neither names acause, and a timeout is indistinguishable from a dead actor or a serialization
error.
Suggested fix
Give this check the policy the ping below it already has:
deployments actually do
type(e).__name__in the log linesA stronger version would decouple liveness from availability: track the last time
the entry deployment was seen alive and only deregister once that goes stale,
rather than probing synchronously on every tick.
Workaround in use
Patched locally as
bioengine-proxy-entry-health-tolerance: consecutive-failuretolerance reusing
_MAX_CONSECUTIVE_PING_FAILURES, timeout fromCHIRON_ENTRY_HEALTH_TIMEOUT(default 30 s), and the exception type in both loglines.
Part 2: the outer health-check window is shorter than a normal transfer
ProxyDeploymentis declared with:health_check_timeout_sis not a budget insidecheck_health. It is Ray Serve'swait for the actor to return from it. If the replica's event loop is not
scheduled within five seconds, the window closes before the method body executes.
This is why Part 1 cannot be fixed on its own. Adding tolerance inside
check_healthis unreachable when the method never runs, and a longer innertimeout is actively incoherent, because an inner wait cannot complete inside a
shorter outer window.
Observed failure
Same run, one round later, with the Part 1 workaround already in place. The
services stayed registered, but the trainer's proxy replica was restarted anyway
and the orchestrator was left holding a stale handle mid-round:
What blocks the loop
Host load during the failures was 3.02 across 16 cores, so this is not blanket CPU
starvation. The misses cluster in the window where a round's weights move, and
they alternate between the two proxies in sender-then-receiver order:
Our working explanation is that aiortc's SCTP and DTLS processing of a large
parameter blob runs on the same asyncio loop that answers health checks, so a
transfer starves the check for as long as it takes. The fit occupies the same
window, so this is consistent rather than proven. Either way, a five second
scheduling guarantee is a strong assumption for a replica that also carries
application traffic.
Observed instance: 194 seconds of silence while the app kept working
A Geneformer run on 2026-08-22 gives a cleaner trace than the health-check
misses above, because the entry replica went completely quiet rather than
merely slow.
start_fiton the trainer, round 1FederatedTrainingOrchestratorreplicaFailed to route requeststart_fitfor round 2The replica emitted no log line at all for 194 seconds, and Ray's controller
never marked the entry deployment unhealthy or restarted it. The application was
not wedged: at 12:17:05, while still unreachable over RPC and with its Hypha
service already deregistered, it drove the federation forward by starting round
2 on the trainer. Only the request-serving path was starved.
This matters for the fix because it rules out the two readings that would make
the current behaviour correct. The replica was not dead, so deregistering it lost
a working service. And the stall was not caused by the health check itself, so
tightening the probe cannot help. Whatever occupies the loop during the round
boundary, the correct response is to wait for it, not to withdraw the app from
the network.
Suggested fix
Make both values configurable rather than fixed in the decorator, and pick
defaults that assume the proxy is doing real work. More fundamentally, the
liveness check should not be answerable only by the same loop that serves bulk
data. A cached "last seen alive" timestamp updated out of band would let the check
return immediately regardless of what the application is doing.
Workaround in use
Patched locally as
bioengine-proxy-health-check-window: both values read fromthe environment, defaulting to a 30 s period and a 60 s timeout, chosen to stay
above the Part 1 inner timeout so the two compose.
Related
The same method's WebSocket service lookup has the identical no-tolerance shape and
is reported separately, together with the pinned-id lookup that makes it fire.