fix(apps): treat absence from the Serve status report as unknown - #177
Draft
nilsmechtel wants to merge 1 commit into
Draft
fix(apps): treat absence from the Serve status report as unknown#177nilsmechtel wants to merge 1 commit into
nilsmechtel wants to merge 1 commit into
Conversation
monitor_applications computed
unhealthy = application is None or state in ("DEPLOY_FAILED", "UNHEALTHY")
which reads "Ray did not mention this app" as "Ray says this app is
broken". Those are different claims: the first can simply mean the
status report was momentarily incomplete, and acting on it tears down a
healthy app and destroys whatever it was in the middle of. The first
such observation fired a redeploy immediately -- the backoff only starts
after the first fire, so there was nothing to absorb it.
Absence is now tolerated for _MISSING_FROM_STATUS_TOLERANCE (3)
consecutive ticks, logged at INFO, and the counter is cleared the moment
the app reappears in any state. The tolerance is deliberately finite:
never redeploying a genuinely dead app would be its own outage, so
unknown must still converge.
The stale-replica branch deletes an app expecting the next tick to
redeploy it. That gap is known rather than unknown, so it is marked and
skips the tolerance instead of tripling that path's recovery latency.
_fire_redeploy now takes a required reason and logs it. An "absent from
the report" restart and a genuine UNHEALTHY restart previously produced
byte-identical log lines, which is why nobody could say how often the
first already happened.
Refs: svamp issue #12
This was referenced Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A momentary gap in Ray's status report restarts a healthy app.
monitor_applicationscomputedunhealthy = application is None or state in ("DEPLOY_FAILED", "UNHEALTHY"), which reads "Ray did not mention this app" as if it were "Ray says this app is broken". Those are different claims — the first can simply mean the status report was incomplete for a tick — and acting on it tears down a running app and destroys whatever it was in the middle of. There was nothing to absorb it either: the backoff only starts after the first fire, so the very first such observation redeployed immediately.What changed
Absence is tolerated for
_MISSING_FROM_STATUS_TOLERANCE(3) consecutive ticks — roughly 30 s at the default monitor interval — logged at INFO each time, with the counter cleared the moment the app reappears in any state. The tolerance is finite on purpose: never redeploying a genuinely dead app is its own outage, so unknown has to converge rather than become a permanent veto.The stale-replica branch added in #50 deletes an app expecting the next tick to see it missing and redeploy fresh replicas. That gap is known rather than unknown, so it is marked and skips the tolerance — otherwise this PR would have tripled the recovery latency of a path that is already degraded.
_fire_redeploynow takes a requiredreasonand logs it. An "absent from the report" restart and a genuineUNHEALTHYrestart previously produced byte-identical warning lines, which is why nobody can say how often the first already happens in production. That was the second half of the issue and it is what makes the next incident readable.Tests
tests/apps/test_monitor_missing_from_status.py, 10 tests. Both behavioural halves were positive-controlled rather than asserted:reasonfrom the log line fails 4, one of which shows the two warning lines coming out byte-identical — the production symptom verbatim.One of those 10 asserts
1 < _MISSING_FROM_STATUS_TOLERANCE < 100directly, because every other test counts its ticks off that constant: at 1 the loops run zero iterations and the suite would pass vacuously while the bug was back.Full run on this branch: 187 passed (
tests/_app tests/apps tests/worker,--noconftest; the 32 errors are the pre-existing model-runner/cellpose fixture errors that flag needs).Not yet done
No version bump — per CLAUDE.md that lands as a separate commit just before this leaves draft. This touches
bioengine/**, so it also wants a dev-image run on a live cluster before it is marked ready; it can share one image with #175 and #176.Closes svamp issue #12.