fix(scheduler): skip resurrection for idle-sleeping agents (cherry-pick of #446 → main) - #448
Merged
Conversation
The session watchdog has been emitting "resurrection attempt 5/5 for dead agent X" every 30s for every idle-sleeping agent on the Pi fleet since PR #348/#349 landed. Those PRs added the idle_sleeping check, but only at the API callback site — by the time it fires, the scheduler has already consumed a rate-limit budget slot and logged the attempt. With 7 agents idle-sleeping most of the time, the journal fills with noise and Brad reports it "feels too eager." Root cause: _maybe_resurrect's only gate was the rate limit. The API callback would refuse the actual reconnect, but the scheduler had no way to know that ahead of time, so it kept burning budget + log slots. Fix: add is_resurrectable_fn precondition to AgentScheduler. Called before budget/log/callback. If it returns False, return silently — no budget consumed, no log emitted, no callback fired. api.py wires in _is_resurrectable which returns False when the streaming session is idle_sleeping (or missing entirely). Fails open on precondition exceptions so a buggy resolver can't silently disable resurrection for the whole fleet. Tests: - precondition False → callback never fires, budget stays at 0 - precondition True → original behavior preserved - precondition raises → falls through (fail-open) Pi was hot-patched in parallel (early-return in _maybe_resurrect) to quiet the noise immediately; this PR replaces that with the proper fix, so the next deploy cleanly overwrites the hot-patch. Co-authored-by: Oleg <oleg@Olegs-Mac-mini.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #446 (closes #445) onto main, since beta↔main reconciliation is non-trivial (tracked in #449) and Pi is scheduled to update tonight.
Why this (not the full beta→main rollup)
Tried promoting all of beta → main (#447) — beta and main have diverged because of hotfixes that went straight to main (#416 SDK floor, #411 discord backport, #408/#398/#397 broker fixes, #407 Docker, etc., 10 commits total). The full merge would require resolving 3+ conflict hunks across
api.py,agent_registry.py,pyproject.toml, andtests/test_admin_update.py— risky right before a deploy without Brad's institutional context.Brad gave me the call. This is the safer path: deliver the watchdog fix (the original ask) cleanly; defer the full reconciliation to a separate effort with proper eyes (#449).
What's in this PR
Just the cherry-pick of
30f2990(#446 squash on beta):src/pinky_daemon/scheduler.py—is_resurrectable_fnpreconditionsrc/pinky_daemon/api.py— wires_is_resurrectableto skip idle-sleepingtests/test_scheduler.py— 3 new testsWhat's deferred (still on beta, NOT in this PR)
Full reconciliation in #449.
Test plan
pytest tests/test_scheduler.py -k resurrect— 8 passed locallyupdate_and_restartat 21:00 PDT pulls main, hot-patch in scheduler.py gets cleanly overwritten by the proper fix🤖 Opened by Barsik