Fix a macOS cell stopping when it kills a worker's tools - #91
Merged
Merged
Conversation
On macOS, `Process.kill` raised `Errno::EPERM`, not `Errno::ESRCH`, for a process group whose only members were zombies. A tool killed with its worker stays a zombie until `launchd` reaps it, so the supervisor's reap could raise, stop the cell, and leave the caller with no response. `test_a_deadline_kills_what_the_worker_started_too` failed intermittently on the macOS CI job for this reason. Rescue `Errno::EPERM` from the group kill in `sweep_group` and `kill_group`.
There was a problem hiding this comment.
Copilot review overview
🟢 Approved
The narrow exception-handling change preserves existing fallback behavior, covers both paths with regression tests, and has no identified blocking issues.
Review effort: Balanced
Findings: None
What changed in this PR
Fixes macOS cells stopping when signaling a worker’s process group containing only zombies.
Changes:
- Handles
Errno::EPERMin group cleanup and termination, preserving the worker-PID fallback. - Adds regression tests for both error paths.
- Documents the fix in the changelog.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
| File | Description |
|---|---|
| hotcell-server/test/scheduling_test.rb | Tests refused group signals during deadline termination and cleanup. |
| hotcell-server/lib/hot_cell/supervisor.rb | Handles group-signal permission errors without stopping the cell. |
| CHANGELOG.md | Records the macOS fix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On macOS,
Process.killraisesErrno::EPERM, notErrno::ESRCH, for a process group whose only members are zombies. Linux signals such a group without error.A tool that a worker spawns is in the worker's process group. When the supervisor kills the worker, the tool becomes a zombie until
launchdreaps it. If the supervisor reaps the worker beforelaunchdreaps the tool,sweep_groupraisesErrno::EPERM. It rescues onlyErrno::ESRCH, so the error unwindsSupervisor#runand stops the cell before the reap answers the caller. The caller then reads end of stream instead of thekilledverdict.kill_grouphas the same gap.test_a_deadline_kills_what_the_worker_started_toofailed intermittently on the macOS job for this reason (run):Production cells run on Linux and are not affected.
Details
sweep_groupandkill_grouprescueErrno::EPERMfrom the group kill. A group kill raisesErrno::EPERMonly when no member received the signal, the same outcome asErrno::ESRCH.kill_groupthen falls back to the worker's own pid, as it does forErrno::ESRCH. The fallback still rescues onlyErrno::ESRCH, because macOS signals a zombie pid without error.The new tests stub
Process.killin the cell to raiseErrno::EPERMfor a group:kill_group: before this change the cell stopped and left the worker running, so the caller timed out.