Repository navigation
Keep running when an orphan exits - #408
Open
namespaceMarcello wants to merge 1 commit into
Open
namespaceMarcello wants to merge 1 commit into
namespaceMarcello wants to merge 1 commit into
Conversation
In the image bin/boot is PID 1, which adopts orphaned processes. It trapped SIGCHLD to stop every process when one of them exited, but the signal comes for any child, so an orphan exiting (a command left in the background by `docker exec`, or the child of a process that died first) stopped all of Campfire. Instead of the trap, bin/boot now waits for any child. That reaps orphans too, as PID 1 has to, and stops the processes only when the child that exited is one of them. It also sets SIGCHLD back to its default, as the trap did in passing, so that one ignored by whatever started bin/boot can't hide the exits from the wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In the image,
bin/bootruns as PID 1, and PID 1 adopts every orphaned process in the container.bin/boottraps SIGCHLD to stop all the processes when one of them exits, but SIGCHLD comes for any child, so when an orphan exits, Campfire stops as if it had been sentdocker stop: Thruster, Puma, resque-pool and Redis get TERM and the container exits 0. With a restart policy such as ONCE'salways, that's a restart; with none, Campfire stays down.An orphan is any process whose parent exits first. A command left running in the background by
docker execis one:The shell exits at once,
sleepis reparented to PID 1 (PPid: 1in its/proc/<pid>/status), and when it exits 8 seconds later the whole container stops. So is the child of any process that dies before it.This PR drops the SIGCHLD trap:
bin/bootnow waits for any child, withProcess.waitand no pid. That reaps orphans too, as PID 1 has to (otherwise they stay zombies), and it stops the processes only when the child that exited is one of them. A single wait on the main thread also leaves no race between a trap and a per-process wait over who reaps a process first.bin/bootstill sets SIGCHLD back to its default, as the trap did in passing: one ignored by whatever started it would have the kernel reap the children, unseen by the wait.An init such as tini (
docker run --init) would reap orphans too, but the image doesn't run one and ONCE doesn't add it, andbin/bootis already the process that is PID 1.How it was tested
test/bin/boot_test.rbrunsbin/booton a Procfile of stand-in processes:bin/bootas a child subreaper (prctl(PR_SET_CHILD_SUBREAPER)through Fiddle, which stays set acrossexec), so this test runs on Linux and skips elsewhere. Onmainthe orphan's exit sends TERM to both processes; here they keep running, the orphan is reaped instead of being left a zombie, and a TERM still stops them.bin/bootis started with SIGCHLD ignored.Each piece of the change, undone on its own (stopping on any child's exit, not stopping on a process's exit, waiting only for the processes, keeping an inherited ignored SIGCHLD), makes one of them fail. The three pass five runs out of five.
Live, on our machine (laptop, Docker Desktop), on the production image built from
eccb2d6with 2 CPUs and 1.5 GB, side by side with and without this PR'sbin/boot:docker exec … sh -c 'sleep 8 &': onmainthe container stops as thesleepexits, and with--restart alwaysit restarts,/upfailing for about 4 seconds. Here/upkeeps answering 200 throughout, and thesleepis gone from/proc, reaped.kill -9ofredis-serverstops the container within 30 ms on both, exiting 0.docker stopexits 0 in 0.5 to 0.7 s on both, three runs each.pre-backuphook, which ONCE runs withdocker execin the live container, leaves no orphan behind: Campfire keeps running with or without this change.The unit suite, the system tests, rubocop, herb and brakeman pass.
Risk
Only how
bin/bootwaits for its children changes, and it reaches every container. When a Procfile process exits, crashes or is killed, the container still stops as a whole, as before. Reverting the commit restores the old behavior. #401 changes the same lines ofbin/bootand adds the same test file; whichever is merged second will be rebased on the other.Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.