Skip to content

Keep running when an orphan exits - #408

Open
namespaceMarcello wants to merge 1 commit into
basecamp:mainfrom
namespaceMarcello:keep-running-when-an-orphan-exits
Open

namespaceMarcello wants to merge 1 commit into
basecamp:mainfrom
namespaceMarcello:keep-running-when-an-orphan-exits

Conversation

@namespaceMarcello

Copy link
Copy Markdown
Contributor

Summary

In the image, bin/boot runs as PID 1, and PID 1 adopts every orphaned process in the container. bin/boot traps SIGCHLD to stop all the processes when one of them exits, but SIGCHLD comes for any child, so when an orphan exits, Campfire stops as if it had been sent docker stop: Thruster, Puma, resque-pool and Redis get TERM and the container exits 0. With a restart policy such as ONCE's always, that's a restart; with none, Campfire stays down.

An orphan is any process whose parent exits first. A command left running in the background by docker exec is one:

docker exec campfire sh -c 'sleep 8 &'

The shell exits at once, sleep is reparented to PID 1 (PPid: 1 in its /proc/<pid>/status), and when it exits 8 seconds later the whole container stops. So is the child of any process that dies before it.

This PR drops the SIGCHLD trap: bin/boot now waits for any child, with Process.wait and no pid. That reaps orphans too, as PID 1 has to (otherwise they stay zombies), and it stops the processes only when the child that exited is one of them. A single wait on the main thread also leaves no race between a trap and a per-process wait over who reaps a process first. bin/boot still sets SIGCHLD back to its default, as the trap did in passing: one ignored by whatever started it would have the kernel reap the children, unseen by the wait.

An init such as tini (docker run --init) would reap orphans too, but the image doesn't run one and ONCE doesn't add it, and bin/boot is already the process that is PID 1.

How it was tested

test/bin/boot_test.rb runs bin/boot on a Procfile of stand-in processes:

  • One of them leaves an orphan that exits 0.1 s later. To adopt it the way PID 1 does, the test starts bin/boot as a child subreaper (prctl(PR_SET_CHILD_SUBREAPER) through Fiddle, which stays set across exec), so this test runs on Linux and skips elsewhere. On main the orphan's exit sends TERM to both processes; here they keep running, the orphan is reaped instead of being left a zombie, and a TERM still stops them.
  • The other two check that when one process exits on its own, the others are stopped, also when bin/boot is started with SIGCHLD ignored.

Each piece of the change, undone on its own (stopping on any child's exit, not stopping on a process's exit, waiting only for the processes, keeping an inherited ignored SIGCHLD), makes one of them fail. The three pass five runs out of five.

Live, on our machine (laptop, Docker Desktop), on the production image built from eccb2d6 with 2 CPUs and 1.5 GB, side by side with and without this PR's bin/boot:

  • docker exec … sh -c 'sleep 8 &': on main the container stops as the sleep exits, and with --restart always it restarts, /up failing for about 4 seconds. Here /up keeps answering 200 throughout, and the sleep is gone from /proc, reaped.
  • kill -9 of redis-server stops the container within 30 ms on both, exiting 0. docker stop exits 0 in 0.5 to 0.7 s on both, three runs each.
  • ONCE's pre-backup hook, which ONCE runs with docker exec in the live container, leaves no orphan behind: Campfire keeps running with or without this change.

The unit suite, the system tests, rubocop, herb and brakeman pass.

Risk

Only how bin/boot waits for its children changes, and it reaches every container. When a Procfile process exits, crashes or is killed, the container still stops as a whole, as before. Reverting the commit restores the old behavior. #401 changes the same lines of bin/boot and adds the same test file; whichever is merged second will be rebased on the other.

Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.

In the image bin/boot is PID 1, which adopts orphaned processes. It
trapped SIGCHLD to stop every process when one of them exited, but the
signal comes for any child, so an orphan exiting (a command left in the
background by `docker exec`, or the child of a process that died first)
stopped all of Campfire.

Instead of the trap, bin/boot now waits for any child. That reaps
orphans too, as PID 1 has to, and stops the processes only when the
child that exited is one of them. It also sets SIGCHLD back to its
default, as the trap did in passing, so that one ignored by whatever
started bin/boot can't hide the exits from the wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 24, 2026 14:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants