Repository navigation
Stop Redis after the processes that use it - #401
Open
namespaceMarcello wants to merge 1 commit into
Open
namespaceMarcello wants to merge 1 commit into
namespaceMarcello wants to merge 1 commit into
Conversation
On `docker stop`, bin/boot sent TERM to Redis, Thruster and resque-pool at once. Redis exits straight away, while a request or a job still running needs it: a message posted in that moment was saved, then failed to enqueue its push notifications and answered 500. Puma and resque-pool didn't even get the signal. bin/start-app runs Rails under bash, and the Procfile's environment prefix runs resque-pool under sh; both shells exit on TERM without passing it on, so the two servers were killed outright when bin/boot exited, and a running job with them. Now bin/boot asks the other processes to stop, waits for them to exit, then stops Redis. It asks once, because a second TERM tells resque-pool to stop waiting for its workers, and it starts each process in a group of its own, so a Ctrl-C in a terminal reaches only bin/boot. bin/start-app and the Procfile exec their servers, so the TERM reaches Puma, which finishes its requests, and resque-pool, which with --term-graceful-wait lets its workers finish their current job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approved
The focused shutdown changes and regression coverage have no identified blocking issues; tests were not independently executed.
No open issues 🎉
What changed in this PR
This PR makes container shutdown stop the web and worker processes before Redis, allowing in-flight requests and jobs to finish while Redis remains available.
Changes:
- Orders shutdown in
bin/boot, sends termination only once, and isolates child process groups. - Uses
execso shutdown signals reach Puma and resque-pool directly, with graceful worker shutdown enabled. - Adds regression tests for shutdown ordering, signal counts, child exits, and Ctrl-C.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
| File | Description |
|---|---|
test/bin/boot_test.rb |
Tests shutdown ordering and signal handling with stand-in processes. |
Procfile |
Enables direct signal delivery and graceful resque-pool shutdown. |
bin/start-app |
Replaces the shell with the Rails server using exec. |
bin/boot |
Stops Redis last and prevents repeated termination signals. |
Review effort: Balanced
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On
docker stop,bin/bootsends TERM to Thruster, Redis and resque-pool at the same time. Redis, which saves nothing (save ""), exits within milliseconds, and whatever is still running and needs it fails. A bot's message still being processed when the container stopped (kept in flight, for the test below, by holding SQLite's write lock) was saved and then answered 500, because enqueuing its push notifications found Redis gone. A bot's webhook job waiting for its answer was killed, and the bot's reply never posted.Two shells also keep the signal from the servers.
bin/start-apprunsrails serverunder bash, and theFORK_PER_JOB=false INTERVAL=0.1prefix in the Procfile makes Ruby spawn resque-pool throughsh -c, sobin/bootsignalssh. Both shells exit on TERM without passing it on: Puma never logs "Gracefully shutting down", resque-pool never logs a shutdown, and both are killed outright whenbin/boot, PID 1, exits. Thruster meanwhile waits up to 5 seconds for the request it is proxying, so Puma, never told to stop, finishes it after Redis is gone.This PR:
bin/bootasks the other processes to stop, waits for them to exit, then stops Redis. It asks once, since its trap also runs whenever a process exits, and resque-pool takes a second TERM as "stop waiting for the workers". Each process now starts in a group of its own, so a Ctrl-C in a terminal (docker run -it) reaches onlybin/bootand goes through the same order, instead of reaching Redis directly.execs the servers inbin/start-appand in the Procfile'sworkersline, so the TERM reaches Puma, which finishes its requests, and resque-pool, which with--term-graceful-waitsends QUIT to its workers and waits for them to finish their current job.When a process dies on its own, the container still stops as a whole and the restart policy brings it back, as before; the others now stop in the same order.
How it was tested
test/bin/boot_test.rbrunsbin/booton a Procfile of stand-in processes that record when they're asked to stop and how many TERMs they get, and covers a TERM, a process exiting on its own and a Ctrl-C sent to the whole process group. The tests fail onmain(Redis is asked to stop before the others have exited, the workers get three TERMs) and pass here, ten runs out of ten. Each piece of the change, undone on its own (a TERM on every exit, Redis stopped with the others, children inbin/boot's group, no trap on child exit), makes at least one of them fail.Live, on the production image built from
eccb2d6: two containers side by side, one with this PR's three files copied in, 2 CPUs and 1.5 GB each,docker stop -t 10, three runs of each case.docker stopcomes a second later. Onmainthe bot gets 500 every time (the message is saved, its push job isn't enqueued); here it gets 201 every time, and the container exits 0 once the request is done.docker stopcomes a second into the wait. Onmainthe job is killed and no reply is posted, the stop taking 0.6 to 1.0 seconds; here the job finishes and the reply is posted every time, the stop taking 2.5 to 2.8 seconds.The unit suite (653 tests), the 30 system tests, rubocop, herb and brakeman pass.
Risk
docker stopnow lasts as long as the work in progress, up to Docker's stop timeout, where it used to end in well under a second; a job longer than the timeout is still killed when it runs out. Jobs enqueued while the container stops, such as the push notifications of a message saved in those seconds, still go to a Redis that is about to stop without saving: this lets running work finish, it doesn't keep the queue. ONCE updates are unchanged, since ONCE removes the old container with force; its Stop goes throughdocker stop. Reverting the commit restores the old behavior. #346 and #347 change the same lines of the Procfile,bin/bootandbin/start-app.Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.