Skip to content

Stop Redis after the processes that use it - #401

Open
namespaceMarcello wants to merge 1 commit into
basecamp:mainfrom
namespaceMarcello:stop-redis-after-the-app
Open

namespaceMarcello wants to merge 1 commit into
basecamp:mainfrom
namespaceMarcello:stop-redis-after-the-app

Conversation

@namespaceMarcello

Copy link
Copy Markdown
Contributor

Summary

On docker stop, bin/boot sends TERM to Thruster, Redis and resque-pool at the same time. Redis, which saves nothing (save ""), exits within milliseconds, and whatever is still running and needs it fails. A bot's message still being processed when the container stopped (kept in flight, for the test below, by holding SQLite's write lock) was saved and then answered 500, because enqueuing its push notifications found Redis gone. A bot's webhook job waiting for its answer was killed, and the bot's reply never posted.

Two shells also keep the signal from the servers. bin/start-app runs rails server under bash, and the FORK_PER_JOB=false INTERVAL=0.1 prefix in the Procfile makes Ruby spawn resque-pool through sh -c, so bin/boot signals sh. Both shells exit on TERM without passing it on: Puma never logs "Gracefully shutting down", resque-pool never logs a shutdown, and both are killed outright when bin/boot, PID 1, exits. Thruster meanwhile waits up to 5 seconds for the request it is proxying, so Puma, never told to stop, finishes it after Redis is gone.

This PR:

  • Stops Redis last: bin/boot asks the other processes to stop, waits for them to exit, then stops Redis. It asks once, since its trap also runs whenever a process exits, and resque-pool takes a second TERM as "stop waiting for the workers". Each process now starts in a group of its own, so a Ctrl-C in a terminal (docker run -it) reaches only bin/boot and goes through the same order, instead of reaching Redis directly.
  • execs the servers in bin/start-app and in the Procfile's workers line, so the TERM reaches Puma, which finishes its requests, and resque-pool, which with --term-graceful-wait sends QUIT to its workers and waits for them to finish their current job.

When a process dies on its own, the container still stops as a whole and the restart policy brings it back, as before; the others now stop in the same order.

How it was tested

test/bin/boot_test.rb runs bin/boot on a Procfile of stand-in processes that record when they're asked to stop and how many TERMs they get, and covers a TERM, a process exiting on its own and a Ctrl-C sent to the whole process group. The tests fail on main (Redis is asked to stop before the others have exited, the workers get three TERMs) and pass here, ten runs out of ten. Each piece of the change, undone on its own (a TERM on every exit, Redis stopped with the others, children in bin/boot's group, no trap on child exit), makes at least one of them fail.

Live, on the production image built from eccb2d6: two containers side by side, one with this PR's three files copied in, 2 CPUs and 1.5 GB each, docker stop -t 10, three runs of each case.

  • A bot posts a message while another process holds SQLite's write lock for 4 seconds, and docker stop comes a second later. On main the bot gets 500 every time (the message is saved, its push job isn't enqueued); here it gets 201 every time, and the container exits 0 once the request is done.
  • A bot's webhook answers after 3 seconds, and docker stop comes a second into the wait. On main the job is killed and no reply is posted, the stop taking 0.6 to 1.0 seconds; here the job finishes and the reply is posted every time, the stop taking 2.5 to 2.8 seconds.
  • Killing Redis, the Puma master or the resque-pool master with SIGKILL stops the container within 1.1 seconds on both.

The unit suite (653 tests), the 30 system tests, rubocop, herb and brakeman pass.

Risk

docker stop now lasts as long as the work in progress, up to Docker's stop timeout, where it used to end in well under a second; a job longer than the timeout is still killed when it runs out. Jobs enqueued while the container stops, such as the push notifications of a message saved in those seconds, still go to a Redis that is about to stop without saving: this lets running work finish, it doesn't keep the queue. ONCE updates are unchanged, since ONCE removes the old container with force; its Stop goes through docker stop. Reverting the commit restores the old behavior. #346 and #347 change the same lines of the Procfile, bin/boot and bin/start-app.

Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.

On `docker stop`, bin/boot sent TERM to Redis, Thruster and resque-pool at
once. Redis exits straight away, while a request or a job still running
needs it: a message posted in that moment was saved, then failed to
enqueue its push notifications and answered 500.

Puma and resque-pool didn't even get the signal. bin/start-app runs Rails
under bash, and the Procfile's environment prefix runs resque-pool under
sh; both shells exit on TERM without passing it on, so the two servers
were killed outright when bin/boot exited, and a running job with them.

Now bin/boot asks the other processes to stop, waits for them to exit,
then stops Redis. It asks once, because a second TERM tells resque-pool
to stop waiting for its workers, and it starts each process in a group of
its own, so a Ctrl-C in a terminal reaches only bin/boot. bin/start-app
and the Procfile exec their servers, so the TERM reaches Puma, which
finishes its requests, and resque-pool, which with --term-graceful-wait
lets its workers finish their current job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 24, 2026 11:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The focused shutdown changes and regression coverage have no identified blocking issues; tests were not independently executed.

No open issues 🎉

What changed in this PR

This PR makes container shutdown stop the web and worker processes before Redis, allowing in-flight requests and jobs to finish while Redis remains available.

Changes:

  • Orders shutdown in bin/boot, sends termination only once, and isolates child process groups.
  • Uses exec so shutdown signals reach Puma and resque-pool directly, with graceful worker shutdown enabled.
  • Adds regression tests for shutdown ordering, signal counts, child exits, and Ctrl-C.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
test/​bin/​boot_test.rb Tests shutdown ordering and signal handling with stand-in processes.
Procfile Enables direct signal delivery and graceful resque-pool shutdown.
bin/​start-app Replaces the shell with the Rails server using exec.
bin/​boot Stops Redis last and prevents repeated termination signals.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants