Skip to content

Replace Resque with Solid Queue - #346

Open
thomasklemm wants to merge 1 commit into
basecamp:mainfrom
thomasklemm:feat/solid-queue-cutover
Open

thomasklemm wants to merge 1 commit into
basecamp:mainfrom
thomasklemm:feat/solid-queue-cutover

Conversation

@thomasklemm

@thomasklemm thomasklemm commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Jobs move from Resque in the container's Redis to Solid Queue in its own SQLite database. That Redis doesn't persist (save ""), so today a restart or an update drops every queued job, such as the file purges of a room deletion. With Solid Queue they're still there when the container starts again. Jobs still in Resque at the upgrade are lost, as on any restart today.

The queue database is backed up and restored with the primary, so a room deletion or a banned user's content removal still queued at the backup runs after a restore. The two are copied one after the other and the app writes them in separate commits, so a job enqueued while the backup runs can be missing from it: a room deleted in that moment comes back closed and without members, with its messages. A job running at the backup comes back claimed by its process and fails once that process is pruned as dead.

The worker runs three threads in one process, as Solid Queue's installer sets it up, with its own pool of five connections.

@thomasklemm thomasklemm changed the title Replace Resque with Solid Queue Migrate background jobs from Resque to Solid Queue Oct 8, 2026
@dhh

dhh commented Oct 8, 2026

Copy link
Copy Markdown
Member

Reviewed the updated 0b39bc85b8d6: auxiliary WAL auto-checkpointing and the Solid Queue checkpoint lifecycle hooks resolve the earlier integration findings; the native supervisor runs its start hooks before starting children. One current configuration gap remains: the queue still inherits RAILS_MAX_THREADS=1, while Solid Queue's own calculation requires three database connections for a one-thread worker, its poller and heartbeat. Please give the queue an independent pool suitable for that supported layout. The backup script also snapshots primary then queue separately, so ordinary backup guidance needs coordinated quiescence or explicit tested restore guarantees; otherwise a later queue snapshot can contain jobs referencing records absent from the earlier primary snapshot. The documented Resque drain and smaller dependency chain are useful, but adoption still needs independent real job delivery/failure/retry, startup, worker shutdown/failover and primary-plus-queue restore checks. This remains an operational migration with no independently measured HTTP or job-throughput gain.

GPT on behalf of DHH

@dhh

dhh commented Oct 8, 2026

Copy link
Copy Markdown
Member

Reviewed the follow-up at 3914ce5. The dedicated queue pool of three resolves the worker/poller/heartbeat sizing issue. The quiesced backup procedure and paused-volume restore handling also address the separate-snapshot concern in the source; the previous comment should not be read as saying those fixes are still missing.

This remains an operational backend migration rather than a demonstrated HTTP throughput optimization. Before merging, we still need independent production checks of queued-job execution and shutdown, migration/drain behavior, and backup/restore recovery, including validation of the assumed ONCE pause-before-copy fallback. The current CI checks are green.

GPT on behalf of DHH

@thomasklemm
thomasklemm marked this pull request as ready for review October 8, 2026 14:09
Copilot AI balanced review requested due to automatic review settings October 8, 2026 14:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@thomasklemm

Copy link
Copy Markdown
Contributor Author

Updated in df9ed80, ready for another review:

  • Backups no longer pause Campfire. Following the earlier point about separate primary/queue snapshots, the queue database is now treated as disposable rather than snapshotted. hooks/pre-backup and script/admin/prepare-backup are back to exactly what main has (online snapshot of the primary only). The archive command excludes production_queue.sqlite3*, and hooks/post-restore deletes any queue database left in the volume so db:prepare starts an empty queue. That removes the paused-volume marker, the legacy-snapshot refusal and pre_backup_test.rb; net about 150 fewer lines. Jobs pending at backup time are not replayed after a restore.
  • Worker layout (e0d5446): one process with three threads by default, matching the Solid Queue installer, instead of one process per two cores. The queue pool is now 5 (three threads plus poller and heartbeat), still independent of RAILS_MAX_THREADS. JOB_CONCURRENCY still adds processes.

Unit suite (581 runs) and RuboCop pass locally on Ruby 4.0.7; hosted CI is green. A local production supervisor started the 3-thread worker, dispatcher and scheduler and picked up an enqueued job.


Generated by Claude Code

@namespaceMarcello

Copy link
Copy Markdown
Contributor

The thread notes twice that this change has no throughput measurement yet. Here is one for the README's "Post a message" row: head df9ed80 against its merge base 05c5a2c.

Setup. From our machine: a laptop with an AMD Ryzen 9 7940HX, Docker. Two production images built from the two commits; their lock files differ only by Resque and its dependencies against solid_queue 1.7.0 and its own. The shape is the one of bench/compare.rb in once-campfire-verification: its seed, loadgen http --post-room with 16 clients posting to one room for 8 s after 2 s of warm-up, 4 Puma workers × 1 thread, Redis and Thruster in the same container on 4 CPUs, the load generator on 4 other CPUs, both SQLite files on tmpfs. One difference from that driver: Docker's bridge network instead of the host's. Every response was validated and every write audited: 0 invalid responses and 0 failed jobs in 21 runs. Figures are the median of 3 runs in alternating order (4 for the base), with min–max.

bench/compare.rb sets JOB_CONCURRENCY=3 by default. resque-pool does not read it (2 workers on 4 CPUs); on this branch config/queue.yml does, so the same command starts 3 worker processes of 3 threads instead of the default single one. All three layouts are below.

posts/s vs base job CPU per post jobs still waiting when the 8 s end
base 05c5a2c, Resque, 2 workers 255.4 (244.4–279.0) 4.3 ms 0
df9ed80, JOB_CONCURRENCY=3: 3 processes × 3 threads 166.7 (164.1–167.8) −35% 8.7 ms 0
df9ed80, JOB_CONCURRENCY=2 180.4 (170.0–189.1) −29% 7.2 ms 16–120
df9ed80, default: 1 process × 3 threads 232.9 (221.4–242.9) −9% see below 1,934–2,140

With the default layout the posts per second are at the edge of the noise (the two ranges touch), but the jobs do not keep up with this workload: when the window ends, 45–46% of the jobs enqueued in it have run, in each of the 3 runs. The rest drain in 6.6–7.7 s, at 277–294 jobs/s with the worker process on 1.02 cores, which is about 3.5 ms of CPU per job.

Where the time goes, per post with 16 clients. One instrumented run per column, so these are indicative:

base 3 processes 1 process
request, wall time 15.7 ms 27.8 ms 18.3 ms
enqueueing Room::PushMessageJob and Message::BroadcastUnreadRoomJob 1.9 ms 11.4 ms 6.7 ms
of which inside BEGIN IMMEDIATE on the queue database 7.3 ms 3.6 ms

One more single run: with the two jobs enqueued but never worked (scheduled an hour ahead), 289 posts/s, against 386 with no job enqueued at all. With Resque the same pair of lanes had come out equal (405 and 406, on edbc779 the evening before).

What this does not show.

  • The driver points push endpoints at 127.0.0.1:9, so Room::PushMessageJob never waits on the network here. The threads on this branch are there to overlap that wait, and this workload cannot show it.
  • tmpfs: no disk writes for either database.
  • 8 seconds, one machine, one room, 16 clients. The posting rate below which a single worker process keeps up was not measured.
  • No profile of the Solid Queue workers: where the extra job CPU goes is not known.

None of this contradicts the PR, which claims no throughput gain. It is the cost side on this one row, on our machine; which row applies to the README depends on whether the verification runs keep the driver's default JOB_CONCURRENCY=3. The harness (a wrapper around loadgen, a CPU sampler by process role, the counters read from the queue database) can be posted here, and any lane rerun with another layout.

Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.

@namespaceMarcello

Copy link
Copy Markdown
Contributor

The review above asks for independent checks of queued-job execution and shutdown before merging. Here is one: a room deletion cut off by the container going away, on this head (df9ed80) and on its merge base (05c5a2c), two runs per row.

Setup. Production images built from the two commits (the same two as in the throughput comment above), on Docker Desktop. One container on 4 CPUs, 1.5 GB, no network, its own volume. A closed room with three members and 10,000 messages, 1,000 of them with a 1 KB text file. Deleting it calls Room#destroy_later, as RoomsController#destroy does: the members go at once, and Room::DestroyJob destroys the messages one at a time (13.1 s for all 10,000 on the base, 13.6 s on this branch, when nothing interrupts it); each file goes through its own ActiveStorage::PurgeJob. Without an interruption, both commits delete the room, every message and every file. Then three ways for the container to go while the job runs:

  • docker kill (SIGKILL) 5 s after the job is enqueued, then the same container started again on the same volume;
  • docker stop -t 10 (SIGTERM, SIGKILL after 10 s, Docker Engine's default on Linux) 5 s after, then started again. The first base run had no -t; it was down in 0.45 s, inside any grace period;
  • as ONCE updates an app (internal/docker/application.go, Stop the old app container before removing it on an update once#125): a second container from the same image starts on the same volume 1.2 to 1.3 s after the job is enqueued, and once it answers /up the old one is removed with docker rm -f (5.8 to 6.5 s after). kamal-proxy is left out: there is no HTTP traffic here.
messages left in the deleted room files on disk with no message the deletion job afterwards
base, kill 6,221 / 6,580 9 / 7 no trace
base, stop 6,511 / 6,537 4 / 5 no trace
base, ONCE update 6,174 / 5,967 7 / 4 no trace
base with #401, stop 519 / none, room deleted 834 / 631 killed at 10 s in the first run, finished in the second
this PR, kill 7,085 / 6,439 1 / 3 claimed by the dead worker, then failed with ProcessPrunedError 318 / 321 s after the restart
this PR, stop 2,649 / 2,789 0 / 0 failed with ProcessExitError during the shutdown
this PR, ONCE update 6,096 / 6,835 0 / 2 claimed by the dead worker, then failed with ProcessPrunedError 314 / 313 s after the old container went

What the rows show:

  • Base. The queue lives in the container's Redis, which doesn't save to disk (config/redis.conf: save "", appendonly no), so the running job and every queued purge go with the container. The room stays in the database without members, with the messages not yet destroyed and their files, and nothing in Resque's stats or failures shows it. docker stop ends the same way: on SIGTERM resque-pool 0.7.1 signals its workers and returns without waiting for them (its join loop just breaks), so bin/boot has nothing left to wait for and the container is down in half a second, job included.
  • Base with Stop Redis after the processes that use it #401. That open PR of ours has resque-pool wait for its workers on SIGTERM (--term-graceful-wait) and stops Redis after them, so the running job gets Docker's grace period. In one run it finished (room deleted, container down in 8.0 s); in the other it was still running at 10 s and was killed with 519 messages left. Either way the purge jobs queued during the deletion go with Redis: 834 and 631 files stay on disk with no message. Our other open PR on bin/boot, Keep running when an orphan exits #408, leaves the stop row as on the base: the room stays (6,179 and 9,943 messages left in two runs).
  • This PR, queued jobs. They survive. The purge jobs still waiting when the container went (276 and 350 in the two kill runs, 730 and 717 in the two stop runs) ran within seconds of the restart. The 0 to 3 files left are from the moment of the cut: purge jobs that were running then, which fail with the deletion job, or a message destroyed just before its purge was enqueued.
  • This PR, the running job. After a SIGKILL, or the forced removal of an ONCE update, its claimed execution stays with the dead worker's process row until the supervisor prunes processes whose heartbeat is older than process_alive_threshold (5 min), on a maintenance pass every 5 min, and fails what they had claimed. With docker stop -t 10 the container goes down cleanly in 5.5 s: the supervisor waits its shutdown_timeout (5 s), the worker is still inside the job, so it gets QUIT and the job is failed with ProcessExitError (the log: "Supervisor wasn't terminated gracefully - shutdown timeout exceeded (5008.7ms)", then "Fail claimed jobs"). Either way the room is left as on the base, now with a row in solid_queue_failed_executions. Nothing in this branch reads or retries failed executions.
  • Retrying it works. At the end of both ONCE-update runs, SolidQueue::FailedExecution#retry from a console on the failed Room::DestroyJob finished the room within seconds: room and messages gone, since the job starts over from the messages that are left.

So on this branch an interrupted room deletion is no longer silent and its queued file purges are no longer lost, while the deletion itself still needs someone to retry it.

The scripts (seed, deletion, probe, the three ways down), the raw logs and the predictions written before each run can be posted here.

Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work.

@thomasklemm

Copy link
Copy Markdown
Contributor Author

Thanks @namespaceMarcello, both runs are useful. I've summarized them in the description.

The default layout follows the Solid Queue installer. On SQLite, extra worker processes buy job throughput at the cost of request throughput, as your numbers show, so I've kept the default and left JOB_CONCURRENCY for installs that need it.

Automatic retry of the interrupted job isn't in this PR. It's now a failed execution that can be retried, instead of being lost.

@thomasklemm

Copy link
Copy Markdown
Contributor Author

@namespaceMarcello I followed up on both of your measurements in two new commits:

  • Batched enqueues: the jobs a new message enqueues (push, unread fanout, bot webhooks) are now written to the queue database in one transaction instead of one each.
  • Shutdown timeout: raised to 8 s, so a running job gets most of Docker's 10 s before it's failed.
  • Performance environment: queue.yml now has a performance entry.

Posting load, same shape as yours (16 clients, 4 Puma workers × 1 thread, one worker with 3 threads), 9 interleaved rounds on one machine. The A/A row is a second copy of the same build:

Build Posts/s, median (range)
before 206 (191–216)
before, A/A copy 208 (186–218)
batched 239 (219–259)

More posts also means more queued work: about 2,200 jobs pending at the end of the window instead of 1,800, which drained in about 9 s instead of 7.5 s.

A room deletion where docker stop arrives late in the job now finishes in 2 of 2 runs. Before, it failed with ProcessExitError both times. If the stop arrives early, the job still doesn't finish in 8 s and is failed as before; queued jobs survive either way.

I also tried a 1 s polling interval and not keeping finished jobs. Neither made a measurable difference, so both stay as they were.

Retrying an interrupted job automatically is still out of scope here. I'll open that as a separate PR on top of this one, limited to jobs that are safe to run twice.

@thomasklemm
thomasklemm force-pushed the feat/solid-queue-cutover branch 2 times, most recently from a14b19b to 0853d7a Compare October 27, 2026 13:05
@thomasklemm
thomasklemm force-pushed the feat/solid-queue-cutover branch from 0853d7a to cbee3f9 Compare November 5, 2026 17:56
@thomasklemm
thomasklemm force-pushed the feat/solid-queue-cutover branch from cbee3f9 to 2394bc3 Compare November 18, 2026 15:52
@thomasklemm thomasklemm changed the title Migrate background jobs from Resque to Solid Queue Replace Resque with Solid Queue Nov 18, 2026
Jobs move from Resque in the container's Redis to Solid Queue in its own
SQLite database, so jobs queued when the container stops are still there
when it starts again. Redis doesn't persist (save "", appendonly no), so
jobs still queued in Resque at the upgrade are lost, as on any restart.

The queue database is backed up and restored with the primary. A room
deletion or banned user's content removal still queued at the backup
would otherwise never run after a restore. The two databases are copied
one after the other and the app writes them in separate commits, so a
job enqueued while the backup runs can be missing from it: a room deleted
in that moment comes back closed and without members, its messages left.
A job running at the backup comes back claimed by its process, and fails
once that process is pruned as dead.

bin/boot prepares the databases before starting the web server and the
workers, which now start together and both need the queue's tables.
Solid Queue forks its workers from a supervisor, so the supervisor stops
its checkpoint contender before forking and each worker starts its own,
as Puma's workers do. Workers poll every 0.1 s, as Resque did.

The queue's pool is five, for the worker's three threads plus its poller
and heartbeat, independent of the web server's RAILS_MAX_THREADS.

The test environment gets a queue database, so the backup test can use
real Solid Queue records; jobs still run on the test adapter.

The benchmark drivers take --baseline-image, since a baseline from before
this change needs Resque's gems, which the current image no longer has,
and prepare the databases before starting Puma, so a seed made before
the queue database existed still boots.
@thomasklemm
thomasklemm force-pushed the feat/solid-queue-cutover branch from 2394bc3 to 272d7a5 Compare November 20, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants