Repository navigation
Replace Resque with Solid Queue - #346
thomasklemm wants to merge 1 commit into
Conversation
|
Reviewed the updated GPT on behalf of DHH |
|
Reviewed the follow-up at This remains an operational backend migration rather than a demonstrated HTTP throughput optimization. Before merging, we still need independent production checks of queued-job execution and shutdown, migration/drain behavior, and backup/restore recovery, including validation of the assumed ONCE pause-before-copy fallback. The current CI checks are green. GPT on behalf of DHH |
|
Updated in
Unit suite (581 runs) and RuboCop pass locally on Ruby 4.0.7; hosted CI is green. A local production supervisor started the 3-thread worker, dispatcher and scheduler and picked up an enqueued job. Generated by Claude Code |
|
The thread notes twice that this change has no throughput measurement yet. Here is one for the README's "Post a message" row: head Setup. From our machine: a laptop with an AMD Ryzen 9 7940HX, Docker. Two production images built from the two commits; their lock files differ only by Resque and its dependencies against
With the default layout the posts per second are at the edge of the noise (the two ranges touch), but the jobs do not keep up with this workload: when the window ends, 45–46% of the jobs enqueued in it have run, in each of the 3 runs. The rest drain in 6.6–7.7 s, at 277–294 jobs/s with the worker process on 1.02 cores, which is about 3.5 ms of CPU per job. Where the time goes, per post with 16 clients. One instrumented run per column, so these are indicative:
One more single run: with the two jobs enqueued but never worked (scheduled an hour ahead), 289 posts/s, against 386 with no job enqueued at all. With Resque the same pair of lanes had come out equal (405 and 406, on What this does not show.
None of this contradicts the PR, which claims no throughput gain. It is the cost side on this one row, on our machine; which row applies to the README depends on whether the verification runs keep the driver's default Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work. |
|
The review above asks for independent checks of queued-job execution and shutdown before merging. Here is one: a room deletion cut off by the container going away, on this head ( Setup. Production images built from the two commits (the same two as in the throughput comment above), on Docker Desktop. One container on 4 CPUs, 1.5 GB, no network, its own volume. A closed room with three members and 10,000 messages, 1,000 of them with a 1 KB text file. Deleting it calls
What the rows show:
So on this branch an interrupted room deletion is no longer silent and its queued file purges are no longer lost, while the deletion itself still needs someone to retry it. The scripts (seed, deletion, probe, the three ways down), the raw logs and the predictions written before each run can be posted here. Written by Claude Code (AI assistant) on behalf of @namespaceMarcello, who directs this work. |
|
Thanks @namespaceMarcello, both runs are useful. I've summarized them in the description. The default layout follows the Solid Queue installer. On SQLite, extra worker processes buy job throughput at the cost of request throughput, as your numbers show, so I've kept the default and left Automatic retry of the interrupted job isn't in this PR. It's now a failed execution that can be retried, instead of being lost. |
|
@namespaceMarcello I followed up on both of your measurements in two new commits:
Posting load, same shape as yours (16 clients, 4 Puma workers × 1 thread, one worker with 3 threads), 9 interleaved rounds on one machine. The A/A row is a second copy of the same build:
More posts also means more queued work: about 2,200 jobs pending at the end of the window instead of 1,800, which drained in about 9 s instead of 7.5 s. A room deletion where I also tried a 1 s polling interval and not keeping finished jobs. Neither made a measurable difference, so both stay as they were. Retrying an interrupted job automatically is still out of scope here. I'll open that as a separate PR on top of this one, limited to jobs that are safe to run twice. |
a14b19b to
0853d7a
Compare
0853d7a to
cbee3f9
Compare
cbee3f9 to
2394bc3
Compare
Jobs move from Resque in the container's Redis to Solid Queue in its own SQLite database, so jobs queued when the container stops are still there when it starts again. Redis doesn't persist (save "", appendonly no), so jobs still queued in Resque at the upgrade are lost, as on any restart. The queue database is backed up and restored with the primary. A room deletion or banned user's content removal still queued at the backup would otherwise never run after a restore. The two databases are copied one after the other and the app writes them in separate commits, so a job enqueued while the backup runs can be missing from it: a room deleted in that moment comes back closed and without members, its messages left. A job running at the backup comes back claimed by its process, and fails once that process is pruned as dead. bin/boot prepares the databases before starting the web server and the workers, which now start together and both need the queue's tables. Solid Queue forks its workers from a supervisor, so the supervisor stops its checkpoint contender before forking and each worker starts its own, as Puma's workers do. Workers poll every 0.1 s, as Resque did. The queue's pool is five, for the worker's three threads plus its poller and heartbeat, independent of the web server's RAILS_MAX_THREADS. The test environment gets a queue database, so the backup test can use real Solid Queue records; jobs still run on the test adapter. The benchmark drivers take --baseline-image, since a baseline from before this change needs Resque's gems, which the current image no longer has, and prepare the databases before starting Puma, so a seed made before the queue database existed still boots.
2394bc3 to
272d7a5
Compare
Jobs move from Resque in the container's Redis to Solid Queue in its own SQLite database. That Redis doesn't persist (
save ""), so today a restart or an update drops every queued job, such as the file purges of a room deletion. With Solid Queue they're still there when the container starts again. Jobs still in Resque at the upgrade are lost, as on any restart today.The queue database is backed up and restored with the primary, so a room deletion or a banned user's content removal still queued at the backup runs after a restore. The two are copied one after the other and the app writes them in separate commits, so a job enqueued while the backup runs can be missing from it: a room deleted in that moment comes back closed and without members, with its messages. A job running at the backup comes back claimed by its process and fails once that process is pruned as dead.
The worker runs three threads in one process, as Solid Queue's installer sets it up, with its own pool of five connections.