Repository navigation
Retry interrupted jobs that are safe to run again - #417
thomasklemm wants to merge 2 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🔵 Needs a closer look
Direct mutation of queue execution state warrants human validation of crash consistency and concurrent manual retries.
No open issues 🎉
What changed in this PR
Adds automatic recovery for safely repeatable jobs interrupted by worker shutdown or death, building on the Solid Queue migration in #346. The retry follow-up is the requested review scope; the supplied diff also includes the underlying migration.
Changes:
- Adds a minute-by-minute sweep that retries interrupted room-deletion and blob-purge jobs up to three times.
- Leaves other job classes and ordinary application failures untouched.
- Adds retry-policy coverage using real Solid Queue records in a test queue database.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
| File | Description |
|---|---|
test/scripts/post_restore_test.rb |
Tests primary restoration and queue cleanup. |
test/jobs/retry_interrupted_jobs_job_test.rb |
Tests retry eligibility and the retry cap. |
test/controllers/messages_controller_test.rb |
Tests batched message-job enqueueing. |
Procfile |
Replaces Resque workers with Solid Queue. |
lib/tasks/resque.rake |
Removes Resque lifecycle tasks. |
lib/rails_ext/sqlite_wal_checkpoint.rb |
Generalizes fork guidance in comments. |
hooks/post-restore |
Clears disposable queue data during restore. |
Gemfile.lock |
Locks Solid Queue and its dependencies. |
Gemfile |
Replaces Resque dependencies with Solid Queue. |
docs/upgrades/solid-queue.md |
Documents migration, backups, and rollback. |
docs/self-hosting.md |
Updates upgrade and backup instructions. |
docs/development.md |
Clarifies Redis’s remaining uses. |
db/queue_schema.rb |
Adds the Solid Queue database schema. |
config/resque-pool.yml |
Removes Resque worker configuration. |
config/recurring.yml |
Schedules cleanup and interrupted-job recovery. |
config/queue.yml |
Configures Solid Queue workers and dispatchers. |
config/initializers/sqlite_wal_checkpoint.rb |
Generalizes lifecycle comments for forking servers. |
config/initializers/solid_queue.rb |
Configures shutdown and checkpoint lifecycle hooks. |
config/initializers/resque.rb |
Removes Resque initialization. |
config/environments/test.rb |
Connects Solid Queue records to the test queue database. |
config/environments/production.rb |
Enables the Solid Queue adapter and database. |
config/environments/development.rb |
Removes obsolete commented Resque configuration. |
config/database.yml |
Adds test, performance, and production queue databases. |
bin/start-app |
Removes database preparation from web startup. |
bin/jobs |
Adds the Solid Queue command entry point. |
bin/boot |
Prepares databases before starting services. |
app/jobs/retry_interrupted_jobs_job.rb |
Implements filtered, capped, transactional retries. |
app/jobs/application_job.rb |
Adds batched job enqueueing. |
app/controllers/messages_controller.rb |
Batches jobs created when posting messages. |
Review effort: Balanced
007b9dc to
e9b24d9
Compare
e9b24d9 to
eb8c6d9
Compare
eb8c6d9 to
1791623
Compare
Jobs move from Resque in the container's Redis to Solid Queue in its own SQLite database, so jobs queued when the container stops are still there when it starts again. Redis doesn't persist (save "", appendonly no), so jobs still queued in Resque at the upgrade are lost, as on any restart. The queue database is backed up and restored with the primary. A room deletion or banned user's content removal still queued at the backup would otherwise never run after a restore. The two databases are copied one after the other and the app writes them in separate commits, so a job enqueued while the backup runs can be missing from it: a room deleted in that moment comes back closed and without members, its messages left. A job running at the backup comes back claimed by its process, and fails once that process is pruned as dead. bin/boot prepares the databases before starting the web server and the workers, which now start together and both need the queue's tables. Solid Queue forks its workers from a supervisor, so the supervisor stops its checkpoint contender before forking and each worker starts its own, as Puma's workers do. Workers poll every 0.1 s, as Resque did. The queue's pool is five, for the worker's three threads plus its poller and heartbeat, independent of the web server's RAILS_MAX_THREADS. The test environment gets a queue database, so the backup test can use real Solid Queue records; jobs still run on the test adapter. The benchmark drivers take --baseline-image, since a baseline from before this change needs Resque's gems, which the current image no longer has, and prepare the databases before starting Puma, so a seed made before the queue database existed still boots.
Solid Queue fails the job a worker was still running when the container stops or is killed, or when a restore brings back a job that was running at the backup. Every minute, retry such a job if running it again from the start does no harm: room deletion and file purges. Webhooks, Web Push, broadcasts and banned-content removal stay failed. Give up after three retries.
1791623 to
c2a5062
Compare
Stacked on #346, which should merge first; only the last commit (
c2a5062) is new here.With #346, a job still running when its worker is stopped or killed is marked failed (
ProcessExitError,ProcessMissingErrorat the next boot, orProcessPrunedErroronce the dead worker is pruned), and someone has to retry it by hand. Solid Queue deliberately leaves this to the app, since the job may be what killed the worker.This adds a recurring task that retries such a job once a minute, at most three times, for the jobs that are safe to run again from the start:
Room::DestroyJob(messages go one at a time and the room last) andActiveStorage::PurgeJob. Webhooks, Web Push and broadcasts would be delivered twice, and banned-content removal could reach a user unbanned since, so they stay failed, as do jobs that failed with an error of their own. A job missing from a restored queue isn't there to retry.