Skip to content

Retry interrupted jobs that are safe to run again - #417

Open
thomasklemm wants to merge 2 commits into
basecamp:mainfrom
thomasklemm:feat/retry-interrupted-jobs
Open

thomasklemm wants to merge 2 commits into
basecamp:mainfrom
thomasklemm:feat/retry-interrupted-jobs

Conversation

@thomasklemm

@thomasklemm thomasklemm commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #346, which should merge first; only the last commit (c2a5062) is new here.

With #346, a job still running when its worker is stopped or killed is marked failed (ProcessExitError, ProcessMissingError at the next boot, or ProcessPrunedError once the dead worker is pruned), and someone has to retry it by hand. Solid Queue deliberately leaves this to the app, since the job may be what killed the worker.

This adds a recurring task that retries such a job once a minute, at most three times, for the jobs that are safe to run again from the start: Room::DestroyJob (messages go one at a time and the room last) and ActiveStorage::PurgeJob. Webhooks, Web Push and broadcasts would be delivered twice, and banned-content removal could reach a user unbanned since, so they stay failed, as do jobs that failed with an error of their own. A job missing from a restored queue isn't there to retry.

Copilot AI balanced review requested due to automatic review settings October 26, 2026 15:52
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-26T15:52:45.567913Z 0678eeb PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Direct mutation of queue execution state warrants human validation of crash consistency and concurrent manual retries.

No open issues 🎉

What changed in this PR

Adds automatic recovery for safely repeatable jobs interrupted by worker shutdown or death, building on the Solid Queue migration in #346. The retry follow-up is the requested review scope; the supplied diff also includes the underlying migration.

Changes:

  • Adds a minute-by-minute sweep that retries interrupted room-deletion and blob-purge jobs up to three times.
  • Leaves other job classes and ordinary application failures untouched.
  • Adds retry-policy coverage using real Solid Queue records in a test queue database.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
test/​scripts/​post_restore_test.rb Tests primary restoration and queue cleanup.
test/​jobs/​retry_interrupted_jobs_job_test.rb Tests retry eligibility and the retry cap.
test/​controllers/​messages_controller_test.rb Tests batched message-job enqueueing.
Procfile Replaces Resque workers with Solid Queue.
lib/​tasks/​resque.rake Removes Resque lifecycle tasks.
lib/​rails_ext/​sqlite_wal_checkpoint.rb Generalizes fork guidance in comments.
hooks/​post-restore Clears disposable queue data during restore.
Gemfile.lock Locks Solid Queue and its dependencies.
Gemfile Replaces Resque dependencies with Solid Queue.
docs/​upgrades/​solid-queue.md Documents migration, backups, and rollback.
docs/​self-hosting.md Updates upgrade and backup instructions.
docs/​development.md Clarifies Redis’s remaining uses.
db/​queue_schema.rb Adds the Solid Queue database schema.
config/​resque-pool.yml Removes Resque worker configuration.
config/​recurring.yml Schedules cleanup and interrupted-job recovery.
config/​queue.yml Configures Solid Queue workers and dispatchers.
config/​initializers/​sqlite_wal_checkpoint.rb Generalizes lifecycle comments for forking servers.
config/​initializers/​solid_queue.rb Configures shutdown and checkpoint lifecycle hooks.
config/​initializers/​resque.rb Removes Resque initialization.
config/​environments/​test.rb Connects Solid Queue records to the test queue database.
config/​environments/​production.rb Enables the Solid Queue adapter and database.
config/​environments/​development.rb Removes obsolete commented Resque configuration.
config/​database.yml Adds test, performance, and production queue databases.
bin/​start-app Removes database preparation from web startup.
bin/​jobs Adds the Solid Queue command entry point.
bin/​boot Prepares databases before starting services.
app/​jobs/​retry_interrupted_jobs_job.rb Implements filtered, capped, transactional retries.
app/​jobs/​application_job.rb Adds batched job enqueueing.
app/​controllers/​messages_controller.rb Batches jobs created when posting messages.

Review effort: Balanced

@thomasklemm
thomasklemm force-pushed the feat/retry-interrupted-jobs branch 2 times, most recently from 007b9dc to e9b24d9 Compare October 27, 2026 13:06
@thomasklemm
thomasklemm force-pushed the feat/retry-interrupted-jobs branch from e9b24d9 to eb8c6d9 Compare November 5, 2026 17:56
@thomasklemm
thomasklemm force-pushed the feat/retry-interrupted-jobs branch from eb8c6d9 to 1791623 Compare November 18, 2026 15:52
Jobs move from Resque in the container's Redis to Solid Queue in its own
SQLite database, so jobs queued when the container stops are still there
when it starts again. Redis doesn't persist (save "", appendonly no), so
jobs still queued in Resque at the upgrade are lost, as on any restart.

The queue database is backed up and restored with the primary. A room
deletion or banned user's content removal still queued at the backup
would otherwise never run after a restore. The two databases are copied
one after the other and the app writes them in separate commits, so a
job enqueued while the backup runs can be missing from it: a room deleted
in that moment comes back closed and without members, its messages left.
A job running at the backup comes back claimed by its process, and fails
once that process is pruned as dead.

bin/boot prepares the databases before starting the web server and the
workers, which now start together and both need the queue's tables.
Solid Queue forks its workers from a supervisor, so the supervisor stops
its checkpoint contender before forking and each worker starts its own,
as Puma's workers do. Workers poll every 0.1 s, as Resque did.

The queue's pool is five, for the worker's three threads plus its poller
and heartbeat, independent of the web server's RAILS_MAX_THREADS.

The test environment gets a queue database, so the backup test can use
real Solid Queue records; jobs still run on the test adapter.

The benchmark drivers take --baseline-image, since a baseline from before
this change needs Resque's gems, which the current image no longer has,
and prepare the databases before starting Puma, so a seed made before
the queue database existed still boots.
Solid Queue fails the job a worker was still running when the container
stops or is killed, or when a restore brings back a job that was running
at the backup. Every minute, retry such a job if running it again from
the start does no harm: room deletion and file purges. Webhooks, Web
Push, broadcasts and banned-content removal stay failed. Give up after
three retries.
@thomasklemm
thomasklemm force-pushed the feat/retry-interrupted-jobs branch from 1791623 to c2a5062 Compare November 20, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants