Skip to content

Replace Redis with Solid Cable - #347

Open
thomasklemm wants to merge 3 commits into
basecamp:mainfrom
thomasklemm:deps/solid-cable-removal
Open

thomasklemm wants to merge 3 commits into
basecamp:mainfrom
thomasklemm:deps/solid-cable-removal

Conversation

@thomasklemm

@thomasklemm thomasklemm commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #345 and #346, which should merge first; only the last commit (4540f47) is new here.

With the cache on Solid Cache and the jobs on Solid Queue, Action Cable is the last thing using Redis. It moves to Solid Cable, in its own SQLite database, and Redis goes: the server process, the redis and kredis gems, the Debian package, bin/setup's container and CI's service.

Development uses Solid Cable as well, because bin/dev runs several Puma workers and a broadcast has to reach sockets held by the others. Broadcasts are transient, so the restore hook deletes the cable database together with the cache's.

@thomasklemm

Copy link
Copy Markdown
Contributor Author

Updated in 7ae38af, ready for another review:

  • Merged the current Use Solid Cache for the production Rails cache #345 and Replace Resque with Solid Queue #346 branches, so this PR carries the disposable-queue backup approach: backups snapshot only the primary database online, the archive excludes queue, cache and Cable databases, and hooks/post-restore clears all three so db:prepare recreates them empty.
  • Development now uses Rails' default async Action Cable adapter (jobs run in the server process there), which drops the development cable database.

Unit suite (582 runs) and RuboCop pass locally on Ruby 4.0.7. Production db:prepare creates all four databases and a Solid Cable broadcast is persisted.


Generated by Claude Code

@thomasklemm
thomasklemm marked this pull request as ready for review October 9, 2026 13:24
Copilot AI balanced review requested due to automatic review settings October 9, 2026 13:24
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T18:05:28.514481Z 775adcd New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The migration command uses an invalid Resque API, and the benchmark cannot reliably start either dependency set or prepare the new databases.

3 open findings
What changed in this PR

Replaces Redis-backed cache, jobs, and Action Cable with dedicated SQLite-backed Solid adapters.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

Changes:

  • Adds Solid Cache, Queue, and Cable databases and configuration.
  • Removes Redis, Resque, and Kredis runtime dependencies.
  • Updates startup, restore, benchmarking, and deployment documentation.
File Description
test/​scripts/​solid_cache_restore_test.rb Tests removal of cache and cable databases.
test/​scripts/​post_restore_test.rb Tests primary restore and queue cleanup.
Procfile Replaces Redis/Resque processes with Solid Queue.
lib/​tasks/​resque.rake Removes Resque lifecycle tasks.
lib/​rails_ext/​sqlite_wal_checkpoint.rb Generalizes fork documentation.
hooks/​post-restore Clears disposable databases during restore.
Gemfile.lock Locks Solid adapter dependencies.
Gemfile Replaces Redis dependencies with Solid adapters.
docs/​upgrades/​solid-queue.md Adds Solid Queue migration guidance.
docs/​upgrades/​solid-cache.md Adds Solid Cache migration guidance.
docs/​upgrades/​solid-cable.md Adds Solid Cable migration guidance.
docs/​self-hosting.md Updates backup, restore, and upgrade instructions.
docs/​development.md Removes development Redis instructions.
Dockerfile Removes the Redis package.
db/​schema.rb Normalizes schema formatting.
db/​queue_schema.rb Adds Solid Queue schema.
db/​cache_schema.rb Adds Solid Cache schema.
db/​cable_schema.rb Adds Solid Cable schema.
config/​resque-pool.yml Removes Resque pool configuration.
config/​redis.conf Removes Redis configuration.
config/​recurring.yml Schedules queue record cleanup.
config/​queue.yml Configures Solid Queue processes.
config/​initializers/​sqlite_wal_checkpoint.rb Updates fork lifecycle documentation.
config/​initializers/​solid_queue.rb Manages checkpoint threads around queue forks.
config/​initializers/​resque.rb Removes Resque initialization.
config/​environments/​production.rb Enables Solid Cache and Queue.
config/​environments/​development.rb Uses in-memory development caching.
config/​database.yml Defines dedicated adapter databases.
config/​cache.yml Configures Solid Cache retention and size.
config/​cable.yml Enables Solid Cable outside development/test.
bin/​start-app Moves database preparation out of web startup.
bin/​setup Removes Redis setup.
bin/​jobs Adds the Solid Queue executable.
bin/​boot Prepares databases before launching services.
bench/​README.md Documents the frozen Redis benchmark baseline.
bench/​message_hot_paths.rb Generalizes adapter isolation wording.
bench/​compare_http.rb Adjusts baseline Redis environment handling.
app/​models/​room.rb Generalizes cache-store documentation.
.github/​workflows/​ci.yml Removes Redis CI services.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread bench/compare_http.rb Outdated
environment_variables = { RAILS_ENV: "production", SECRET_KEY_BASE: "isolated-benchmark-fixture-key", DISABLE_SSL: true,
SKIP_TELEMETRY: true, RAILS_LOG_LEVEL: "fatal", WEB_CONCURRENCY: 1, JOB_CONCURRENCY: 1, RAILS_MAX_THREADS: 5
}
environment_variables[:REDIS_URL] = "redis://#{redis}:6379/0" if side == "before"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9300e44: the driver accepts --baseline-image for the before side (defaulting to --image), and the bench README notes when to pass it.


Generated by Claude Code

Comment thread bench/compare_http.rb Outdated
Comment on lines 50 to 51
command.concat environment(environment_variables)
command.concat [ options[:image], "bundle", "exec", "puma", "-C", "config/puma.rb" ]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9300e44: the after container now runs bin/rails db:prepare before execing Puma, so a frozen seed gets the cache, cable and queue databases. The baseline side still starts Puma directly to stay frozen.


Generated by Claude Code

Comment thread docs/upgrades/solid-queue.md Outdated
require "resque"
info = Resque.info
puts "pending=#{info[:pending]} working=#{info[:working]} failed=#{info[:failed]}"
Resque.queue_sizes.each { |queue, size| puts "#{queue}=#{size}" }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is: Resque.queue_sizes is defined in Resque 2.7.0 (lib/resque.rb, "Returns a hash, mapping queue names to queue sizes"), pipelining LLEN over Resque.queues. The drain check works on the release being upgraded from.


Generated by Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ae38af30e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread hooks/post-restore Resolved
Comment thread config/cable.yml Resolved
Comment thread bench/compare_http.rb Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5f8a87c57

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread config/queue.yml Resolved
@thomasklemm
thomasklemm force-pushed the deps/solid-cable-removal branch 2 times, most recently from f4619a0 to 6277f5b Compare October 27, 2026 13:05
@thomasklemm
thomasklemm force-pushed the deps/solid-cable-removal branch from 6277f5b to 775adcd Compare November 5, 2026 17:56
@thomasklemm
thomasklemm force-pushed the deps/solid-cable-removal branch from 775adcd to 8b22b56 Compare November 18, 2026 15:52
@thomasklemm thomasklemm changed the title Remove Redis runtime dependency with Solid Cable Replace Redis with Solid Cable Nov 18, 2026
Rails.cache moves from Redis to Solid Cache, in its own SQLite database
that db:prepare creates from db/cache_schema.rb. Entries aren't copied, so
the cache starts cold after the upgrade.

The cache database keeps SQLite's own WAL checkpoints, since the
checkpoint contender only checkpoints the primary database. The restore
hook deletes it: a copy taken from a running volume can be torn, and the
cache refills on demand.

The HTTP benchmark prepares the databases before starting Puma, so a seed
made before a new database or migration still boots.
Jobs move from Resque in the container's Redis to Solid Queue in its own
SQLite database, so jobs queued when the container stops are still there
when it starts again. Redis doesn't persist (save "", appendonly no), so
jobs still queued in Resque at the upgrade are lost, as on any restart.

The queue database is backed up and restored with the primary. A room
deletion or banned user's content removal still queued at the backup
would otherwise never run after a restore. The two databases are copied
one after the other and the app writes them in separate commits, so a
job enqueued while the backup runs can be missing from it: a room deleted
in that moment comes back closed and without members, its messages left.
A job running at the backup comes back claimed by its process, and fails
once that process is pruned as dead.

bin/boot prepares the databases before starting the web server and the
workers, which now start together and both need the queue's tables.
Solid Queue forks its workers from a supervisor, so the supervisor stops
its checkpoint contender before forking and each worker starts its own,
as Puma's workers do. Workers poll every 0.1 s, as Resque did.

The queue's pool is five, for the worker's three threads plus its poller
and heartbeat, independent of the web server's RAILS_MAX_THREADS.

The test environment gets a queue database, so the backup test can use
real Solid Queue records; jobs still run on the test adapter.

The benchmark drivers take --baseline-image, since a baseline from before
this change needs Resque's gems, which the current image no longer has,
and prepare the databases before starting Puma, so a seed made before
the queue database existed still boots.
With the cache and the jobs on Solid Cache and Solid Queue, Action Cable
was the last thing using Redis. It moves to Solid Cable, in its own SQLite
database, and Redis goes: the server process, the redis and kredis gems,
the Debian package, bin/setup's container and CI's service.

Development uses Solid Cable too, rather than the async adapter, because
bin/dev runs several Puma workers and a broadcast has to reach sockets
held by the others. Development caching uses the memory store.

Broadcasts are transient, so the restore hook deletes the cable database
with the cache's and db:prepare recreates both.
@thomasklemm
thomasklemm force-pushed the deps/solid-cable-removal branch from 8b22b56 to 4540f47 Compare November 20, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants