Upgrade hotcell to 0.6.0 - #3145
Conversation
Pin the app's and the cell's hotcell gems to 0.6.0, and pin the accessory to the cell image this tree builds. `image_processing` moves to 2.2.0 in the cell's bundle, because `activestorage-hotcell-server` 0.6.0 requires it.
Since the upgrade to 0.6.0, `HotCell::LogSubscriber` and our `Yabeda::HotCell` both logged each HotCell call, so each call appeared twice in the log. Replace `Yabeda::HotCell` with the `yabeda-hotcell` gem, which records the same metrics and writes no log line.
hotcell 0.6.0 ships `HotCell::HealthController`, `HotCell::DiagnosticsController`, and the `health.echo` and `health.reopen` operations, which do what our `HotcellzController`, `Fizzy::Saas::Cell.diagnostics` and copied example operations did. Route `/hotcellz` and `/hotcellz/test` to the gem's controllers, make `AdminController` the diagnostics controller's parent, and load the operations from `hot_cell/health_operations`. `/hotcellz/test` now redirects a signed-out caller to sign in instead of answering 403. Its JSON nests each cell's checks under `cells` and no longer includes `root`.
`activestorage-hotcell-client` loaded its ImageMagick transformer even when unused, so the bundle needed `mini_magick` though nothing called it. Since 0.6.0 the gem loads that transformer only when the app references it, so remove `mini_magick`.
🤖 Upgrade Plan: hotcell v0.4.1..v0.5.0Upgrade Plan: hotcell v0.4.1..v0.5.0 for fizzy
Summary
The upgrade needs no Fizzy code change. Fizzy pins every hotcell gem exactly at Commits Requiring Mitigation849f358c: Sweep a killed request's tree from a supervisor-forked sweeper (hotcell#63)Impact: unlikely impact What changed (from the v0.4.1..v0.5.0 diff):
Matched in:
No matches for Mitigation:
Override auditNone of Fizzy's hotcell surface depends on what the range changed. The range changed
The in-range assessments contain no CVE or GHSA identifiers. No Impact (Skipped)
|
🤖 Upgrade Plan: hotcell v0.5.0..v0.6.0Upgrade Plan: hotcell v0.5.0..v0.6.0 for fizzy
Fizzy is on 0.4.1, not 0.5.0: Summary
The upgrade is not a lockfile bump. Without code changes, fizzy will log every HotCell call twice, and the Tasks
DecisionsD1: adopt
|
Fizzy hotcellz#test |
Upstream HotCell::DiagnosticsController#show |
|
|---|---|---|
| Body | flat {at, host, root, describe, metrics, echo, reopen} |
{at, host, healthy, cells: {"active_storage" => {describe, metrics, echo, reopen}}} |
| Round-trip result | {bytes, staged, echoed: true} |
{bytes, staged} |
| Operations | example.echo, example.reopen |
health.echo, health.reopen |
| Cell off | "HOTCELL_ROOT is unset, so no cell is configured" | "no socket directory, so this cell is off" |
| Wrong bytes | "returned N bytes, not the M sent" | "returned other bytes than it was sent" |
| Exceptions | message as-is | Failure.one_line |
| Auth | signed out → 403, non-staff → 403 (ensure_staff_access) |
none by default (parent ActionController::Base, cells.rb:88-92) |
Both use the cell's timeout and control_timeout, and neither caches. Fizzy loses root and gains
healthy. The probe operation rename also changes the operation label on probe calls in
hotcell_requests_total and hotcell_perform_seconds.
Upstream offers these ways to authenticate the diagnostics route, plus a third that keeps fizzy's controller:
HotCell.diagnostics_controller_parent = "AdminController".AdminControllergives staff-only access, but a signed-out caller gets a 302 to login instead of 403. Fizzy's controller comment rejects that on purpose ("a prober wants an answer"), andsaas/test/controllers/hotcellz_controller_test.rb:30asserts the 403. A development reload ofAdminControlleralso does not reach the parent.- Subclassing
HotCell::DiagnosticsController. Its parent isActionController::Base, soCurrent.identityis never resumed and fizzy's staff check always answers 403, unless the subclass re-includes the authentication concern. - Recommended: keep
HotcellzController, its routes and its auth, and haveshow/testcallHotCell.diagnose/HotCell.diagnose(work: true)andhealthy?. This keeps the path and the 403 and gains the quiet poll. DeleteFizzy::Saas::Cell.diagnostics,round_tripand theEcho/Reopenclients (saas/lib/fizzy/saas/cell.rb:79-163). The response body changes to upstream's nested shape.
The tests that change with any of these:
- In
saas/test/controllers/hotcellz_controller_test.rb:- the
Fizzy::Saas::Cell.echostub at:22 - the flat keys at
:50 - the four checks and "HOTCELL_ROOT is unset" at
:75-76 parsed_body["echo"]at:85-86,91
- the
saas/test/lib/cell_test.rb:26-45goes with the code it tests.
Rollout window
The pre-deploy hook reboots the cell before the app. A 0.4.1 app against a 0.6.0 cell without
example.* gets unsupported on /hotcellz/test until the app is replaced. Only the diagnostic is
affected. The wire protocol is version 1 in both releases. The 0.4.1 app also logs no skew warning,
because the warning is a 0.6.0 client feature.
Commits Requiring Mitigation
2ce02743: Log every HotCell call to the Rails log by default (hotcell#78)
Impact: likely impact
Matched in:
saas/lib/yabeda/hot_cell.rb:52:ActiveSupport::Notifications.subscribe "perform.hot_cell"saas/lib/yabeda/hot_cell.rb:78:::Rails.logger.info " HotCell (#{duration_ms}ms) " + labels.merge(...).to_json
The railtie's hot_cell.log_subscriber initializer attaches HotCell::LogSubscriber whenever Rails is
loaded. Fizzy loads it: saas/lib/fizzy/saas/cell.rb:5 requires hot_cell/client. Every call logs twice
until fizzy's line goes. The two lines have the same prefix, the same level (info) and the same destination (Rails.logger).
Upstream's line differs from fizzy's
(log_subscriber.rb):
- Upstream drops nil fields. Fizzy always writes
"cause":null,"stderr":nulland"bytes_in":null. - Upstream adds
exception(the class name). For an escaped exception it omitscode, where fizzy writes"code":"ok". - Upstream moves
stderrto the end. - Upstream encodes with
JSON.generate(..., ascii_only: true), which escapes non-ASCII. ASto_jsonescapes< > &and U+2028/2029. Only the rendering ofstderrchanges.
Log queries that match on a literal "cause":null or rely on field order would break.
Mitigation: T4 (delete fizzy's file). To keep fizzy's line instead, add HotCell::LogSubscriber.detach_from :hot_cell to an initializer.
a55a6a24: Ship Yabeda metrics as the yabeda-hotcell gem (hotcell#77)
Impact: unlikely impact
Matched in: saas/lib/yabeda/hot_cell.rb (whole file; group :hotcell at :8, perform.hot_cell at :52, HotCell.cells.each_value at :35); saas/lib/fizzy/saas/engine.rb:174-175.
Mitigation: D1 and T4.
fc98fb2d: Ship health and diagnostics controllers for registered cells (hotcell#70)
Impact: unlikely impact (opt-in)
Matched in: saas/config/routes.rb:10-11; saas/app/controllers/hotcellz_controller.rb:6-33; saas/lib/fizzy/saas/cell.rb:79-152.
Mitigation: D2. Do not route HotCell::DiagnosticsController with its default parent, because every request takes a worker and the response exposes the hostname and cell description.
ea48cfc2: Ship the two health probes instead of leaving them as examples (hotcell#39)
Impact: unlikely impact
Matched in:
saas/hotcell/operations/echo.rb:3,7-8: copied from the now-deletedexamples/operations/echo.rb, answers toexample.echosaas/hotcell/operations/reopen.rb:3,9-10: copied fromexamples/operations/reopen.rb, answers toexample.reopensaas/lib/fizzy/saas/cell.rb:157,162: the client classes for those names
The perform bodies and {bytes:, staged:} results match
health_operations.rb
exactly. Only the names differ. The copies keep working under example.*. A copy renamed to health.*
while the require is present raises HotCell::ConfigurationError at boot.
Mitigation: T5. Remove the copies and the require in the same change.
1ff4726f: Require image_processing 2.2.0 and drop the transform rename (hotcell#82)
Impact: unlikely impact
Matched in: saas/hotcell/Gemfile.lock:6,23: image_processing (>= 2.1.0) / image_processing (2.1.0).
activestorage-hotcell-server now requires image_processing >= 2.2.0, and transforms write directly to
the final path. Fizzy's saas/hotcell/operations/active_storage.rb:6-18 only requires files and calls
Vips.block on loaders, so the change does not touch it. The one behavior change, an ImageMagick format with no name such as jfif now failing as
unreadable, does not apply to fizzy, which uses Vips. config/initializers/vips.rb configures app-side
Vips only when no cell is enabled and is unaffected.
Mitigation: T2.
be7b286a: Stop requiring mini_magick for apps on the Vips transformer (hotcell#74)
Impact: unlikely impact
Matched in: Gemfile.saas:16: gem "mini_magick", require: false # activestorage-hotcell-client loads the ImageMagick transformer even when unused; saas/lib/fizzy/saas/cell.rb:71 uses only Transformers::Image::Vips.
Transformers::Image::Magick is now an autoload, so the comment no longer holds. Nothing else in
Gemfile.saas.lock depends on mini_magick. The cell still gets mini_magick through
activestorage-hotcell-server's own dependency.
Mitigation: T6.
163c62c1: Warn at boot when a cell runs another hotcell release (hotcell#90)
Impact: unlikely impact
Matched in: saas/lib/fizzy/saas/engine.rb:72 (::HotCell.describe_cells in after_initialize); saas/lib/fizzy/saas/cell.rb:85 (cell.describe on every /hotcellz poll).
A 0.4.1 cell reports no server_version, so a 0.6.0 app warns against it at boot and, through
cell.rb:85, on every health poll. Upstream provides no option to turn the warning off.
Mitigation: deploy client and cell on the same release (T1-T3, T8). D2's recommended option removes the per-poll warning.
cc8dc2ed: Pre-fork workers so requests don't wait on fork (hotcell#67)
Impact: likely impact
Matched in: saas/hotcell/config.rb:6 (concurrency: 4); saas/config/deploy.yml:93-110 (memory: 2g, memory-swap: 2g, pids-limit: 512).
The supervisor now forks 4 idle workers at boot and runs Process.warmup first. The accessory memory
is already sized from "concurrency × peak RSS" (deploy.yml:93-94), so 4 idle workers sit below the
sized ceiling, and 6 processes is far below 512 pids. hotcell_running counts busy children only, so
idle workers do not raise it. On the client, a broken pipe while sending to a full cell now returns
capacity instead of unavailable. Both codes are transient, so ProcessingUnavailable still raises.
Saturation moves from code="unavailable" to code="capacity" in hotcell_requests_total.
Mitigation: none in code. After deploy, check idle cell memory, and check any alert keyed on code="unavailable" as the saturation signal.
ffc92154: Release v0.6.0
Impact: unlikely impact
Matched in: saas/hotcell/Gemfile:12-13, Gemfile.saas:14-15 (exact "0.4.1" pins).
Mitigation: T1-T3, T8.
Analyzed, No Action
| Commit | Summary | Impact Level | Why no action |
|---|---|---|---|
c8e10ac8 (#73) |
hotcell.describe adds server_version |
unlikely impact | engine.rb:72 discards the result; no test compares the full describe hash |
6263bcbf (#91) |
macOS supervisor rescues Errno::EPERM killing a zombie group |
unlikely impact | No match; production is Linux |
849f358c (#63, v0.4.1..v0.5.0) |
Supervisor forks a sweeper every sweep_interval (10s) |
unlikely impact | One extra process against pids-limit: 512; config.rb needs no sweep_interval |
Override audit
All findings are folded into D1, D2 and the commits above. In summary:
saas/lib/yabeda/hot_cell.rbis redundant, with metrics identical to upstream's. Onceyabeda-hotcellis installed it is broken by the load-path clash, and it double-logs from 0.6.0 even without the gem.saas/test/lib/yabeda/hot_cell_test.rbis broken after the switch. Its log tests at:85-114test removed code, and:37looks upcause: :memorywhere the gem sets"memory".HotcellzControllerandFizzy::Saas::Cell.diagnosticsare a superset in auth only. Their checks are the same asHotCell.diagnose. See D2.saas/hotcell/operations/echo.rbandreopen.rbare redundant copies ofhot_cell/health_operationsunder other names.Gemfile.saas:16(mini_magick) is redundant.saas/hotcell/bin/buildis insufficient for this bump. See T1.saas/hotcell/config.rb,saas/hotcell/Dockerfile(theHEALTHCHECKstill uses the shippedhotcell-health),config/initializers/vips.rb,saas/.kamal/hooks/*,saas/Procfile.dev,bin/devandsaas/bin/setupare not at risk.
No CVE or GHSA identifiers appear in the range's assessments. That is not evidence of no security fixes.
No Impact (Skipped)
11 commits assessed as "no impact" during recon, not analyzed against the app:
805dde1693f4afb2d467bbcc715602b2079e811533559aa56c775e26d267bcd1a005bbb8879be9a3f1618942
These are version bumps, CI, docs, dev scripts, test helpers, and the root lockfile.
Transitive Dependency Upgrades
| Gem | From | To | Commits | Mitigations | Plan |
|---|---|---|---|---|---|
| image_processing (cell bundle only) | 2.1.0 | 2.2.0 | 4 | 0 | fizzy-20261001-image_processing_v2.1.0..v2.2.0.md |
| yabeda-hotcell (new, app bundle) | — | 0.6.0 | — | — | covered by a55a6a24 above |
| mini_magick (removed from app bundle) | 5.3.3 | — | — | — | covered by be7b286a above |
No other gem moved in Gemfile.saas.lock or saas/hotcell/Gemfile.lock.
Execution (2026-10-01, branch card-5280-hotcell-0-6-0)
- T1–T3, T8:
Upgrade hotcell to 0.6.0. T8 rebuilt the cell image;saas/config/deploy.ymlpinsd8c4c8004de0at the head of the branch. - T4, T7 (build help):
Record HotCell metrics with the yabeda-hotcell gem. - T5, T7 (README):
Serve /hotcellz from hotcell's health controllers. D2 executed as option 1, not the recommended option 3: the release notes ask apps to replace their health endpoints with the gem's controllers, so/hotcellzroutes toHotCell::HealthControllerand/hotcellz/testtoHotCell::DiagnosticsControllerwithHotCell.diagnostics_controller_parent = "AdminController". A signed-out caller of/hotcellz/testis redirected to sign in instead of getting 403; the PR flags this for review. - T6:
Drop mini_magick from the SaaS bundle. - T9: a dev cell booted from the branch registers
health.echoandhealth.reopen;HotCell.diagnose(work: true)passes all four checks; each call writes oneHotCell (…ms)line; a vips variant renders through the cell. The built image boots with the same operation list.
🤖 Transitive Upgrade Plan: image_processing v2.1.0..v2.2.0Upgrade Plan: image_processing v2.1.0..v2.2.0 for fizzy
ScopeThis gem moves only in the cell's lockfile, pulled by The cell loads Summary
The upgrade is safe. The two code commits change what happens when a pipeline with Commits Affecting the Cell5827e836: Save in the
|
| Commit | Summary |
|---|---|
| 8f7b5383 | Skip AVIF vips tests when no encoder is present; add libheif-plugin-aomenc to CI. No lib/ change. |
| d9737fbf | Version bump to 2.2.0 and CHANGELOG entry. |
Override audit
Override surface for image_processing in the cell: saas/hotcell/operations/active_storage.rb
(sets Transformers::Image::Vips.limits and Vips.block for openslide and tiff) and
saas/hotcell/config.rb (HotCell.limits). Neither touches an API the range changed.
In activestorage-hotcell-server v0.6.0, no code overrides Processor.call or save_image.
transformers/image/vips.rb reopens the output with Vips::Image.new_from_file, which sniffs
content rather than extension, so the extensionless output describes correctly.
Residual delta: none for fizzy. The only workaround the range made redundant (the
extension-and-rename in transforming.rb) was already removed in hotcell 1ff4726f. Any
leftover Destination#path(extension:) / Destination#adopt helpers are hotcell's cleanup.
Advisories
No CVE- or GHSA- identifiers in any assessment. This is not evidence of absence; the range is
a behavior fix, a test fix and a release bump.
Plan
- Accept the transitive bump as part of the
activestorage-hotcell-server 0.6.0upgrade;
nothing to change in app code. - Verification is the cell's existing vips transform coverage: a variant request must return
bytes in the requested format. A failure there on 2.2.0 would point at this range.
There was a problem hiding this comment.
Copilot review overview
🟢 Approved
The upgrade consistently adopts the documented 0.6.0 APIs, preserves authorization and telemetry, and keeps the deployment pin synchronized.
Review effort: Balanced
Findings: None
What changed in this PR
Upgrades HotCell to 0.6.0 and replaces Fizzy’s duplicated diagnostics, telemetry, and health operations with gem-provided implementations.
Changes:
- Upgrades application and cell dependencies to HotCell 0.6.0.
- Adopts packaged health controllers, operations, logging, and Yabeda metrics.
- Updates tests, documentation, routes, and deployment image pin.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
| File | Description |
|---|---|
Gemfile.saas |
Adds the upgraded clients and Yabeda integration. |
Gemfile.saas.lock |
Locks application dependencies to 0.6.0. |
saas/README.md |
Documents the gem-provided diagnostics behavior. |
saas/app/controllers/hotcellz_controller.rb |
Removes the superseded controller. |
saas/config/deploy.yml |
Pins the verified upgraded cell image. |
saas/config/routes.rb |
Routes checks to HotCell controllers. |
saas/hotcell/Gemfile |
Upgrades cell server dependencies. |
saas/hotcell/Gemfile.lock |
Locks server and image-processing upgrades. |
saas/hotcell/bin/build |
Includes yabeda-hotcell in upgrade instructions. |
saas/hotcell/operations/echo.rb |
Removes the copied echo operation. |
saas/hotcell/operations/health.rb |
Loads packaged health operations. |
saas/hotcell/operations/reopen.rb |
Removes the copied reopen operation. |
saas/lib/fizzy/saas/cell.rb |
Removes superseded diagnostics implementation. |
saas/lib/fizzy/saas/engine.rb |
Configures diagnostics authorization inheritance. |
saas/lib/yabeda/hot_cell.rb |
Removes the duplicated telemetry integration. |
saas/test/controllers/hotcellz_controller_test.rb |
Removes tests for the deleted controller. |
saas/test/integration/hotcellz_test.rb |
Tests the replacement health endpoints. |
saas/test/lib/cell_test.rb |
Removes tests for deleted diagnostics helpers. |
saas/test/lib/hotcell_telemetry_test.rb |
Verifies logging and metrics remain singular. |
saas/test/lib/yabeda/hot_cell_test.rb |
Removes tests now owned by the gem. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Looks good in staging. |
Motivation
This upgrades hotcell from 0.4.1 to 0.6.0 (v0.5.0 and v0.6.0 release notes). The 0.6.0 gems ship code that Fizzy wrote for itself, and the release's upgrading notes say to delete the application's own copies:
Yabeda::HotCellmoves into the newyabeda-hotcellgem.HotCell::LogSubscriber./hotcellzendpoints move intoHotCell::HealthControllerandHotCell::DiagnosticsController.hot_cell/health_operations.Item 2 is not optional. The 0.6.0 railtie attaches
HotCell::LogSubscriber, and our subscriber also writes that line, so without this change every call is logged twice.Details
Upgrade hotcell to 0.6.0pins the app's and the cell's gems to 0.6.0. In the cell's bundle,image_processingmoves to 2.2.0, whichactivestorage-hotcell-server0.6.0 requires. The accessory is pinned to cell imaged8c4c8004de0.Record HotCell metrics with the yabeda-hotcell gemdeletessaas/lib/yabeda/hot_cell.rb. The gem defines the same metrics with the same names, tags and buckets, so dashboards and alerts are unchanged. The gem sets akilledgauge for every cause on the first scrape, where ours set one only after that cause's first kill. The log line now omits null fields, addsexceptionwhen one escapes the call, and escapes non-ASCII.Serve /hotcellz from hotcell's health controllerskeeps both paths./hotcellzbehaves as before./hotcellz/testis nowHotCell::DiagnosticsControllerwithAdminControlleras its parent, with these effects:cells.active_storageand addshealthy. It no longer includesroot.health.echoandhealth.reopen, which is now theiroperationlabel inhotcell_requests_total.AdminControllerdoes not reach the diagnostics controller, which resolves its parent once, when it loads. After changingAdminController's filters, restart the server for them to apply there.Drop mini_magick from the SaaS bundleremoves the gem. Fizzy needed it only becauseactivestorage-hotcell-client0.4.1 loaded its ImageMagick transformer unconditionally.Additional information
The pre-deploy hook reboots the cell before the app. Until the app containers are replaced, a 0.4.1 app gets
unsupportedfrom/hotcellz/test, because the 0.6.0 cell no longer hasexample.echoorexample.reopen. Conversions keep working, because both releases use wire protocol version 1.A client that hits a full cell now records
capacityin cases where it used to recordunavailable(see the v0.6.0 notes). Addcapacityto any alert that usesunavailableas the saturation signal.Cell image
d8c4c8004de0is built locally only. Thepre-buildkamal hook publishes it on the next deploy.The upgrade plans are in the 🤖 comments below and in 37signals-hq: v0.4.1..v0.5.0, v0.5.0..v0.6.0, and image_processing v2.1.0..v2.2.0.