Skip to content

Upgrade hotcell to 0.4.1 - #3108

Merged
flavorjones merged 4 commits into
mainfrom
upgrade-hotcell-0-4-0
Sep 8, 2026
Merged

flavorjones merged 4 commits into
mainfrom
upgrade-hotcell-0-4-0

Conversation

@flavorjones

@flavorjones flavorjones commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

Upgrade hotcell from 0.3.1 to 0.4.1 (changelog). 19 commits analyzed (16 in v0.3.1..v0.4.0, 3 in v0.4.0..v0.4.1), 5 needing attention, 2 transitive gems in the cell's lockfile analyzed with no impact.

Basecamp card: https://app.basecamp.com/2914079/buckets/1666/card_tables/cards/10281760225

Changes

  • Gemfile.saas and saas/hotcell/Gemfile pinned to exactly 0.4.1; cell image rebuilt and saas/config/deploy.yml pinned to 2f4ddc90eae3. The cell's lockfile moves mini_magick to 5.4.0 and image_processing to 2.1.0 to meet the new activestorage-hotcell-server floors.
  • saas/lib/yabeda/hot_cell.rb writes the new stderr payload field on the existing HotCell log line (hotcell#60). It stays out of the Yabeda labels. Closes the stderr-logging item on Fizzy card 5270 and this Basecamp card.
  • saas/Procfile.dev boots the dev cell with hotcell --development (hotcell#61, new in 0.4.1). The 0.4.0 supervisor empties the scratch at boot (hotcell#52), which for a plain-process cell with no TMPDIR of its own meant the developer's /tmp; the first cut of this PR worked around that with a hand-made TMPDIR, and 0.4.1 replaces the workaround with the flag. With it the cell keeps its scratch in a directory of its own under the system temporary directory and never sweeps the directory it was given. Without the flag 0.4.1 behaves exactly like 0.4.0, so the production accessory (dedicated scratch mounted at /tmp, no flag) is unchanged.

ImageMagick resource limits: not needed

hotcell#59 forwards MAGICK_*_LIMIT to the magick child, which only matters in a cell that runs an ImageMagick operation. Fizzy's cell does not: saas/hotcell/operations/active_storage.rb requires the vips, ffprobe, mutool and ffmpeg operation files one at a time and never the gem entry point that loads the magick ones, saas/hotcell/Dockerfile installs no ImageMagick package, and magickload stays blocked in libvips. That is unchanged across the range. A dev cell booted on 0.4.0 registers exactly active_storage.analyzers.image.vips, active_storage.analyzers.media.ffprobe, active_storage.previewers.pdf.mutool, active_storage.previewers.video.ffmpeg, active_storage.transformers.image.vips plus the two example operations.

After merge

The cell image 2f4ddc90eae3 is built locally only. The pre-build kamal hook publishes it on the next deploy, and pre-deploy reboots the accessory, so a normal bin/kamal deploy ships both.

Upgrade plan

Full analysis is in the 🤖 comments below and at 37signals-hq/upgrade-analysis/fizzy-20260908-hotcell_v0.3.1..v0.4.0.md and fizzy-20260908-hotcell_v0.4.0..v0.4.1.md.

A killed worker left its tool's scratch files at the top of the cell's
`/tmp`, outside the slot tree that is removed when a request ends, so
the scratch volume filled over time. A worker now points `TMPDIR` at
the request's home, and the supervisor empties the scratch at boot.

That boot sweep is why the dev cell in `saas/Procfile.dev` now gets a
`TMPDIR` of its own: unset, it would sweep the developer's `/tmp`.

The release's ImageMagick changes do not reach this cell, which loads
only the vips, ffprobe, mutool and ffmpeg operations and installs no
ImageMagick, so the `MAGICK_*_LIMIT` variables stay unset.

ref: https://github.com/basecamp/hotcell/blob/v0.4.0/CHANGELOG.md
A crash's diagnosis — `libgomp: Thread creation failed` — survived only
in the exception's message, so a failure that was discarded rather than
retried lost it. hotcell 0.4.0 carries the captured stream on the
`perform.hot_cell` event; write it as a field on the existing log line.

ref: basecamp/hotcell#60
Copilot AI balanced review requested due to automatic review settings September 8, 2026 15:20
@flavorjones

Copy link
Copy Markdown
Member Author

🤖 Upgrade Plan: hotcell v0.3.1..v0.4.0

Upgrade Plan: hotcell v0.3.1..v0.4.0 for fizzy

  • Date: 2026-09-08
  • Gem: hotcell (hotcell-client, activestorage-hotcell-client in Gemfile.saas; hotcell-server, activestorage-hotcell-server in saas/hotcell/Gemfile)
  • Range: v0.3.1..v0.4.0 (16 commits, a38c8e95..919491de)
  • Target: fizzy at /home/flavorjones/Work/basecamp/fizzy--upgrade-hotcell-0-4-0

Summary

  • Total commits: 16
  • No impact: 9 (skipped at recon)
  • Analyzed, not affected: 3
  • Requires mitigation: 4 (one is a one-line Rails change; the rest are bundle/dev-env housekeeping)

Answers to the two questions asked up front:

  1. Fizzy's cell runs no ImageMagick operation after the bump. saas/hotcell/operations/active_storage.rb:5-9 requires the vips, ffprobe, mutool and ffmpeg operation files one at a time, deliberately bypassing the gem entry point active_storage/hot_cell/server.rb, which is the only file that loads magick_operation, transformers/image/magick and analyzers/image/magick. That require list is unchanged between v0.3.1 and v0.4.0. saas/hotcell/Dockerfile installs libvips42 mupdf-tools ffmpeg, no ImageMagick package; libvips' magickload delegate is blocked by image_processing/vips as loaded by vips_operation.rb. MAGICK_MEMORY_LIMIT / MAGICK_MAP_LIMIT / MAGICK_DISK_LIMIT are not needed (docs/IMAGEMAGICK.md, PR Bump selenium-webdriver from 4.22.0 to 4.24.0 #59: they apply to a cell image that installs ImageMagick).
  2. Log stderr by adding one key to the existing subscriber in saas/lib/yabeda/hot_cell.rb (log_perform, lines 78-86), which already writes a JSON log line with explicitly picked payload keys. Add stderr: event.payload[:stderr] to that hash. Do not add it to the Yabeda labels in record_perform (lines 64-70). Details below under 460a8e1b.

Also: the mini_magick line in Gemfile.saas:16 still applies at v0.4.0 (the client entry point is byte-identical except the version string and still unconditionally requires the magick transformer). Only the 0.3.1 in its comment goes stale.

Execution order

  1. BUNDLE_GEMFILE=Gemfile.saas bundle update --conservative hotcell-client activestorage-hotcell-client hotcell-core (per saas/hotcell/bin/build --help), after bumping the ~> 0.3.1 pins in Gemfile.saas:14-15 and saas/hotcell/Gemfile:12-13 to ~> 0.4.0. Update the comment on Gemfile.saas:16 to say 0.4.0 (or drop the version from it).
  2. saas/hotcell/bin/build — relocks saas/hotcell/Gemfile.lock and pins saas/config/deploy.yml. The cell relock must also pull mini_magick 5.3.3 → ≥ 5.4.0 and image_processing 2.0.3 → ≥ 2.1.0 (new activestorage-hotcell-server gemspec floors); if --conservative refuses, add both to the update list. saas/test/lib/hotcell_lockfiles_test.rb asserts both lockfiles name the same hotcell-core.
  3. saas/lib/yabeda/hot_cell.rb: add stderr to the log line; extend perform_event and add an assertion in saas/test/lib/yabeda/hot_cell_test.rb.
  4. saas/Procfile.dev: set TMPDIR (and optionally HOTCELL_WORKSPACE) for the cell process so the boot sweep does not empty the developer's /tmp.
  5. Run saas/test, then saas/test/hotcell-check-test / saas/hotcell/bin/check against the built image. Commit lockfiles, deploy.yml pin, and code together as bin/build instructs.

Commits Requiring Mitigation

4eda0bc7: Empty the scratch at supervisor boot (#52)

Impact: definite impact
Matched in:

  • saas/Procfile.dev:4 — cell process sets HOTCELL_DIR=$PWD/tmp/hotcell/active_storage but no TMPDIR or HOTCELL_WORKSPACE. The supervisor now deletes every top-level entry of Dir.tmpdir (system /tmp here) and of the workspace's parent that its uid owns, at every boot. In development that is the developer's own /tmp files.
  • saas/config/deploy.yml:90 — /var/lib/hotcell-scratch:/tmp; saas/hotcell/Dockerfile:48,53 — HOME=/tmp, HOTCELL_DIR=/run/hotcell/cell. Production matches the reference layout: absolute, normalized, socket dir outside the scratch. The sweep is the intended behavior there.

Mitigation:

  • Development: add TMPDIR=$PWD/tmp/hotcell/tmp HOTCELL_WORKSPACE=$PWD/tmp/hotcell/tmp/workspace to the cell: line in saas/Procfile.dev, mirroring upstream examples/devcell (TMPDIR=<root>/tmp, HOTCELL_WORKSPACE=<root>/tmp/workspace). Keep HOTCELL_DIR outside it (it already is). Both paths must exist before boot — saas/bin/setup is the place to mkdir -p them; $PWD must be a normalized absolute path (a symlinked checkout now fails with ConfigurationError).
  • Production: nothing required. Optionally add the new WARN event scratch.unswept to whatever alerting watches slot.unswept.
  • No app code references Slot#remove_tree (moved to HotCell::Filesystem.remove_tree).

41288269: Read the ImageMagick operations' input through its descriptor (#54)

Impact: likely impact
Matched in:

  • saas/hotcell/Gemfile.lock — mini_magick (5.3.3), image_processing (2.0.3), both below the new activestorage-hotcell-server gemspec floors (mini_magick >= 5.4.0, image_processing >= 2.1.0).
  • Gemfile.saas.lock:355,406 — image_processing (2.0.3), mini_magick (5.3.3). The client gemspec carries no such floors, so the app lockfile need not move, but keeping both lockfiles at the same versions is simpler.

Mitigation:
Bump mini_magick and image_processing in the cell bundle with the hotcell bump (see execution order step 2). The behavior change itself (magick inputs no longer bounded by file_size, Transforming#pipeline signature change) does not reach fizzy: no Transforming includers, pipeline/source_path/identified overrides, and the magick operations are never loaded in fizzy's cell.


460a8e1b: Carry a failure's stderr on the perform.hot_cell event (#60)

Impact: unlikely impact (pattern search found no unsafe payload serialization), but the CHANGELOG "Upgrading" section asks client apps to log the field, and fizzy has the subscriber to do it.
Matched in:

  • saas/lib/yabeda/hot_cell.rb:52 — ActiveSupport::Notifications.subscribe "perform.hot_cell".
  • saas/lib/yabeda/hot_cell.rb:78-86 — log_perform builds the log hash from explicit picks (code, cause, perform_ms, duration_ms, bytes_in, bytes_out) and .to_jsons it. stderr is currently dropped.

Mitigation:

  • Add stderr: event.payload[:stderr] to the labels.merge(...) hash in log_perform. It lands as a JSON field on the existing HotCell (Nms) {...} line, which is exactly what upstream asks ("write it to a log field and interpolate it nowhere else"); Failure.sanitize already bounds it. Nil on success.
  • Do not add it to record_perform's Yabeda labels (saas/lib/yabeda/hot_cell.rb:64-70) — unbounded cardinality.
  • Test: add a stderr: keyword to perform_event (saas/test/lib/yabeda/hot_cell_test.rb:139-144) and a test beside "logs why a worker was killed" (line 103) asserting logged["stderr"].
  • No other subscriber exists (no lograge; Sentry deliberately not reported from here, per the comment at line 58).

ce25447d: Point a worker's TMPDIR at the request's home

Impact: unlikely impact
Matched in:

  • saas/hotcell/Dockerfile:48 — HOME=/tmp; slot homes are made under it, so per-request TMPDIR now points inside the same 4 GB scratch volume (saas/config/deploy.yml:90), one level deeper.

Mitigation:
None required. Capacity and mount flags are unchanged; scratch now lands in the slot tree and is removed with the request, which is an improvement for fizzy (killed vips workers no longer orphan files at the top of /tmp). Fizzy does not vendor the gate/battery examples' tool_env allowlist.


Override audit (3a-bis)

Override surface examined: saas/hotcell/config.rb, saas/hotcell/operations/{active_storage,echo,reopen}.rb, saas/hotcell/{Dockerfile,Gemfile,bin/*}, saas/lib/fizzy/saas/{cell,engine}.rb, saas/lib/yabeda/hot_cell.rb, saas/app/controllers/hotcellz_controller.rb, saas/config/deploy.yml, saas/Procfile.dev, saas/bin/setup, config/initializers/vips.rb, Gemfile.saas, and the saas/test hotcell tests.

Residual delta:

  • saas/lib/yabeda/hot_cell.rb — incomplete, not broken: see 460a8e1b.
  • Gemfile.saas:16 mini_magick workaround — still load-bearing at v0.4.0; comment's version number is stale after the bump.
  • saas/hotcell/config.rb — only HotCell.limits(...); hotcell-server/lib/hot_cell/configuration.rb has no diff in range.
  • echo.rb, reopen.rb — match upstream examples/operations/* which are unchanged; Descriptor#fd_path/#staged?/#to_io untouched.
  • saas/lib/fizzy/saas/cell.rb — registers only vips/ffprobe/mutool/ffmpeg; HotCell.register/HotCell.cell unchanged.
  • config/initializers/vips.rb — app-side only, guarded by Cell.enabled?; unrelated.
  • Nothing else is broken, redundant, or inert.

Advisories: the only identifier in the range is CVE-2026-80212 (e6d004f4, a .trivyignore.yaml for the resolv default gem in the base image, CI-only). Nothing to act on in fizzy; absence of identifiers is not evidence of absence.

Analyzed — No App Impact

Commit Summary Impact Level
903ecdb2 Every operation sets MAGICK_TMPDIR from TMPDIR per request unlikely impact — no MAGICK_TMPDIR set anywhere, no Operation#initialize overrides
b3361762 Stop passing create_additions to JSON.parse (json 3.0 fix) unlikely impact — json (2.21.2) in both app lockfiles; no Payload.parse/MessageError usage. Prerequisite for any future json 3.x bump
9c887ead Forward MAGICK_*_LIMIT to spawned magick; add docs/IMAGEMAGICK.md likely impact — cell image has no ImageMagick, no MAGICK_* vars, no MiniMagick.cli_env use, no MagickOperation subclasses

No Impact (Skipped)

9 commits assessed as "no impact" during recon, not analyzed against the app: 543e6b55 (version bump), 3273d4c3 (CONTRIBUTING), e6d004f4 (Trivy ignore, CI), d17aeff6 (test only), 67c51383 (merge), 502afe14 (examples battery), 22009fd7 (merge), 461afa4b (CONTRIBUTING), 919491de (release commit; its CHANGELOG "Upgrading" items are covered under 460a8e1b and question 1 above).

Transitive Dependency Upgrades

Both moved only in the cell's lockfile (saas/hotcell/Gemfile.lock), to meet the new activestorage-hotcell-server gemspec floors. The app lockfiles did not move.

Gem From To Commits Mitigations Plan
image_processing 2.0.3 2.1.0 2 0 fizzy-20260908-image_processing_v2.0.3..v2.1.0.md
mini_magick 5.3.3 5.4.0 2 0 fizzy-20260908-mini_magick_v5.3.3..v5.4.0.md

No security advisories in either range.

Execution record

Executed 2026-09-08 on branch upgrade-hotcell-0-4-0:

  1. Gemfile.saas and saas/hotcell/Gemfile pins to ~> 0.4.0; app bundle updated (no transitive movement); saas/hotcell/bin/build --platform=linux/amd64 relocked the cell and pinned saas/config/deploy.yml to 684eb866cdda.
  2. saas/lib/yabeda/hot_cell.rb logs stderr on the HotCell line; test added.
  3. saas/Procfile.dev gives the dev cell its own TMPDIR under tmp/hotcell/tmp, created on the same line. Not added to saas/bin/setup. The dev cell booted on 0.4.0 with only the vips, ffprobe, mutool and ffmpeg operations registered.
  4. bin/ci green (OSS suites); SAAS=true BUNDLE_GEMFILE=Gemfile.saas bin/rails test green (1846 tests).

@flavorjones

Copy link
Copy Markdown
Member Author

🤖 Transitive Upgrade Plan: image_processing 2.0.3..2.1.0

Upgrade Plan: image_processing v2.0.3..v2.1.0 for fizzy

  • Date: 2026-09-08
  • Gem: image_processing
  • Range: v2.0.3..v2.1.0 (2 commits)
  • Target: fizzy (hotcell cell bundle only — saas/hotcell/Gemfile.lock)

Scope

This gem moves only in the cell's lockfile, pulled by activestorage-hotcell-server 0.4.0
(image_processing >= 2.1.0, mini_magick >= 5.4.0). The app lockfiles (Gemfile.lock,
Gemfile.saas.lock) stay at 2.0.3 and are out of scope.

The cell loads image_processing/vips only, through
saas/hotcell/operations/active_storage.rb, which requires the gem's vips transformer and
analyzer files one at a time. image_processing/mini_magick (via the gem's
magick_operation.rb) is never required in this image.

Summary

  • Total commits: 2
  • No impact: 2 (skipped at recon)
  • Analyzed, not affected: 0
  • Requires mitigation: 0

Commits Requiring Mitigation

None.

Analyzed — No App Impact

None reached this stage; both commits were assessed "no impact" at recon.

No Impact (Skipped)

Commit Summary
722d900e Add opt-in inherit_fds: loader option to ImageProcessing::MiniMagick, forwarded to MiniMagick.convert; raises LoadError only when used on mini_magick < 5.4.0. Vips backend untouched.
3d990c3c Version bump to 2.1.0 and CHANGELOG entry.

The whole diff is lib/image_processing/mini_magick.rb plus docs and tests;
lib/image_processing/vips.rb is unchanged between the tags.

Override audit

Override surface for image_processing in the cell: saas/hotcell/operations/active_storage.rb
(sets Transformers::Image::Vips.limits and Vips.block for openslide and tiff). Neither
touches an API the range changed. config/initializers/vips.rb belongs to the app bundle
(2.0.3) and is out of scope.

Why the range exists at all: activestorage-hotcell-server 0.4.0 is the consumer of the new
option. Its transforming.rb appends source_loader_options(source) last in the pipeline;
the base implementation returns {} and only Transformers::Image::Magick overrides it to
pass inherit_fds:. The vips transformer and analyzer the cell loads never send
inherit_fds: to ImageProcessing::Vips, so the version floor is satisfied but no new code
path executes in Fizzy's cell. The >= 2.1.0 floor is therefore a gem-level requirement of
the magick operations the cell does not carry, not a behavior change for this app.

Residual delta: none. No patch is broken, redundant, or inert.

Advisories

No CVE- or GHSA- identifiers in either assessment. This is not evidence of absence; the
range is a feature addition and a release bump.

Plan

  1. Accept the transitive bump as part of the activestorage-hotcell-server 0.4.0 upgrade;
    nothing to change in app code.
  2. Verification is the cell's existing vips transform/analyze coverage; no image_processing
    specific test is warranted.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@flavorjones

Copy link
Copy Markdown
Member Author

🤖 Transitive Upgrade Plan: mini_magick 5.3.3..5.4.0

Upgrade Plan: mini_magick v5.3.3..v5.4.0 for fizzy

  • Date: 2026-09-08
  • Gem: mini_magick
  • Range: v5.3.3..v5.4.0 (2 commits, d9c2a0d7..daf541ba)
  • Target: fizzy (saas/hotcell/Gemfile.lock only; Gemfile.saas.lock stays at 5.3.3, Gemfile.lock has no mini_magick)

Summary

  • Total commits: 2
  • No impact: 2 (skipped at recon)
  • Analyzed, not affected: 0
  • Requires mitigation: 0

The whole release is one additive feature: MiniMagick::Shell#execute(inherit_fds: []),
which adds the listed IOs to the spawn map so the tool can read them as /dev/fd/N. With the
option omitted, spawn behaviour is byte-for-byte what 5.3.3 did. No removals, no default changes,
no advisory (neither assessment carries a CVE/GHSA identifier; that is not evidence of absence).

Why the cell moved

The bump is a hard floor, not a drift: activestorage-hotcell-server 0.4.0's gemspec requires
mini_magick >= 5.4.0 (and image_processing >= 2.1.0) because its two ImageMagick operations
now hand the input descriptor to magick/identify with inherit_fds: instead of staging a
copy (activestorage-hotcell-server/lib/active_storage/hot_cell/server/analyzers/image/magick.rb:40,
transformers/image/magick.rb:42; hotcell CHANGELOG.md:57). Bundler resolved 5.4.0 to satisfy
that; there is no fizzy-side choice to make.

Fizzy exposure

None at runtime. Verified:

  • saas/hotcell/operations/active_storage.rb requires only the vips, ffprobe, mutool and ffmpeg
    operation files; magick_operation.rb (which sets MiniMagick.restricted_env/cli_env at require
    time) is never loaded.
  • saas/hotcell/Dockerfile installs libvips42 mupdf-tools ffmpeg; no ImageMagick binary.
  • No file under app/, lib/, config/, saas/, test/ references MiniMagick/mini_magick
    (override audit: nothing to audit).
  • Gemfile.saas lists mini_magick, require: false only because the client gem loads the
    ImageMagick transformer; the app lockfile remains 5.3.3 and is unaffected by this change.

If magick operations are ever enabled in the cell

Not required now; recorded so the next person does not rediscover it:

  1. inherit_fds: is already satisfied by this bump (5.4.0) and image_processing 2.1.0 in the
    cell lockfile.
  2. The Dockerfile would need ImageMagick 7 (magick binary); MagickOperation resolves the binary
    at boot and fails the cell if absent.
  3. MagickOperation sets MiniMagick.restricted_env = true and a minimal cli_env process-wide at
    require time — check it does not clobber anything vips-side.
  4. Keep Gemfile.saas.lock's mini_magick at or above the cell's version is not needed — the two
    lock independently; the client/server protocol check is on the hotcell gems, not mini_magick.

Commits Requiring Mitigation

None.

Analyzed — No App Impact

None above "no impact".

No Impact (Skipped)

Commit Summary
6030c350 Add opt-in inherit_fds: to MiniMagick::Shell#execute (default []); spawn unchanged when omitted
ca66da07 Version constant 5.3.3 → 5.4.0

Assessments: ~/Work/basecamp/37signals-hq/upgrade-recon/mini_magick/{6030c350,ca66da07}.md

@flavorjones
flavorjones marked this pull request as draft September 8, 2026 15:39
The 0.4.0 dev cell needed a `TMPDIR` of its own, made by hand, or its
boot sweep emptied the developer's `/tmp`. hotcell 0.4.1 adds
`hotcell --development`, which keeps the scratch in a directory of the
cell's own under the system temporary directory and never sweeps the
directory it was given. Use it in `saas/Procfile.dev` instead.

ref: basecamp/hotcell#61
@flavorjones flavorjones changed the title Upgrade hotcell to 0.4.0 Upgrade hotcell to 0.4.1 Sep 8, 2026
@flavorjones

Copy link
Copy Markdown
Member Author

🤖 Upgrade Plan: hotcell v0.4.0..v0.4.1

Upgrade Plan: hotcell v0.4.0..v0.4.1 for fizzy

  • Date: 2026-09-08
  • Gem: hotcell (hotcell-client, activestorage-hotcell-client, hotcell-server, activestorage-hotcell-server, hotcell-core)
  • Range: v0.4.0 (919491de)..v0.4.1 (46bd334d), 3 commits
  • Target: fizzy, worktree fizzy--upgrade-hotcell-0-4-0 (branch upgrade-hotcell-0-4-0, basecamp/fizzy#3108), already on 0.4.0

Summary

  • Total commits: 3
  • No impact: 2 (e123993d version bump to 0.5.0.dev, 46bd334d release v0.4.1)
  • Analyzed, not affected: 0
  • Requires mitigation: 1 (4ddff8d6, hotcell#61) — a Procfile change, not a code change

The pins (~> 0.4.0 in Gemfile.saas and saas/hotcell/Gemfile) already admit 0.4.1. Only
Gemfile.saas.lock and saas/hotcell/Gemfile.lock move. No library API used by Fizzy changed;
the only behavioral change is opt-in via the new hotcell --development flag.

Decision: replace the TMPDIR/mkdir workaround with --development

Yes. Change saas/Procfile.dev's cell: line from

cell: mkdir -p $PWD/tmp/hotcell/tmp && env -u ... HOTCELL_DIR=$PWD/tmp/hotcell/active_storage TMPDIR=$PWD/tmp/hotcell/tmp HOTCELL_CONFIG=... HOTCELL_OPERATIONS=... bundle exec hotcell

to

cell: env -u ... HOTCELL_DIR=$PWD/tmp/hotcell/active_storage HOTCELL_CONFIG=... HOTCELL_OPERATIONS=... bundle exec hotcell --development

and drop the comment above it ("TMPDIR is set because the supervisor sweeps the scratch at boot").
#61 exists to fix exactly the hazard that comment describes, and it is the shape the README's
"Run it in development" now documents (... HOTCELL_DIR=... bundle exec hotcell --development, no
TMPDIR). examples/devcell made the same switch.

What --development does (from the 4ddff8d6 diff)

exe/hotcell:

  • development = ARGV.delete("--development") ? true : false, passed to Supervisor.new(..., development:).
  • abort "usage: hotcell [--development]" unless ARGV.empty? — any other argument now aborts (0.4.0 ignored ARGV). Fizzy passes none.
  • Env is unchanged: HOTCELL_DIR (default /run/hotcell/cell) and HOTCELL_WORKSPACE are still read from ENV; HOTCELL_CONFIG / HOTCELL_OPERATIONS are consumed by HotCell.load! exactly as before. The flag is orthogonal to all of them, and bundle exec hotcell --development passes it through untouched.

Supervisor (supervisor.rb):

  • tmpdir becomes @development_tmpdir || configured_tmpdir || Etc.systmpdir. With the flag, @development_tmpdir = own_tmpdir =
    File.join(TMPDIR-or-/tmp, "hotcell-#{HOTCELL_DIR.delete_prefix("/").tr("/", "-")}"). For Fizzy's Procfile that is
    /tmp/hotcell-home-<user>-Work-basecamp-<checkout>-tmp-hotcell-active_storage — unique per checkout, so two worktrees' cells do not sweep each other.
  • The default workspace is still File.join(tmpdir, "hotcell-workspace"), now beneath the cell's own scratch.
  • scratches is unchanged: [tmpdir, File.dirname(workspace)].uniq. With the flag and no HOTCELL_WORKSPACE, that is just the cell's own scratch, so the sweep still runs, but only inside /tmp/hotcell-<...>. /tmp (or a set TMPDIR) itself is never swept. An explicit HOTCELL_WORKSPACE still has its parent swept, flag or not — Fizzy sets none.
  • New claim_tmpdir, run in boot before verify_scratches!: Dir.mkdir claimed, 0o700; on EEXIST it lstats and requires a plain directory owned by this uid, then chmod 0o700. This replaces the mkdir -p in the Procfile. Note Dir.mkdir is not recursive: the parent (/tmp) must already exist, which it does.
  • cell.boot log line gains tmpdir:. Fizzy does not parse it.

What it refuses

ConfigurationError (boot fails) when:

  • --development scratch name is taken by anything but a directory this uid owns (a file, a symlink, or another user's directory): "is not a directory this uid owns, and a boot sweeps it. Set TMPDIR to a directory of the cell's own."
  • An ancestor of the scratch is a symlink this uid owns (owned_symlink_on, checked in claim_tmpdir and again in verify_scratches!). /tmp on Linux and macOS's /tmp -> /private/tmp (root-owned) both pass.
  • HOTCELL_DIR is inside a scratch. $PWD/tmp/hotcell/active_storage is not inside /tmp/hotcell-<...>, so this passes.
  • mkdir/chmod fail for any other SystemCallError.

Compatibility of HOTCELL_DIR=$PWD/tmp/hotcell/active_storage

Compatible. HOTCELL_DIR is only used to (a) name the scratch and (b) hold the sockets, which
bin/dev already points the app at via HOTCELL_ROOT=tmp/hotcell. Nothing about the flag moves the
sockets. The scratch name is ~95 characters for a typical checkout path, well under NAME_MAX.

Why drop TMPDIR rather than keep it alongside the flag

Keeping TMPDIR=$PWD/tmp/hotcell/tmp plus the flag also works (scratch becomes
$PWD/tmp/hotcell/tmp/hotcell-<...>), but:

  • mkdir -p would still be needed, because claim_tmpdir does not create the parent.
  • The path passes through $PWD, so any uid-owned symlink in the checkout path (a symlinked home or worktree) trips owned_symlink_on and the boot refuses. Under /tmp there is none.
  • It keeps a workaround whose reason no longer holds, and diverges from the upstream-documented Procfile.

If a stale non-directory ever occupies the scratch name in /tmp, the error message says what to do
(remove it, or set TMPDIR); that is the one situation the old line handled that the new one does not.

What must NOT change

  • saas/hotcell/Dockerfile CMD ["bundle", "exec", "hotcell"] — production accessory, where /tmp
    is the cell's own mount (/var/lib/hotcell-scratch:/tmp in saas/config/deploy.yml) and must be swept. No --development.

Commits Requiring Mitigation

4ddff8d6: Add hotcell --development for a scratch of the cell's own (#61)

Impact: unlikely impact (opt-in flag; the only non-opt-in change is exe/hotcell aborting on unknown arguments)

Matched in:

  • saas/Procfile.dev:5 — ... TMPDIR=$PWD/tmp/hotcell/tmp ... bundle exec hotcell (uncontainerized dev boot; the workaround #61 supersedes)
  • saas/hotcell/Dockerfile:87 — CMD ["bundle", "exec", "hotcell"] (production; correct as is)

No Supervisor.new, TestCell.new/.boot, HOTCELL_WORKSPACE, or cell.boot log consumers in app/, lib/, test/, config/, saas/, or bundled gems.

Mitigation: the Procfile change above. Steps:

  1. bundle lock --update hotcell-client activestorage-hotcell-client hotcell-core with BUNDLE_GEMFILE=Gemfile.saas; then the same for hotcell-server activestorage-hotcell-server hotcell-core in saas/hotcell/. Verify both lockfiles show 0.4.1 and matching hotcell-core versions.
  2. Edit saas/Procfile.dev as above.
  3. Verify: bin/dev in SaaS mode; the cell's cell.boot log line should show tmpdir: "/tmp/hotcell-...-tmp-hotcell-active_storage", that directory should exist with mode 0700, $PWD/tmp/hotcell/active_storage/work.sock should appear, and /tmp must keep its other contents. Upload an image and confirm a variant renders through the cell.
  4. Run the saas test suite as the 0.4.0 PR did; nothing in the client changed, so no new failures are expected.

Override audit

Fizzy's override surface (lib/rails_ext/*.rb, config/initializers/*.rb, saas/hotcell/config.rb,
saas/hotcell/operations/) touches nothing the range changed. The range changed only
hotcell-server (exe/hotcell, supervisor.rb, test_cell.rb) plus docs and version constants;
Fizzy calls none of those directly. The single Fizzy-side workaround for hotcell behavior is the
TMPDIR/mkdir -p Procfile line, now redundant — see the decision above. No CVE/GHSA identifiers in
the range's assessments.

No Impact (Skipped)

Commit Summary
e123993d version bump to 0.5.0.dev
46bd334d Release v0.4.1 (version constants and CHANGELOG heading only)

@flavorjones
flavorjones marked this pull request as ready for review September 8, 2026 18:34
Comment thread saas/hotcell/Gemfile Outdated
Comment thread Gemfile.saas Outdated
A `~>` pin let a patch release move the app's client and the cell's
server independently, and one version apart is a `protocol` failure on
every request. Pin both Gemfiles to `0.4.1`.
@flavorjones
flavorjones merged commit 5f6d90c into main Sep 8, 2026
13 checks passed
@flavorjones
flavorjones deleted the upgrade-hotcell-0-4-0 branch September 8, 2026 19:41
rhysb27 pushed a commit to rhysb27/fizzy that referenced this pull request Sep 16, 2026
* Upgrade hotcell to 0.4.0

A killed worker left its tool's scratch files at the top of the cell's
`/tmp`, outside the slot tree that is removed when a request ends, so
the scratch volume filled over time. A worker now points `TMPDIR` at
the request's home, and the supervisor empties the scratch at boot.

That boot sweep is why the dev cell in `saas/Procfile.dev` now gets a
`TMPDIR` of its own: unset, it would sweep the developer's `/tmp`.

The release's ImageMagick changes do not reach this cell, which loads
only the vips, ffprobe, mutool and ffmpeg operations and installs no
ImageMagick, so the `MAGICK_*_LIMIT` variables stay unset.

ref: https://github.com/basecamp/hotcell/blob/v0.4.0/CHANGELOG.md

* Log what a failed cell tool wrote to stderr

A crash's diagnosis — `libgomp: Thread creation failed` — survived only
in the exception's message, so a failure that was discarded rather than
retried lost it. hotcell 0.4.0 carries the captured stream on the
`perform.hot_cell` event; write it as a field on the existing log line.

ref: basecamp/hotcell#60

* Upgrade hotcell to 0.4.1 and boot the dev cell with --development

The 0.4.0 dev cell needed a `TMPDIR` of its own, made by hand, or its
boot sweep emptied the developer's `/tmp`. hotcell 0.4.1 adds
`hotcell --development`, which keeps the scratch in a directory of the
cell's own under the system temporary directory and never sweeps the
directory it was given. Use it in `saas/Procfile.dev` instead.

ref: basecamp/hotcell#61

* Pin the hotcell gems to exact versions

A `~>` pin let a patch release move the app's client and the cell's
server independently, and one version apart is a `protocol` failure on
every request. Pin both Gemfiles to `0.4.1`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants