Carry a failure's stderr on the perform.hot_cell event - #60
Conversation
The `HotCell (…)` log line an application writes from the event read `"cause":"crashed"` with no diagnosis. The tool's stderr survived only in the exception's message, and only a retry logs that, so a discarded failure lost it. Publish `stderr` beside `signal`. The field is bounded by `Failure.sanitize` on the way in; the comment on `publish` says what a subscriber may do with it. ref: https://app.basecamp.com/2914079/buckets/1666/card_tables/cards/10270096231
There was a problem hiding this comment.
🟢 Approval recommended
The change is small, consistent with existing Failure sanitization/bounding behavior, and is covered by targeted tests and changelog documentation.
Pull request overview
This PR improves observability of HotCell failures by including the cell/worker’s captured stderr on the perform.hot_cell event payload, so subscribers can log the crash diagnosis even when a failure is discarded (i.e., not retried) and its exception message isn’t otherwise emitted.
Changes:
- Add
event[:stderr] = failure&.stderrtoHotCell::Client#publishalongside existingcode/cause/signal/permanentfields. - Extend classification tests to assert
stderris present for acrashedfailure andnilon success. - Document the new event field in
CHANGELOG.mdunderHotCell::Client / Added.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
File summaries
| File | Description |
|---|---|
| hotcell-client/lib/hot_cell/client.rb | Publishes stderr on perform.hot_cell events so subscribers can log crash diagnostics. |
| hotcell-client/test/classification_test.rb | Adds assertions covering stderr propagation for crash failures and nil on success. |
| CHANGELOG.md | Documents the addition of stderr on the perform.hot_cell event. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
* Upgrade hotcell to 0.4.0 A killed worker left its tool's scratch files at the top of the cell's `/tmp`, outside the slot tree that is removed when a request ends, so the scratch volume filled over time. A worker now points `TMPDIR` at the request's home, and the supervisor empties the scratch at boot. That boot sweep is why the dev cell in `saas/Procfile.dev` now gets a `TMPDIR` of its own: unset, it would sweep the developer's `/tmp`. The release's ImageMagick changes do not reach this cell, which loads only the vips, ffprobe, mutool and ffmpeg operations and installs no ImageMagick, so the `MAGICK_*_LIMIT` variables stay unset. ref: https://github.com/basecamp/hotcell/blob/v0.4.0/CHANGELOG.md * Log what a failed cell tool wrote to stderr A crash's diagnosis — `libgomp: Thread creation failed` — survived only in the exception's message, so a failure that was discarded rather than retried lost it. hotcell 0.4.0 carries the captured stream on the `perform.hot_cell` event; write it as a field on the existing log line. ref: basecamp/hotcell#60 * Upgrade hotcell to 0.4.1 and boot the dev cell with --development The 0.4.0 dev cell needed a `TMPDIR` of its own, made by hand, or its boot sweep emptied the developer's `/tmp`. hotcell 0.4.1 adds `hotcell --development`, which keeps the scratch in a directory of the cell's own under the system temporary directory and never sweeps the directory it was given. Use it in `saas/Procfile.dev` instead. ref: basecamp/hotcell#61 * Pin the hotcell gems to exact versions A `~>` pin let a patch release move the app's client and the cell's server independently, and one version apart is a `protocol` failure on every request. Pin both Gemfiles to `0.4.1`.
* Upgrade hotcell to 0.4.0 A killed worker left its tool's scratch files at the top of the cell's `/tmp`, outside the slot tree that is removed when a request ends, so the scratch volume filled over time. A worker now points `TMPDIR` at the request's home, and the supervisor empties the scratch at boot. That boot sweep is why the dev cell in `saas/Procfile.dev` now gets a `TMPDIR` of its own: unset, it would sweep the developer's `/tmp`. The release's ImageMagick changes do not reach this cell, which loads only the vips, ffprobe, mutool and ffmpeg operations and installs no ImageMagick, so the `MAGICK_*_LIMIT` variables stay unset. ref: https://github.com/basecamp/hotcell/blob/v0.4.0/CHANGELOG.md * Log what a failed cell tool wrote to stderr A crash's diagnosis — `libgomp: Thread creation failed` — survived only in the exception's message, so a failure that was discarded rather than retried lost it. hotcell 0.4.0 carries the captured stream on the `perform.hot_cell` event; write it as a field on the existing log line. ref: basecamp/hotcell#60 * Upgrade hotcell to 0.4.1 and boot the dev cell with --development The 0.4.0 dev cell needed a `TMPDIR` of its own, made by hand, or its boot sweep emptied the developer's `/tmp`. hotcell 0.4.1 adds `hotcell --development`, which keeps the scratch in a directory of the cell's own under the system temporary directory and never sweeps the directory it was given. Use it in `saas/Procfile.dev` instead. ref: basecamp/hotcell#61 * Pin the hotcell gems to exact versions A `~>` pin let a patch release move the app's client and the cell's server independently, and one version apart is a `protocol` failure on every request. Pin both Gemfiles to `0.4.1`.
The
HotCell (…)line each application logs from theperform.hot_cellevent reads"cause":"crashed"with no diagnosis. The cell sends the worker's stderr on the wire andFailure#to_srenders it into the exception's message, but only theRetrying …line logs that message, and only when a retry follows. A failure that was discarded lost the one line that explained it,libgomp: Thread creation failed: Resource temporarily unavailable, and reading it meant querying the cell's own Loki stream. Post-mortem: card 10261975276; this is the "Log the cell's stderr and signal Rails-side" action item, card 10270096231.Client#publishnow setsevent[:stderr] = failure&.stderrbesidesignal. The field is already bounded byFailure.sanitizeon the way in, tail-kept, so a hostile cell cannot grow it pastMAX_MESSAGE_BYTES. It is text a tool wrote while processing a hostile file, and the comment onpublishsays a subscriber should write it to a log field and interpolate it nowhere else.The test is a
crashedfailure with stderr publishing an event that carries it, beside the existing cause and signal cases; the success case asserts the field is nil. Changelog underHotCell::Client / Added.VERSIONis already0.3.2.dev, so no bump.The application half is separate: once this ships,
Yabeda::HotCell.log_performin bc3, haystack and fizzy logssignalandstderron theHotCell (…)line.