From 3da543463136c8b12c01a902a448c0e4bd5e3632 Mon Sep 17 00:00:00 2001 From: Mike Dalessio Date: Sat, 10 Oct 2026 14:28:27 -0400 Subject: [PATCH 1/3] Add a `start_time_seconds` gauge and deprecate `uptime_seconds` `yabeda-hotcell` exported each cell's uptime, a value that changes at every scrape. The Prometheus guidance is to export a start time instead, and to compute the uptime as `time() - start_time`. Report the supervisor's boot time from `metrics` as `start_time_s`, export it as `start_time_seconds`, and deprecate `uptime_seconds`. A cell from an earlier release reports no start time, so the gem sets no `start_time_seconds` for it. ref: https://github.com/basecamp/hotcell/issues/112 --- CHANGELOG.md | 15 +++++++++++++++ docs/observability.md | 3 ++- hotcell-server/lib/hot_cell/counters.rb | 5 +++-- hotcell-server/test/control_test.rb | 12 ++++++++++++ yabeda-hotcell/lib/yabeda/hot_cell.rb | 6 +++++- yabeda-hotcell/test/collect_test.rb | 23 +++++++++++++++++++++++ 6 files changed, 60 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33a2710..1b57d8e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ Some actions that application developers should consider taking when upgrading f * In your own `perform.hot_cell` subscribers, test for success with `payload[:code] == "ok"` instead of `payload[:code].nil?`, and expect `interrupted`. The event now sets `code` on every call; see [Per-call notification](docs/observability.md#per-call-notification). * Where a test instruments `perform.hot_cell` by hand to stand in for a successful call, pass `code: "ok"`. `HotCell::LogSubscriber` and `yabeda-hotcell` now record `code` as the event gives it, so an event without one records no code. * Count `code="interrupted"` as a failed call wherever a dashboard or alert computes an error rate from `hotcell_requests`. `yabeda-hotcell` counted these calls as `ok` before. +* Replace `hotcell_uptime_seconds` with `time() - hotcell_start_time_seconds` in dashboards and alerts. `hotcell_uptime_seconds` is deprecated, and a future release removes it. Deploy cells running this release too: until a cell runs it, `hotcell_start_time_seconds` has no value for that cell. ### HotCell::Client @@ -39,6 +40,12 @@ Some actions that application developers should consider taking when upgrading f * On macOS, a call to a full cell reports `capacity` instead of occasionally reporting `unavailable`. A full cell answers and closes the connection without reading the request, so the client's send fails. The client already ignored that failure as `EPIPE` or `ECONNRESET` and read the answer, but macOS sometimes raises `ENOTCONN` instead. (#110) +### HotCell::Server + +#### Added + +* `metrics` reports `start_time_s`, the time the supervisor booted, in seconds since the Unix epoch. (#112) + ### HotCell::Client::Legacy #### Breaking @@ -47,6 +54,14 @@ Some actions that application developers should consider taking when upgrading f ### Yabeda::HotCell +#### Added + +* The `start_time_seconds` gauge records when each cell booted, in seconds since the Unix epoch. A cell from an earlier release reports no start time, so the gauge has no value for it. (#112) + +#### Deprecated + +* The `uptime_seconds` gauge. Use `time() - hotcell_start_time_seconds` instead. A future release removes it. (#112) + #### Fixed * A call that an exception interrupted, such as the application's own request timeout, is counted in `requests` under `code="interrupted"` and records no `perform` observation. Previously, it was counted as `code="ok"` with a `perform` of 0 seconds. (#93) diff --git a/docs/observability.md b/docs/observability.md index 8f5f4a5..cefd3f0 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -141,7 +141,8 @@ The metrics are in the `hotcell` group: | `queue_high_water` | gauge | `cell` | The deepest that the queue has been since boot. | | `cancelled` | gauge | `cell` | Callers that gave up before the cell answered. This is a floor. | | `killed` | gauge | `cell`, `cause` | Workers killed since boot, by cause. | -| `uptime_seconds` | gauge | `cell` | Seconds since the supervisor booted. | +| `start_time_seconds` | gauge | `cell` | When the supervisor booted, in seconds since the Unix epoch. `time() - hotcell_start_time_seconds` is the uptime. | +| `uptime_seconds` | gauge | `cell` | Seconds since the supervisor booted. Deprecated: use `start_time_seconds`. A future release removes it. | On each scrape, the gem reads `metrics` from each registered cell and sets the gauges from it. A scrape never fails because a cell misbehaves: the gem reports the error to the Active Support error reporter. diff --git a/hotcell-server/lib/hot_cell/counters.rb b/hotcell-server/lib/hot_cell/counters.rb index 9bad2a0..2d99ec6 100644 --- a/hotcell-server/lib/hot_cell/counters.rb +++ b/hotcell-server/lib/hot_cell/counters.rb @@ -18,6 +18,7 @@ module HotCell class Counters def initialize @started_at = Clock.now + @start_time = Time.now.to_f @requests = Hash.new(0) @killed_by = Hash.new(0) @cancelled = 0 @@ -42,8 +43,8 @@ def observe_queue(depth) end def to_h(running: 0, queued: 0) - { uptime_s: (Clock.now - @started_at).round, running: running, queued: queued, - queue_high_water: @queue_high_water, requests: @requests, killed_by: @killed_by, + { start_time_s: @start_time, uptime_s: (Clock.now - @started_at).round, running: running, + queued: queued, queue_high_water: @queue_high_water, requests: @requests, killed_by: @killed_by, cancelled: @cancelled } end end diff --git a/hotcell-server/test/control_test.rb b/hotcell-server/test/control_test.rb index 55a482e..82d7ed2 100644 --- a/hotcell-server/test/control_test.rb +++ b/hotcell-server/test/control_test.rb @@ -83,6 +83,18 @@ def test_metrics_count_a_caller_that_gave_up_before_the_cell_answered end end + def test_metrics_report_when_the_cell_started + before_boot = Time.now.to_f + + TestCell.boot do |cell| + started = assert_ok(cell.control("hotcell.metrics")).result[:start_time_s] + + assert_operator started, :>=, before_boot + assert_operator started, :<=, Time.now.to_f + assert_equal started, assert_ok(cell.control("hotcell.metrics")).result[:start_time_s] + end + end + def test_metrics_report_what_no_single_caller_can_see TestCell.boot(concurrency: 1, queue_size: 4, deadline: 30) do |cell| blocker = cell.connect diff --git a/yabeda-hotcell/lib/yabeda/hot_cell.rb b/yabeda-hotcell/lib/yabeda/hot_cell.rb index 33c3029..e28a132 100644 --- a/yabeda-hotcell/lib/yabeda/hot_cell.rb +++ b/yabeda-hotcell/lib/yabeda/hot_cell.rb @@ -33,7 +33,9 @@ def self.install! tags: %i[ cell ], aggregation: :most_recent gauge :killed, comment: "Workers killed since boot, by cause", tags: %i[ cell cause ], aggregation: :most_recent - gauge :uptime_seconds, comment: "Seconds since the supervisor booted", + gauge :uptime_seconds, comment: "Seconds since the supervisor booted. Deprecated: use start_time_seconds", + tags: %i[ cell ], aggregation: :most_recent + gauge :start_time_seconds, comment: "When the supervisor booted, in seconds since the Unix epoch", tags: %i[ cell ], aggregation: :most_recent collect { Yabeda::HotCell.collect_stats } @@ -91,6 +93,8 @@ def self.record_perform(event) Yabeda.hotcell.queue_high_water.set(tags, counters[:queue_high_water]) Yabeda.hotcell.cancelled.set(tags, counters[:cancelled]) Yabeda.hotcell.uptime_seconds.set(tags, counters[:uptime_s]) + # A cell from an earlier release reports no start time. + Yabeda.hotcell.start_time_seconds.set(tags, counters[:start_time_s]) if counters[:start_time_s] ::HotCell::Codes::PERMANENT_BY_CAUSE.each_key do |cause| Yabeda.hotcell.killed.set(tags.merge(cause: cause), counters[:killed_by].fetch(cause.to_sym, 0)) end diff --git a/yabeda-hotcell/test/collect_test.rb b/yabeda-hotcell/test/collect_test.rb index f440d8c..d795154 100644 --- a/yabeda-hotcell/test/collect_test.rb +++ b/yabeda-hotcell/test/collect_test.rb @@ -19,6 +19,29 @@ def test_publishes_a_running_cells_counters assert_equal 0, gauge(:queue_high_water) assert_equal 0, gauge(:cancelled) assert_kind_of Integer, gauge(:uptime_seconds) + assert_kind_of Float, gauge(:start_time_seconds) + end + + def test_publishes_when_the_cell_started + HotCell.root = "/nowhere" + register transport: CannedTransport.new(metrics(start_time_s: 1_760_000_000.25)) + + Yabeda.collect! + + assert_equal 1_760_000_000.25, gauge(:start_time_seconds) + end + + # A cell from an earlier release reports no start time, and a mixed deploy has one. + def test_a_cell_that_does_not_report_its_start_time_publishes_none + HotCell.root = "/nowhere" + register transport: CannedTransport.new(metrics) + + Yabeda.collect! + + refute_includes Yabeda::TestAdapter.instance.gauges[Yabeda.hotcell.start_time_seconds], { cell: "test" } + assert_equal 41, gauge(:uptime_seconds) + assert_equal 0, gauge(:killed, cause: "memory") + assert_empty @reported end def test_publishes_kills_by_cause From baeec4e08d9d70ec08f8a3a7a541e2f03cf303dd Mon Sep 17 00:00:00 2001 From: Mike Dalessio Date: Sat, 10 Oct 2026 14:38:16 -0400 Subject: [PATCH 2/3] Reword the `uptime_seconds` deprecation notes --- CHANGELOG.md | 6 +++--- docs/observability.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b57d8e..23bf6e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,7 +23,7 @@ Some actions that application developers should consider taking when upgrading f * In your own `perform.hot_cell` subscribers, test for success with `payload[:code] == "ok"` instead of `payload[:code].nil?`, and expect `interrupted`. The event now sets `code` on every call; see [Per-call notification](docs/observability.md#per-call-notification). * Where a test instruments `perform.hot_cell` by hand to stand in for a successful call, pass `code: "ok"`. `HotCell::LogSubscriber` and `yabeda-hotcell` now record `code` as the event gives it, so an event without one records no code. * Count `code="interrupted"` as a failed call wherever a dashboard or alert computes an error rate from `hotcell_requests`. `yabeda-hotcell` counted these calls as `ok` before. -* Replace `hotcell_uptime_seconds` with `time() - hotcell_start_time_seconds` in dashboards and alerts. `hotcell_uptime_seconds` is deprecated, and a future release removes it. Deploy cells running this release too: until a cell runs it, `hotcell_start_time_seconds` has no value for that cell. +* `hotcell_uptime_seconds` is now deprecated. In dashboards and alerts, replace it with `time() - hotcell_start_time_seconds`. `hotcell_start_time_seconds` has a value only for cells running this release. ### HotCell::Client @@ -56,11 +56,11 @@ Some actions that application developers should consider taking when upgrading f #### Added -* The `start_time_seconds` gauge records when each cell booted, in seconds since the Unix epoch. A cell from an earlier release reports no start time, so the gauge has no value for it. (#112) +* The `start_time_seconds` gauge records when each cell booted, in seconds since the Unix epoch. (#112) #### Deprecated -* The `uptime_seconds` gauge. Use `time() - hotcell_start_time_seconds` instead. A future release removes it. (#112) +* `uptime_seconds` is now deprecated. Use `time() - hotcell_start_time_seconds` instead. It will be removed in a future version. (#112) #### Fixed diff --git a/docs/observability.md b/docs/observability.md index cefd3f0..46ede25 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -142,7 +142,7 @@ The metrics are in the `hotcell` group: | `cancelled` | gauge | `cell` | Callers that gave up before the cell answered. This is a floor. | | `killed` | gauge | `cell`, `cause` | Workers killed since boot, by cause. | | `start_time_seconds` | gauge | `cell` | When the supervisor booted, in seconds since the Unix epoch. `time() - hotcell_start_time_seconds` is the uptime. | -| `uptime_seconds` | gauge | `cell` | Seconds since the supervisor booted. Deprecated: use `start_time_seconds`. A future release removes it. | +| `uptime_seconds` | gauge | `cell` | ⚠️ Deprecated: use `start_time_seconds`. Seconds since the supervisor booted. This field will be removed in a future version. | On each scrape, the gem reads `metrics` from each registered cell and sets the gauges from it. A scrape never fails because a cell misbehaves: the gem reports the error to the Active Support error reporter. From 69e807d9cd3c586e6bda160e0e6c9b90ba6e4d60 Mon Sep 17 00:00:00 2001 From: Mike Dalessio Date: Sat, 10 Oct 2026 14:41:50 -0400 Subject: [PATCH 3/3] Remove needless comments --- yabeda-hotcell/lib/yabeda/hot_cell.rb | 1 - yabeda-hotcell/test/collect_test.rb | 1 - 2 files changed, 2 deletions(-) diff --git a/yabeda-hotcell/lib/yabeda/hot_cell.rb b/yabeda-hotcell/lib/yabeda/hot_cell.rb index e28a132..b119988 100644 --- a/yabeda-hotcell/lib/yabeda/hot_cell.rb +++ b/yabeda-hotcell/lib/yabeda/hot_cell.rb @@ -93,7 +93,6 @@ def self.record_perform(event) Yabeda.hotcell.queue_high_water.set(tags, counters[:queue_high_water]) Yabeda.hotcell.cancelled.set(tags, counters[:cancelled]) Yabeda.hotcell.uptime_seconds.set(tags, counters[:uptime_s]) - # A cell from an earlier release reports no start time. Yabeda.hotcell.start_time_seconds.set(tags, counters[:start_time_s]) if counters[:start_time_s] ::HotCell::Codes::PERMANENT_BY_CAUSE.each_key do |cause| Yabeda.hotcell.killed.set(tags.merge(cause: cause), counters[:killed_by].fetch(cause.to_sym, 0)) diff --git a/yabeda-hotcell/test/collect_test.rb b/yabeda-hotcell/test/collect_test.rb index d795154..c9c0c28 100644 --- a/yabeda-hotcell/test/collect_test.rb +++ b/yabeda-hotcell/test/collect_test.rb @@ -31,7 +31,6 @@ def test_publishes_when_the_cell_started assert_equal 1_760_000_000.25, gauge(:start_time_seconds) end - # A cell from an earlier release reports no start time, and a mixed deploy has one. def test_a_cell_that_does_not_report_its_start_time_publishes_none HotCell.root = "/nowhere" register transport: CannedTransport.new(metrics)