Add a yabeda-hotcell gem for application metrics

CHANGELOG.md

@@ -1,12 +1,13 @@
 # HotCell changelog
 
-This changelog covers five gems, which release together on the same version:
+This changelog covers these gems, which release together on the same version:
 
 - `hotcell-core`
 - `hotcell-client`
 - `hotcell-server`
 - `activestorage-hotcell-client`
 - `activestorage-hotcell-server`
+- `yabeda-hotcell`
 
 A `Tooling` section records changes to the checks and scripts in `bin/` and `examples/`, which ship in no
 gem but are what an operator runs against their own image.
@@ -17,6 +18,7 @@ gem but are what an operator runs against their own image.
 
 Some actions that application developers should consider taking when upgrading from an earlier version:
 
+* If the application has its own Yabeda integration for HotCell, replace it with the `yabeda-hotcell` gem. The gem uses the same metric names. It does not write a log line for each call.
 * Replace a cell's copies of `examples/operations/echo.rb` and `reopen.rb` with `require "hot_cell/health_operations"`, and point the application's clients at `health.echo` and `health.reopen`. See the README's "Rails healthcheck".
 
 ### HotCell::Server
@@ -36,6 +38,12 @@ Some actions that application developers should consider taking when upgrading f
 
 * The client now returns `capacity` when a full cell closes the connection before the client finishes sending the request. Previously, the client returned `unavailable` for the broken pipe.
 
+### Yabeda::HotCell
+
+#### Added
+
+* New gem `yabeda-hotcell`. It publishes Yabeda metrics for the application side. Call `Yabeda::HotCell.install!` one time at boot. The gem counts each call by cell, operation, code and cause, and measures the time the cell spent. On each scrape, it reads the counters of each registered cell. See the README's "Metrics collection".
+
 ### Tooling
 
 #### Changed

CONTRIBUTING.md

@@ -77,6 +77,7 @@ hotcell-client/               the application side, and the installer that scaff
 hotcell-server/               the supervisor, workers, slots, limits, and exe/hotcell
 activestorage-hotcell-server/ the media operations
 activestorage-hotcell-client/ the Rails transformer, analyzers and previewers
+yabeda-hotcell/               Yabeda metrics for the application side
 
 examples/                     one cell's worth of sample operations and the battery that drives them
 bin/                          the container checks: example-image, conformance, load
@@ -288,16 +289,16 @@ breaks either.
 
 ## Making a release
 
-The five gems release together on one version, and `VERSION` at the repository root is what sets it.
+The gems release together on one version, and `VERSION` at the repository root is what sets it.
 
 - Prechecks
   - [ ] make sure CI is green
   - [ ] `bundle exec rake` — the full suite and rubocop
   - [ ] update `CHANGELOG.md`: retitle `next / unreleased` with the version and the date
-  - [ ] `bundle exec rake version:bump[1.2.3]` — writes `VERSION` and the five gems' version constants
+  - [ ] `bundle exec rake version:bump[1.2.3]` — writes `VERSION` and every gem's version constant
   - [ ] commit, and tag as `v1.2.3`
 - Release
-  - [ ] `bundle exec rake gems` — builds the five gems into `pkg/`, which it empties first
+  - [ ] `bundle exec rake gems` — builds every gem into `pkg/`, which it empties first
   - [ ] `git push && git push --tags` — **before** the gems: every gemspec's `changelog_uri` and
         `source_code_uri` name the tag, and they 404 until it is on GitHub
   - [ ] `for f in pkg/*.gem ; do gem push $f ; done`

Gemfile

@@ -9,6 +9,8 @@ gemspec path: "hotcell-server", name: "hotcell-server"
 gemspec path: "activestorage-hotcell-client", name: "activestorage-hotcell-client"
 gemspec path: "activestorage-hotcell-server", name: "activestorage-hotcell-server"
 
+gemspec path: "yabeda-hotcell", name: "yabeda-hotcell"
+
 # activestorage-hotcell-client needs config.active_storage.variant_processor to accept a class, which is
 # rails/rails#58384. Merged, unreleased — so this tracks main until 8.2 ships and the gemspec floor can name
 # a version instead.

Gemfile.lock

@@ -94,9 +94,18 @@ PATH
     hotcell-server (0.6.0.dev)
       hotcell-core (= 0.6.0.dev)
 
+PATH
+  remote: yabeda-hotcell
+  specs:
+    yabeda-hotcell (0.6.0.dev)
+      hotcell-client (= 0.6.0.dev)
+      yabeda (>= 0.12)
+
 GEM
   remote: https://rubygems.org/
   specs:
+    anyway_config (2.8.1)
+      ruby-next-core (~> 1.0)
     ast (2.4.3)
     base64 (0.3.0)
     bigdecimal (4.1.3)
@@ -106,6 +115,7 @@ GEM
     crass (1.0.7)
     date (3.5.1)
     drb (2.2.3)
+    dry-initializer (3.2.0)
     erb (6.0.7)
     erubi (1.13.1)
     ffi (1.17.4)
@@ -244,6 +254,7 @@ GEM
     rubocop-rake (0.7.1)
       lint_roller (~> 1.1)
       rubocop (>= 1.72.1)
+    ruby-next-core (1.2.1)
     ruby-progressbar (1.13.0)
     ruby-vips (2.3.0)
       ffi (~> 1.12)
@@ -260,6 +271,10 @@ GEM
     unicode-emoji (4.3.0)
     uri (1.1.1)
     useragent (0.16.11)
+    yabeda (0.16.0)
+      anyway_config (>= 1.0, < 3)
+      concurrent-ruby
+      dry-initializer
     zeitwerk (2.8.3)
 
 PLATFORMS
@@ -291,6 +306,7 @@ DEPENDENCIES
   rubocop-packaging
   rubocop-performance
   rubocop-rake
+  yabeda-hotcell!
 
 CHECKSUMS
   actionpack (8.2.0.alpha)
@@ -302,6 +318,7 @@ CHECKSUMS
   activestorage-hotcell-client (0.6.0.dev)
   activestorage-hotcell-server (0.6.0.dev)
   activesupport (8.2.0.alpha)
+  anyway_config (2.8.1) sha256=541842b25117fac3e277670c32dc09d3b715b20404826a6538ac35527583624f
   ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383
   base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b
   bigdecimal (4.1.3) sha256=61ebe1e5e559bdc3cc6f2c0ee7f427321fc838f59611c294356eb04d6e21cf66
@@ -312,6 +329,7 @@ CHECKSUMS
   crass (1.0.7) sha256=94868719948664c89ddcaf0a37c65048413dfcb1c869470a5f7a7ceb5390b295
   date (3.5.1) sha256=750d06384d7b9c15d562c76291407d89e368dda4d4fff957eb94962d325a0dc0
   drb (2.2.3) sha256=0b00d6fdb50995fe4a45dea13663493c841112e4068656854646f418fda13373
+  dry-initializer (3.2.0) sha256=37d59798f912dc0a1efe14a4db4a9306989007b302dcd5f25d0a2a20c166c4e3
   erb (6.0.7) sha256=c5ca6dc25b0ef974a44dc8f59fe847577122483b1968a38dec305c60bf91ee92
   erubi (1.13.1) sha256=a082103b0885dbc5ecf1172fede897f9ebdb745a4b97a5e8dc63953db1ee4ad9
   ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf
@@ -389,6 +407,7 @@ CHECKSUMS
   rubocop-packaging (0.6.0) sha256=fb92bd0fb48e6f8cdb1648d2249b0cd51c2497dcc87340132d22f01edbf558a7
   rubocop-performance (1.27.0) sha256=eeeb1374d062a368ee1c787b70eb0b0cc4b184cb1f8565f424760946146d61ce
   rubocop-rake (0.7.1) sha256=3797f2b6810c3e9df7376c26d5f44f3475eda59eb1adc38e6f62ecf027cbae4d
+  ruby-next-core (1.2.1) sha256=ddba3e986e127143299a26bde8bdbeedd4bc738a1a765e499f9634f361551f79
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby-vips (2.3.0) sha256=e685ec02c13969912debbd98019e50492e12989282da5f37d05f5471442f5374
   securerandom (0.4.1) sha256=cc5193d414a4341b6e225f0cb4446aceca8e50d5e1888743fac16987638ea0b1
@@ -401,6 +420,8 @@ CHECKSUMS
   unicode-emoji (4.3.0) sha256=11c02fa73290378c066bb0562cd4c87d8e1b706fbbe0059ca12746a4244de8ce
   uri (1.1.1) sha256=379fa58d27ffb1387eaada68c749d1426738bd0f654d812fcc07e7568f5c57c6
   useragent (0.16.11) sha256=700e6413ad4bb954bb63547fa098dddf7b0ebe75b40cc6f93b8d54255b173844
+  yabeda (0.16.0) sha256=7f6e51acd7d9a51d850ea8c3844f72a24882f1312b3fc3836052bcd63d384cba
+  yabeda-hotcell (0.6.0.dev)
   zeitwerk (2.8.3) sha256=2c85125a8467ce069e20123d1e709a08955c9d29c118c25b46b7b7fafdbb92e5
 
 BUNDLED WITH

README.md

@@ -98,6 +98,7 @@ hogs. Whatever is putting your trusted core application at risk, move it out!
 | `hotcell-server` | the cell | The supervisor, the worker, `HotCell::Operation`, the container image. |
 | `activestorage-hotcell-client` | the application | The transformer, analyzer, and previewers Rails is configured with. |
 | `activestorage-hotcell-server` | the cell | The `transformers.image.*`, `analyzers.image.*`, `analyzers.media.ffprobe`, and `previewers.*` operations. |
+| `yabeda-hotcell` | the application | Yabeda metrics for every call and for each local cell's counters. |
 
 They are in one repository because they are being developed together today. We may split out the
 Active Storage gems into another repository at a later date.
@@ -546,10 +547,25 @@ these too. Alert on the presence of `worker.crashed`, and on `worker.killed` by
 
 ### Metrics collection
 
-Poll `cell.metrics` on a schedule, for example with a [Yabeda](https://github.com/yabeda-rb/yabeda)
-`collect` block. The control socket answers even while the work socket is saturated, and it is
-host-local, so the poller must be a process on the cell's own host. Watch `queued`,
-`queue_high_water`, `cancelled`, and `killed_by` cause.
+The `yabeda-hotcell` gem publishes [Yabeda](https://github.com/yabeda-rb/yabeda) metrics. Add it to the
+application's `Gemfile`, and install it once at boot:
+
+```ruby
+# Gemfile
+gem "yabeda-hotcell"
+
+# config/initializers/hotcell.rb
+Yabeda::HotCell.install!
+```
+
+The metrics are in the `hotcell` group. The `requests` counter counts every call, tagged with `cell`,
+`operation`, `code` and `cause`, and the `perform` histogram measures the time the cell spent. On each
+scrape the gem polls `cell.metrics` from every registered cell and sets the gauges `up`, `running`,
+`queued`, `queue_high_water`, `cancelled`, `killed` (by `cause`), and `uptime_seconds`.
+
+The control socket answers even while the work socket is saturated, and it is host-local, so the
+scraped process must be on the cell's own host. Watch `queued`, `queue_high_water`, `cancelled`, and
+`killed` by cause.
 
 ### Per-call telemetry
 

Rakefile

@@ -1,11 +1,11 @@
 # frozen_string_literal: true
 
-# The three hotcell gems need no tools and no container: their suites run on fixture operations in a few
-# seconds, and so does the development cell the battery drives. The two activestorage-hotcell gems convert
+# The hotcell gems and yabeda-hotcell need no tools and no container: their suites run on fixture operations in
+# a few seconds, and so does the development cell the battery drives. The two activestorage-hotcell gems convert
 # real files and need libvips, mutool, ffmpeg and ffprobe installed. That split is a design property rather
 # than an accident, so the tasks keep it visible: `test:hotcell` is what CI runs on a machine with nothing
 # installed, and on macOS.
-HOTCELL = %w[ hotcell-core hotcell-client hotcell-server ].freeze
+HOTCELL = %w[ hotcell-core hotcell-client hotcell-server yabeda-hotcell ].freeze
 ACTIVE_STORAGE = %w[ activestorage-hotcell-server activestorage-hotcell-client ].freeze
 GEMS = (HOTCELL + ACTIVE_STORAGE).freeze
 
@@ -18,6 +18,7 @@ VERSION_FILES = %w[
   hotcell-server/lib/hot_cell/server/version.rb
   activestorage-hotcell-client/lib/active_storage/hot_cell/client/version.rb
   activestorage-hotcell-server/lib/active_storage/hot_cell/server/version.rb
+  yabeda-hotcell/lib/yabeda/hot_cell/version.rb
 ].freeze
 
 def suites(*names)
@@ -78,9 +79,9 @@ namespace "version" do
   end
 end
 
-# Releasing is a manual local process: build here, check the five gems, and push them by hand. Each
+# Releasing is a manual local process: build here, check the gems, and push them by hand. Each
 # gem's suite has the test that catches a constant left behind by a version bump.
-desc "Build all five gems into pkg/"
+desc "Build every gem into pkg/"
 task :gems do
   require "fileutils"
 
@@ -98,7 +99,7 @@ task :gems do
   Dir["pkg/*-#{version}.gem"].sort.each { |path| puts "  #{File.expand_path(path)}" }
 end
 
-# One configuration and one run for all five gems, because the style is one style.
+# One configuration and one run for every gem, because the style is one style.
 desc "Check style"
 task :rubocop do
   sh "rubocop"

yabeda-hotcell/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 37signals, LLC
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

yabeda-hotcell/README.md

@@ -0,0 +1,3 @@
+# yabeda-hotcell
+
+Part of [HotCell](https://github.com/basecamp/hotcell). See the repository README.

yabeda-hotcell/Rakefile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |task|
+  task.libs << "test"
+  task.test_files = FileList["test/**/*_test.rb"]
+  task.warning = true
+end
+
+task default: :test

yabeda-hotcell/lib/yabeda-hotcell.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+# Bundler auto-requires a gem named "yabeda-hotcell" as "yabeda-hotcell", then as "yabeda/hotcell". This gem
+# uses neither path, because hot_cell/ is what yields the HotCell constant under the default inflection.
+require "yabeda/hot_cell"

yabeda-hotcell/lib/yabeda/hot_cell.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+require "active_support"
+require "hot_cell/client"
+require "yabeda"
+
+require "yabeda/hot_cell/version"
+
+module Yabeda
+  # A cell's control socket is host-local, and the collect block runs in every scraped process on every host,
+  # which matches that topology exactly.
+  #
+  # Two namespace traps here, both silent. Inside `module Yabeda`, `HotCell` resolves to this module, so the
+  # client must be named ::HotCell. And `hotcell` is a DSL method that exists only inside Yabeda.configure, so
+  # anything factored out of the collect block must say Yabeda.hotcell.
+  module HotCell
+    def self.install!
+      Yabeda.configure do
+        group :hotcell
+
+        counter :requests, comment: "Calls through perform_in_hotcell, by outcome",
+          tags: %i[ cell operation code cause ]
+        histogram :perform, comment: "Time the cell spent performing", unit: :seconds,
+          tags: %i[ cell operation ], buckets: [ 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 120 ]
+
+        gauge :up, comment: "1 when the local cell answers its control socket",
+          tags: %i[ cell ], aggregation: :most_recent
+        gauge :running, comment: "Workers busy right now", tags: %i[ cell ], aggregation: :most_recent
+        gauge :queued, comment: "Connections waiting for a worker", tags: %i[ cell ], aggregation: :most_recent
+        gauge :queue_high_water, comment: "Deepest the queue has been since boot",
+          tags: %i[ cell ], aggregation: :most_recent
+        gauge :cancelled, comment: "Callers that gave up before the cell answered (a floor)",
+          tags: %i[ cell ], aggregation: :most_recent
+        gauge :killed, comment: "Workers killed since boot, by cause",
+          tags: %i[ cell cause ], aggregation: :most_recent
+        gauge :uptime_seconds, comment: "Seconds since the supervisor booted",
+          tags: %i[ cell ], aggregation: :most_recent
+
+        collect { Yabeda::HotCell.collect_stats }
+      end
+
+      subscribe_to_performs
+    end
+
+    def self.collect_stats
+      ::HotCell.cells.each_value do |cell|
+        next unless cell.enabled?
+
+        response = cell.metrics
+        Yabeda.hotcell.up.set({ cell: cell.name }, response&.ok? ? 1 : 0)
+        next unless response&.ok?
+
+        set_counters cell, response.result
+      end
+    rescue => error
+      # A scrape must not fail because a cell is misbehaving. In a Rails application this is Rails.error.
+      ::ActiveSupport.error_reporter.report error, handled: true
+    end
+
+    # The subscriber raises into whoever called instrument, so an unguarded bug here would arrive as a failed
+    # call rather than as missing metrics.
+    def self.subscribe_to_performs
+      ::ActiveSupport::Notifications.subscribe "perform.hot_cell" do |event|
+        record_perform event
+      rescue => error
+        ::ActiveSupport.error_reporter.report error, handled: true
+      end
+    end
+
+    # A failed call is not reported from here: the raise the caller sees already reaches the error reporter.
+    def self.record_perform(event)
+      labels = { cell: event.payload[:cell], operation: event.payload[:operation] }
+
+      # Empty rather than absent, because a label that is sometimes missing is a separate series in Prometheus
+      # and a query by code would silently split.
+      Yabeda.hotcell.requests.increment(labels.merge(code: event.payload[:code] || "ok",
+                                                     cause: event.payload[:cause].to_s))
+      Yabeda.hotcell.perform.measure(labels, (event.payload[:perform_ms] || 0) / 1000.0)
+    end
+
+    private_class_method def self.set_counters(cell, counters)
+      tags = { cell: cell.name }
+
+      Yabeda.hotcell.running.set(tags, counters[:running])
+      Yabeda.hotcell.queued.set(tags, counters[:queued])
+      Yabeda.hotcell.queue_high_water.set(tags, counters[:queue_high_water])
+      Yabeda.hotcell.cancelled.set(tags, counters[:cancelled])
+      Yabeda.hotcell.uptime_seconds.set(tags, counters[:uptime_s])
+      ::HotCell::Codes::PERMANENT_BY_CAUSE.each_key do |cause|
+        Yabeda.hotcell.killed.set(tags.merge(cause: cause), counters[:killed_by].fetch(cause.to_sym, 0))
+      end
+    end
+  end
+end

yabeda-hotcell/lib/yabeda/hot_cell/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Yabeda
+  module HotCell
+    VERSION = "0.6.0.dev"
+  end
+end