From 51d61f22ed8f539061c680dff80efb13aff043fa Mon Sep 17 00:00:00 2001 From: Mike Dalessio Date: Mon, 28 Sep 2026 17:28:47 -0400 Subject: [PATCH] Ship Yabeda metrics as the yabeda-hotcell gem Applications that call a cell each wrote the same Yabeda integration for its metrics. Add the yabeda-hotcell gem, which carries that integration: a counter and a histogram for every call, and gauges read from each registered cell on every scrape. [Fix #12] --- CHANGELOG.md | 10 +- CONTRIBUTING.md | 7 +- Gemfile | 2 + Gemfile.lock | 21 +++++ README.md | 24 ++++- Rakefile | 13 +-- yabeda-hotcell/MIT-LICENSE | 20 ++++ yabeda-hotcell/README.md | 3 + yabeda-hotcell/Rakefile | 11 +++ yabeda-hotcell/lib/yabeda-hotcell.rb | 5 + yabeda-hotcell/lib/yabeda/hot_cell.rb | 94 +++++++++++++++++++ yabeda-hotcell/lib/yabeda/hot_cell/version.rb | 7 ++ yabeda-hotcell/test/collect_test.rb | 81 ++++++++++++++++ yabeda-hotcell/test/perform_test.rb | 67 +++++++++++++ yabeda-hotcell/test/test_helper.rb | 63 +++++++++++++ yabeda-hotcell/test/version_test.rb | 13 +++ yabeda-hotcell/yabeda-hotcell.gemspec | 30 ++++++ 17 files changed, 457 insertions(+), 14 deletions(-) create mode 100644 yabeda-hotcell/MIT-LICENSE create mode 100644 yabeda-hotcell/README.md create mode 100644 yabeda-hotcell/Rakefile create mode 100644 yabeda-hotcell/lib/yabeda-hotcell.rb create mode 100644 yabeda-hotcell/lib/yabeda/hot_cell.rb create mode 100644 yabeda-hotcell/lib/yabeda/hot_cell/version.rb create mode 100644 yabeda-hotcell/test/collect_test.rb create mode 100644 yabeda-hotcell/test/perform_test.rb create mode 100644 yabeda-hotcell/test/test_helper.rb create mode 100644 yabeda-hotcell/test/version_test.rb create mode 100644 yabeda-hotcell/yabeda-hotcell.gemspec diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f5c9fe..ed71d96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,12 +1,13 @@ # HotCell changelog -This changelog covers five gems, which release together on the same version: +This changelog covers these gems, which release together on the same version: - `hotcell-core` - `hotcell-client` - `hotcell-server` - `activestorage-hotcell-client` - `activestorage-hotcell-server` +- `yabeda-hotcell` A `Tooling` section records changes to the checks and scripts in `bin/` and `examples/`, which ship in no gem but are what an operator runs against their own image. @@ -17,6 +18,7 @@ gem but are what an operator runs against their own image. Some actions that application developers should consider taking when upgrading from an earlier version: +* If the application has its own Yabeda integration for HotCell, replace it with the `yabeda-hotcell` gem. The gem uses the same metric names. It does not write a log line for each call. * Replace a cell's copies of `examples/operations/echo.rb` and `reopen.rb` with `require "hot_cell/health_operations"`, and point the application's clients at `health.echo` and `health.reopen`. See the README's "Rails healthcheck". ### HotCell::Server @@ -36,6 +38,12 @@ Some actions that application developers should consider taking when upgrading f * The client now returns `capacity` when a full cell closes the connection before the client finishes sending the request. Previously, the client returned `unavailable` for the broken pipe. +### Yabeda::HotCell + +#### Added + +* New gem `yabeda-hotcell`. It publishes Yabeda metrics for the application side. Call `Yabeda::HotCell.install!` one time at boot. The gem counts each call by cell, operation, code and cause, and measures the time the cell spent. On each scrape, it reads the counters of each registered cell. See the README's "Metrics collection". + ### Tooling #### Changed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b87b5a5..9972ac3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -77,6 +77,7 @@ hotcell-client/ the application side, and the installer that scaff hotcell-server/ the supervisor, workers, slots, limits, and exe/hotcell activestorage-hotcell-server/ the media operations activestorage-hotcell-client/ the Rails transformer, analyzers and previewers +yabeda-hotcell/ Yabeda metrics for the application side examples/ one cell's worth of sample operations and the battery that drives them bin/ the container checks: example-image, conformance, load @@ -288,16 +289,16 @@ breaks either. ## Making a release -The five gems release together on one version, and `VERSION` at the repository root is what sets it. +The gems release together on one version, and `VERSION` at the repository root is what sets it. - Prechecks - [ ] make sure CI is green - [ ] `bundle exec rake` — the full suite and rubocop - [ ] update `CHANGELOG.md`: retitle `next / unreleased` with the version and the date - - [ ] `bundle exec rake version:bump[1.2.3]` — writes `VERSION` and the five gems' version constants + - [ ] `bundle exec rake version:bump[1.2.3]` — writes `VERSION` and every gem's version constant - [ ] commit, and tag as `v1.2.3` - Release - - [ ] `bundle exec rake gems` — builds the five gems into `pkg/`, which it empties first + - [ ] `bundle exec rake gems` — builds every gem into `pkg/`, which it empties first - [ ] `git push && git push --tags` — **before** the gems: every gemspec's `changelog_uri` and `source_code_uri` name the tag, and they 404 until it is on GitHub - [ ] `for f in pkg/*.gem ; do gem push $f ; done` diff --git a/Gemfile b/Gemfile index d26ac9e..c65f096 100644 --- a/Gemfile +++ b/Gemfile @@ -9,6 +9,8 @@ gemspec path: "hotcell-server", name: "hotcell-server" gemspec path: "activestorage-hotcell-client", name: "activestorage-hotcell-client" gemspec path: "activestorage-hotcell-server", name: "activestorage-hotcell-server" +gemspec path: "yabeda-hotcell", name: "yabeda-hotcell" + # activestorage-hotcell-client needs config.active_storage.variant_processor to accept a class, which is # rails/rails#58384. Merged, unreleased — so this tracks main until 8.2 ships and the gemspec floor can name # a version instead. diff --git a/Gemfile.lock b/Gemfile.lock index 43a4742..9c4c3bf 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -94,9 +94,18 @@ PATH hotcell-server (0.6.0.dev) hotcell-core (= 0.6.0.dev) +PATH + remote: yabeda-hotcell + specs: + yabeda-hotcell (0.6.0.dev) + hotcell-client (= 0.6.0.dev) + yabeda (>= 0.12) + GEM remote: https://rubygems.org/ specs: + anyway_config (2.8.1) + ruby-next-core (~> 1.0) ast (2.4.3) base64 (0.3.0) bigdecimal (4.1.3) @@ -106,6 +115,7 @@ GEM crass (1.0.7) date (3.5.1) drb (2.2.3) + dry-initializer (3.2.0) erb (6.0.7) erubi (1.13.1) ffi (1.17.4) @@ -244,6 +254,7 @@ GEM rubocop-rake (0.7.1) lint_roller (~> 1.1) rubocop (>= 1.72.1) + ruby-next-core (1.2.1) ruby-progressbar (1.13.0) ruby-vips (2.3.0) ffi (~> 1.12) @@ -260,6 +271,10 @@ GEM unicode-emoji (4.3.0) uri (1.1.1) useragent (0.16.11) + yabeda (0.16.0) + anyway_config (>= 1.0, < 3) + concurrent-ruby + dry-initializer zeitwerk (2.8.3) PLATFORMS @@ -291,6 +306,7 @@ DEPENDENCIES rubocop-packaging rubocop-performance rubocop-rake + yabeda-hotcell! CHECKSUMS actionpack (8.2.0.alpha) @@ -302,6 +318,7 @@ CHECKSUMS activestorage-hotcell-client (0.6.0.dev) activestorage-hotcell-server (0.6.0.dev) activesupport (8.2.0.alpha) + anyway_config (2.8.1) sha256=541842b25117fac3e277670c32dc09d3b715b20404826a6538ac35527583624f ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383 base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b bigdecimal (4.1.3) sha256=61ebe1e5e559bdc3cc6f2c0ee7f427321fc838f59611c294356eb04d6e21cf66 @@ -312,6 +329,7 @@ CHECKSUMS crass (1.0.7) sha256=94868719948664c89ddcaf0a37c65048413dfcb1c869470a5f7a7ceb5390b295 date (3.5.1) sha256=750d06384d7b9c15d562c76291407d89e368dda4d4fff957eb94962d325a0dc0 drb (2.2.3) sha256=0b00d6fdb50995fe4a45dea13663493c841112e4068656854646f418fda13373 + dry-initializer (3.2.0) sha256=37d59798f912dc0a1efe14a4db4a9306989007b302dcd5f25d0a2a20c166c4e3 erb (6.0.7) sha256=c5ca6dc25b0ef974a44dc8f59fe847577122483b1968a38dec305c60bf91ee92 erubi (1.13.1) sha256=a082103b0885dbc5ecf1172fede897f9ebdb745a4b97a5e8dc63953db1ee4ad9 ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf @@ -389,6 +407,7 @@ CHECKSUMS rubocop-packaging (0.6.0) sha256=fb92bd0fb48e6f8cdb1648d2249b0cd51c2497dcc87340132d22f01edbf558a7 rubocop-performance (1.27.0) sha256=eeeb1374d062a368ee1c787b70eb0b0cc4b184cb1f8565f424760946146d61ce rubocop-rake (0.7.1) sha256=3797f2b6810c3e9df7376c26d5f44f3475eda59eb1adc38e6f62ecf027cbae4d + ruby-next-core (1.2.1) sha256=ddba3e986e127143299a26bde8bdbeedd4bc738a1a765e499f9634f361551f79 ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33 ruby-vips (2.3.0) sha256=e685ec02c13969912debbd98019e50492e12989282da5f37d05f5471442f5374 securerandom (0.4.1) sha256=cc5193d414a4341b6e225f0cb4446aceca8e50d5e1888743fac16987638ea0b1 @@ -401,6 +420,8 @@ CHECKSUMS unicode-emoji (4.3.0) sha256=11c02fa73290378c066bb0562cd4c87d8e1b706fbbe0059ca12746a4244de8ce uri (1.1.1) sha256=379fa58d27ffb1387eaada68c749d1426738bd0f654d812fcc07e7568f5c57c6 useragent (0.16.11) sha256=700e6413ad4bb954bb63547fa098dddf7b0ebe75b40cc6f93b8d54255b173844 + yabeda (0.16.0) sha256=7f6e51acd7d9a51d850ea8c3844f72a24882f1312b3fc3836052bcd63d384cba + yabeda-hotcell (0.6.0.dev) zeitwerk (2.8.3) sha256=2c85125a8467ce069e20123d1e709a08955c9d29c118c25b46b7b7fafdbb92e5 BUNDLED WITH diff --git a/README.md b/README.md index c25f265..e35a721 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,7 @@ hogs. Whatever is putting your trusted core application at risk, move it out! | `hotcell-server` | the cell | The supervisor, the worker, `HotCell::Operation`, the container image. | | `activestorage-hotcell-client` | the application | The transformer, analyzer, and previewers Rails is configured with. | | `activestorage-hotcell-server` | the cell | The `transformers.image.*`, `analyzers.image.*`, `analyzers.media.ffprobe`, and `previewers.*` operations. | +| `yabeda-hotcell` | the application | Yabeda metrics for every call and for each local cell's counters. | They are in one repository because they are being developed together today. We may split out the Active Storage gems into another repository at a later date. @@ -546,10 +547,25 @@ these too. Alert on the presence of `worker.crashed`, and on `worker.killed` by ### Metrics collection -Poll `cell.metrics` on a schedule, for example with a [Yabeda](https://github.com/yabeda-rb/yabeda) -`collect` block. The control socket answers even while the work socket is saturated, and it is -host-local, so the poller must be a process on the cell's own host. Watch `queued`, -`queue_high_water`, `cancelled`, and `killed_by` cause. +The `yabeda-hotcell` gem publishes [Yabeda](https://github.com/yabeda-rb/yabeda) metrics. Add it to the +application's `Gemfile`, and install it once at boot: + +```ruby +# Gemfile +gem "yabeda-hotcell" + +# config/initializers/hotcell.rb +Yabeda::HotCell.install! +``` + +The metrics are in the `hotcell` group. The `requests` counter counts every call, tagged with `cell`, +`operation`, `code` and `cause`, and the `perform` histogram measures the time the cell spent. On each +scrape the gem polls `cell.metrics` from every registered cell and sets the gauges `up`, `running`, +`queued`, `queue_high_water`, `cancelled`, `killed` (by `cause`), and `uptime_seconds`. + +The control socket answers even while the work socket is saturated, and it is host-local, so the +scraped process must be on the cell's own host. Watch `queued`, `queue_high_water`, `cancelled`, and +`killed` by cause. ### Per-call telemetry diff --git a/Rakefile b/Rakefile index 4f396bd..62c7ca9 100644 --- a/Rakefile +++ b/Rakefile @@ -1,11 +1,11 @@ # frozen_string_literal: true -# The three hotcell gems need no tools and no container: their suites run on fixture operations in a few -# seconds, and so does the development cell the battery drives. The two activestorage-hotcell gems convert +# The hotcell gems and yabeda-hotcell need no tools and no container: their suites run on fixture operations in +# a few seconds, and so does the development cell the battery drives. The two activestorage-hotcell gems convert # real files and need libvips, mutool, ffmpeg and ffprobe installed. That split is a design property rather # than an accident, so the tasks keep it visible: `test:hotcell` is what CI runs on a machine with nothing # installed, and on macOS. -HOTCELL = %w[ hotcell-core hotcell-client hotcell-server ].freeze +HOTCELL = %w[ hotcell-core hotcell-client hotcell-server yabeda-hotcell ].freeze ACTIVE_STORAGE = %w[ activestorage-hotcell-server activestorage-hotcell-client ].freeze GEMS = (HOTCELL + ACTIVE_STORAGE).freeze @@ -18,6 +18,7 @@ VERSION_FILES = %w[ hotcell-server/lib/hot_cell/server/version.rb activestorage-hotcell-client/lib/active_storage/hot_cell/client/version.rb activestorage-hotcell-server/lib/active_storage/hot_cell/server/version.rb + yabeda-hotcell/lib/yabeda/hot_cell/version.rb ].freeze def suites(*names) @@ -78,9 +79,9 @@ namespace "version" do end end -# Releasing is a manual local process: build here, check the five gems, and push them by hand. Each +# Releasing is a manual local process: build here, check the gems, and push them by hand. Each # gem's suite has the test that catches a constant left behind by a version bump. -desc "Build all five gems into pkg/" +desc "Build every gem into pkg/" task :gems do require "fileutils" @@ -98,7 +99,7 @@ task :gems do Dir["pkg/*-#{version}.gem"].sort.each { |path| puts " #{File.expand_path(path)}" } end -# One configuration and one run for all five gems, because the style is one style. +# One configuration and one run for every gem, because the style is one style. desc "Check style" task :rubocop do sh "rubocop" diff --git a/yabeda-hotcell/MIT-LICENSE b/yabeda-hotcell/MIT-LICENSE new file mode 100644 index 0000000..ba43727 --- /dev/null +++ b/yabeda-hotcell/MIT-LICENSE @@ -0,0 +1,20 @@ +Copyright (c) 37signals, LLC + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/yabeda-hotcell/README.md b/yabeda-hotcell/README.md new file mode 100644 index 0000000..a1be775 --- /dev/null +++ b/yabeda-hotcell/README.md @@ -0,0 +1,3 @@ +# yabeda-hotcell + +Part of [HotCell](https://github.com/basecamp/hotcell). See the repository README. diff --git a/yabeda-hotcell/Rakefile b/yabeda-hotcell/Rakefile new file mode 100644 index 0000000..a6c4f84 --- /dev/null +++ b/yabeda-hotcell/Rakefile @@ -0,0 +1,11 @@ +# frozen_string_literal: true + +require "rake/testtask" + +Rake::TestTask.new(:test) do |task| + task.libs << "test" + task.test_files = FileList["test/**/*_test.rb"] + task.warning = true +end + +task default: :test diff --git a/yabeda-hotcell/lib/yabeda-hotcell.rb b/yabeda-hotcell/lib/yabeda-hotcell.rb new file mode 100644 index 0000000..4c38966 --- /dev/null +++ b/yabeda-hotcell/lib/yabeda-hotcell.rb @@ -0,0 +1,5 @@ +# frozen_string_literal: true + +# Bundler auto-requires a gem named "yabeda-hotcell" as "yabeda-hotcell", then as "yabeda/hotcell". This gem +# uses neither path, because hot_cell/ is what yields the HotCell constant under the default inflection. +require "yabeda/hot_cell" diff --git a/yabeda-hotcell/lib/yabeda/hot_cell.rb b/yabeda-hotcell/lib/yabeda/hot_cell.rb new file mode 100644 index 0000000..4a1ae0f --- /dev/null +++ b/yabeda-hotcell/lib/yabeda/hot_cell.rb @@ -0,0 +1,94 @@ +# frozen_string_literal: true + +require "active_support" +require "hot_cell/client" +require "yabeda" + +require "yabeda/hot_cell/version" + +module Yabeda + # A cell's control socket is host-local, and the collect block runs in every scraped process on every host, + # which matches that topology exactly. + # + # Two namespace traps here, both silent. Inside `module Yabeda`, `HotCell` resolves to this module, so the + # client must be named ::HotCell. And `hotcell` is a DSL method that exists only inside Yabeda.configure, so + # anything factored out of the collect block must say Yabeda.hotcell. + module HotCell + def self.install! + Yabeda.configure do + group :hotcell + + counter :requests, comment: "Calls through perform_in_hotcell, by outcome", + tags: %i[ cell operation code cause ] + histogram :perform, comment: "Time the cell spent performing", unit: :seconds, + tags: %i[ cell operation ], buckets: [ 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 120 ] + + gauge :up, comment: "1 when the local cell answers its control socket", + tags: %i[ cell ], aggregation: :most_recent + gauge :running, comment: "Workers busy right now", tags: %i[ cell ], aggregation: :most_recent + gauge :queued, comment: "Connections waiting for a worker", tags: %i[ cell ], aggregation: :most_recent + gauge :queue_high_water, comment: "Deepest the queue has been since boot", + tags: %i[ cell ], aggregation: :most_recent + gauge :cancelled, comment: "Callers that gave up before the cell answered (a floor)", + tags: %i[ cell ], aggregation: :most_recent + gauge :killed, comment: "Workers killed since boot, by cause", + tags: %i[ cell cause ], aggregation: :most_recent + gauge :uptime_seconds, comment: "Seconds since the supervisor booted", + tags: %i[ cell ], aggregation: :most_recent + + collect { Yabeda::HotCell.collect_stats } + end + + subscribe_to_performs + end + + def self.collect_stats + ::HotCell.cells.each_value do |cell| + next unless cell.enabled? + + response = cell.metrics + Yabeda.hotcell.up.set({ cell: cell.name }, response&.ok? ? 1 : 0) + next unless response&.ok? + + set_counters cell, response.result + end + rescue => error + # A scrape must not fail because a cell is misbehaving. In a Rails application this is Rails.error. + ::ActiveSupport.error_reporter.report error, handled: true + end + + # The subscriber raises into whoever called instrument, so an unguarded bug here would arrive as a failed + # call rather than as missing metrics. + def self.subscribe_to_performs + ::ActiveSupport::Notifications.subscribe "perform.hot_cell" do |event| + record_perform event + rescue => error + ::ActiveSupport.error_reporter.report error, handled: true + end + end + + # A failed call is not reported from here: the raise the caller sees already reaches the error reporter. + def self.record_perform(event) + labels = { cell: event.payload[:cell], operation: event.payload[:operation] } + + # Empty rather than absent, because a label that is sometimes missing is a separate series in Prometheus + # and a query by code would silently split. + Yabeda.hotcell.requests.increment(labels.merge(code: event.payload[:code] || "ok", + cause: event.payload[:cause].to_s)) + Yabeda.hotcell.perform.measure(labels, (event.payload[:perform_ms] || 0) / 1000.0) + end + + private_class_method def self.set_counters(cell, counters) + tags = { cell: cell.name } + + Yabeda.hotcell.running.set(tags, counters[:running]) + Yabeda.hotcell.queued.set(tags, counters[:queued]) + Yabeda.hotcell.queue_high_water.set(tags, counters[:queue_high_water]) + Yabeda.hotcell.cancelled.set(tags, counters[:cancelled]) + Yabeda.hotcell.uptime_seconds.set(tags, counters[:uptime_s]) + ::HotCell::Codes::PERMANENT_BY_CAUSE.each_key do |cause| + Yabeda.hotcell.killed.set(tags.merge(cause: cause), counters[:killed_by].fetch(cause.to_sym, 0)) + end + end + end +end diff --git a/yabeda-hotcell/lib/yabeda/hot_cell/version.rb b/yabeda-hotcell/lib/yabeda/hot_cell/version.rb new file mode 100644 index 0000000..f0424ef --- /dev/null +++ b/yabeda-hotcell/lib/yabeda/hot_cell/version.rb @@ -0,0 +1,7 @@ +# frozen_string_literal: true + +module Yabeda + module HotCell + VERSION = "0.6.0.dev" + end +end diff --git a/yabeda-hotcell/test/collect_test.rb b/yabeda-hotcell/test/collect_test.rb new file mode 100644 index 0000000..f440d8c --- /dev/null +++ b/yabeda-hotcell/test/collect_test.rb @@ -0,0 +1,81 @@ +# frozen_string_literal: true + +require "test_helper" + +# Asserting values rather than that the collect block ran, because the namespace traps in the collector are +# silent: a bare `hotcell` records nothing, and a bare `HotCell` names this gem rather than the client. +class CollectTest < YabedaHotCellTest + def test_publishes_a_running_cells_counters + HotCell::TestCell.boot do |cell| + HotCell.root = cell.socket_root + register + + Yabeda.collect! + end + + assert_equal 1, gauge(:up) + assert_equal 0, gauge(:running) + assert_equal 0, gauge(:queued) + assert_equal 0, gauge(:queue_high_water) + assert_equal 0, gauge(:cancelled) + assert_kind_of Integer, gauge(:uptime_seconds) + end + + def test_publishes_kills_by_cause + HotCell.root = "/nowhere" + register transport: CannedTransport.new(metrics(killed_by: { memory: 5, deadline: 2 })) + + Yabeda.collect! + + assert_equal 5, gauge(:killed, cause: "memory") + assert_equal 2, gauge(:killed, cause: "deadline") + end + + # A restarted cell reports only the causes it has seen since boot, so a cause it omits has to read zero + # rather than keep the count from before the restart. + def test_a_kill_cause_the_cell_omits_reads_zero + HotCell.root = "/nowhere" + register transport: CannedTransport.new(metrics(killed_by: { memory: 5 })) + Yabeda.collect! + + register transport: CannedTransport.new(metrics(killed_by: {})) + Yabeda.collect! + + assert_equal 0, gauge(:killed, cause: "memory") + end + + def test_a_cell_that_does_not_answer_is_down_rather_than_missing + Dir.mktmpdir do |root| + HotCell.root = root + register + + Yabeda.collect! + end + + assert_equal 0, gauge(:up) + assert_nil gauge(:running) + end + + def test_a_cell_with_no_root_publishes_nothing + register + + Yabeda.collect! + + assert_nil gauge(:up) + end + + def test_a_scrape_reports_a_misbehaving_cell_rather_than_failing + HotCell.root = "/nowhere" + register transport: BrokenTransport.new + + Yabeda.collect! + + assert_equal [ "the transport itself is broken" ], @reported.map(&:message) + end + + private + def metrics(**counters) + HotCell::Response.new(result: { uptime_s: 41, running: 2, queued: 3, queue_high_water: 7, cancelled: 1, + requests: {}, killed_by: {}, **counters }) + end +end diff --git a/yabeda-hotcell/test/perform_test.rb b/yabeda-hotcell/test/perform_test.rb new file mode 100644 index 0000000..ca37520 --- /dev/null +++ b/yabeda-hotcell/test/perform_test.rb @@ -0,0 +1,67 @@ +# frozen_string_literal: true + +require "test_helper" + +class PerformTest < YabedaHotCellTest + def test_counts_a_successful_call_as_ok_and_measures_what_the_cell_spent + answer HotCell::Response.new(result: {}, timing: { perform_ms: 250 }) + + Anything.perform_in_hotcell [], [], {} + + assert_equal 1, requests(code: "ok", cause: "") + assert_in_delta 0.25, perform_seconds + end + + def test_counts_a_failure_under_its_own_code + answer failed(code: "capacity") + + assert_raises(TemporarilyUnavailable) { Anything.perform_in_hotcell [], [], {} } + + assert_equal 1, requests(code: "capacity", cause: "") + end + + # `killed` is one code and several verdicts: which limit the worker hit decides whether the file did it. + # Every other code carries an empty cause, because a label that is sometimes absent is a separate series in + # Prometheus, and a query by code would silently split. + def test_counts_a_kill_under_its_cause + answer failed(code: "killed", cause: "fsize") + + assert_raises(Unprocessable) { Anything.perform_in_hotcell [], [], {} } + + assert_equal 1, requests(code: "killed", cause: "fsize") + end + + # A subscriber raises into whoever called instrument, so a bug here would otherwise arrive as a failed call. + def test_a_metrics_bug_arrives_as_a_report_rather_than_as_a_failed_call + ActiveSupport::Notifications.instrument("perform.hot_cell", cell: "test", perform_ms: "not a number") { } + + assert_equal [ NoMethodError ], @reported.map(&:class) + end + + private + def answer(response) + HotCell.root = "/nowhere" + register transport: CannedTransport.new(response) + end + + def failed(code:, cause: nil) + HotCell::Response.failed HotCell::Failure.new(code: code, cause: cause, message: "no"), timing: { perform_ms: 1 } + end + + def requests(**tags) + Yabeda::TestAdapter.instance.counters[Yabeda.hotcell.requests][{ **labels, **tags }] + end + + def perform_seconds + Yabeda::TestAdapter.instance.histograms[Yabeda.hotcell.perform][labels] + end + + def labels + { cell: "test", operation: "test.anything" } + end + + class Anything < HotCell::Client + hotcell "test" + operation "test.anything" + end +end diff --git a/yabeda-hotcell/test/test_helper.rb b/yabeda-hotcell/test/test_helper.rb new file mode 100644 index 0000000..66eab7a --- /dev/null +++ b/yabeda-hotcell/test/test_helper.rb @@ -0,0 +1,63 @@ +# frozen_string_literal: true + +$LOAD_PATH.unshift File.expand_path("../lib", __dir__) + +require "minitest/autorun" +require "logger" + +require "yabeda/hot_cell" +require "yabeda/testing" +require "hot_cell/test_cell" + +Yabeda::HotCell.install! +Yabeda.configure! + +class YabedaHotCellTest < Minitest::Test + # Stand-ins for an application's own classes, which the gem must never name for itself. + class Unprocessable < StandardError; end + class TemporarilyUnavailable < StandardError; end + + def setup + HotCell.reset_registrations! + HotCell.logger = Logger.new(File::NULL) + Yabeda::TestAdapter.instance.reset! + ActiveSupport.error_reporter.subscribe(@reported = ReportedErrors.new) + end + + def teardown + ActiveSupport.error_reporter.unsubscribe @reported + HotCell.reset_registrations! + HotCell.logger = nil + end + + private + def register(**options) + HotCell.register "test", permanent: Unprocessable, transient: TemporarilyUnavailable, **options + end + + def gauge(metric, **tags) + Yabeda::TestAdapter.instance.gauges[Yabeda.hotcell.public_send(metric)][{ cell: "test", **tags }] + end + + class ReportedErrors < Array + def report(error, **) + self << error + end + end + + class CannedTransport + def initialize(response) + @response = response + end + + def call(_cell, _line, _descriptors, socket: nil, timeout: nil) + @response + end + end + + class BrokenTransport + def call(_cell, _line, _descriptors, socket: nil, timeout: nil) + raise "the transport itself is broken" + end + end +end diff --git a/yabeda-hotcell/test/version_test.rb b/yabeda-hotcell/test/version_test.rb new file mode 100644 index 0000000..1a47ef4 --- /dev/null +++ b/yabeda-hotcell/test/version_test.rb @@ -0,0 +1,13 @@ +# frozen_string_literal: true + +require "test_helper" + +class VersionTest < YabedaHotCellTest + # The gemspec builds from the repository's VERSION file and this constant is written from it, so the two + # can drift. `rake version:bump` writes both. + def test_this_gem_reports_the_version_its_gemspec_builds + version = File.read(File.expand_path("../../VERSION", __dir__)).strip + + assert_equal version, Yabeda::HotCell::VERSION, "run `rake version:bump` from the repository root" + end +end diff --git a/yabeda-hotcell/yabeda-hotcell.gemspec b/yabeda-hotcell/yabeda-hotcell.gemspec new file mode 100644 index 0000000..b5a7b3f --- /dev/null +++ b/yabeda-hotcell/yabeda-hotcell.gemspec @@ -0,0 +1,30 @@ +# frozen_string_literal: true + +version = File.read(File.expand_path("../VERSION", __dir__)).strip + +Gem::Specification.new do |spec| + spec.name = "yabeda-hotcell" + spec.version = version + spec.authors = [ "Mike Dalessio" ] + spec.email = [ "mike@37signals.com" ] + spec.license = "MIT" + spec.homepage = "https://github.com/basecamp/hotcell" + spec.summary = "Yabeda metrics for HotCell." + spec.description = <<~TEXT + Yabeda metrics for an application that calls HotCell cells: a counter and a histogram for every call, + and gauges polled from each local cell's control socket. + TEXT + + spec.required_ruby_version = ">= 3.3" + + spec.metadata["homepage_uri"] = spec.homepage + spec.metadata["source_code_uri"] = "#{spec.homepage}/tree/v#{version}/#{spec.name}" + spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/v#{version}/CHANGELOG.md" + spec.metadata["bug_tracker_uri"] = "#{spec.homepage}/issues" + spec.metadata["rubygems_mfa_required"] = "true" + + spec.files = Dir[ "lib/**/*", "MIT-LICENSE", "README.md" ] + + spec.add_dependency "hotcell-client", version + spec.add_dependency "yabeda", ">= 0.12" +end