ci: 👷 add a bare-metal benchmark testbed via RunsOn

[Panadestein]

🤖 AI text below 🤖

Summary

The benchmarks tracked by Bencher currently run on a shared GitHub-hosted runner. Its timing noise is why alerts on main are recorded but do not fail the build — the noise floor is wide enough that a real regression and a noisy neighbour look alike.

This adds a second testbed on a dedicated AWS node provisioned per run by RunsOn. The node exists only while the workflow runs, so it is billed per commit to main rather than continuously, and nothing else is scheduled on it meanwhile. That lower noise floor is the point: it is what a testbed needs before its timing thresholds can be tightened enough to gate on.

To avoid a second copy of the ~100-line pipeline, the build/benchmark/upload steps move into a reusable workflow. Both callers now differ only in the three things that actually differ: the runner, the testbed name and the result label.

The AWS side is deliberately not set up here — that is still to be agreed. Everything in this PR is inert until it is, in two independent ways:

• The job is gated on a BENCH_BARE_METAL repository variable. Without a RunsOn stack answering the label the job would sit queued until it timed out, once per commit, so it is skipped rather than hung.
• On public repositories RunsOn reads .github/runs-on.yml from the default branch only, so the runner definition here has no effect until this PR merges.

Not Bencher's own Self-Hosted Bare Metal Runners product, which is a much heavier thing (its own runner daemon, OCI images, Firecracker sandboxes). This repo only ever hands Bencher a JSON file over HTTP, and that does not change.

Changes

• .github/workflows/bench.yml (new): the reusable build → benchmark → BMF → upload pipeline, taking runner, testbed, label and optional bench-args. Inputs reach the shell through env: rather than ${{ }} interpolation.
• .github/workflows/bench_main.yml: now a caller — GitHub-hosted ubuntu-26.04, testbed ubuntu-26.04, label ci-linux. Triggers, concurrency and every bencher run flag are unchanged, so the existing history continues uninterrupted. It gains an explicit contents: read because a reusable-workflow call has to name the permissions it grants.
• .github/workflows/bench_bare_metal.yml (new): the gated caller — testbed aws-c7i-4xlarge, label ci-bare-metal.
• .github/runs-on.yml (new): the bench runner shape, c7i / 16 vCPU / 32 GB / gp3, on-demand.
• justfile: bench-ci takes *ARGS and forwards them to pytest.
• Docs: a "Testbeds" subsection in docs/content/docs/benchmarks.mdx, plus an AGENTS.md key-files entry.

Notes for review

Why every runner field is pinned rather than ranged. Bencher keys history on (branch, testbed, measure), so a testbed name means one machine shape. Letting RunsOn pick from a family range, or fall back to spot, would let the hardware change under a fixed name — which shows up as a step change in every series rather than a data point. Spot would cost us twice: an interruption also loses the run outright.

Problem sizes are unchanged, on purpose. A bigger machine invites bigger problems, but the sizes cannot be calibrated against a node that does not exist yet, and once a testbed's history starts its sizes are fixed forever. So this adds the capability (bench-args → bench-ci → pytest) without baking in numbers; e.g. -m "" to include the slow fixed models. Worth doing as a follow-up once the node is real and we can see the timings.

A new testbed starts empty. Its first runs raise no alerts, and its thresholds want setting once there are enough points to see the noise. The --thresholds-reset flags in bench.yml apply per testbed, so the bare-metal one gets the same starting thresholds as the shared runner; tightening them is the follow-up that justifies the whole exercise.

Deferred: pinning monoprop_NUM_THREADS for the bare-metal node. An empty-string env var is worse than an unset one here, so it wants a real value chosen against the real machine.

Checklist

• Tests added or updated to cover the changes
• Documentation updated (docstrings, docs/, CONTRIBUTING.md) if needed
• CHANGELOG / release notes updated if applicable

No tests: the change is workflow plumbing. The bench-ci passthrough was verified locally against pytest --collect-only (extra args reach pytest; a later -m overrides the recipe's -m "not slow").

AI/LLM disclosure

• I did not use LLM tooling, or used it only privately for ideation
• I used the following tool to help write this PR description: GitHub Copilot (Claude Opus 4.5)
• I used the following tool to generate or modify code: GitHub Copilot (Claude Opus 4.5)

[github-actions]

Docs preview: https://pr-254.monoprop-docs.pages.dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (8ba1554) to head (55db424).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #254   +/-   ##
=======================================
  Coverage   97.70%   97.70%           
=======================================
  Files          14       14           
  Lines         742      742           
  Branches       98       98           
=======================================
  Hits          725      725           
  Misses         12       12           
  Partials        5        5           

Flag	Coverage Δ
cpp	97.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[robertodr]

this is very exciting 🤓 I have comments

[Copilot]

Pull request overview

Refactors benchmark CI into a reusable workflow and adds an opt-in RunsOn bare-metal testbed.

Changes:

• Centralizes build, benchmark, conversion, and Bencher upload steps.
• Adds gated bare-metal workflow and pinned RunsOn configuration.
• Forwards extra pytest arguments and documents benchmark testbeds.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Review result
justfile	Forwards extra benchmark arguments to pytest.
docs/content/docs/benchmarks.mdx	Documents benchmark testbeds.
AGENTS.md	Updates benchmark workflow guidance.
.github/workflows/bench.yml	Critical (1 vote): Missing libhwloc-dev and pkg-config may cause clean-runner setup to fail. Moderate (3 votes): Unquoted bench-args expansion does not safely preserve quoted or spaced arguments.
.github/workflows/bench_main.yml	Uses the reusable workflow for the GitHub-hosted benchmark caller.
.github/workflows/bench_bare_metal.yml	Adds the gated bare-metal benchmark caller.
.github/runs-on.yml	Critical (1 vote): The Ubuntu 24.04 image may provide Boost 1.83, below the project’s required Boost 1.85, causing CMake configuration to fail.

Suppressed comments (4)

.github/workflows/bench.yml:38

• 🤖 AI text below 🤖
  This reusable job has no explicit timeout-minutes. If a benchmark deadlocks or a RunsOn allocation stalls after startup, the dedicated EC2 node can remain billed until GitHub's default job limit, while cancel-in-progress: false leaves later commits queued behind it. Add an explicit upper bound sized for the build plus benchmark duration before enabling the bare-metal variable.

    runs-on: ${{ inputs.runner }}

.github/workflows/bench_bare_metal.yml:3

• 🤖 AI text below 🤖 This adds a second benchmark testbed and changes the CI workflow plumbing, but the top-level README still only describes the single generic Bencher run. The repository's documentation policy in AGENTS.md:179-184 requires README updates for build/test workflow changes; please add the new reusable/bare-metal testbed information there so the top-level contributor-facing summary remains accurate.

# The same benchmark profile as `bench_main.yml`, on a dedicated EC2 node

AGENTS.md:88

• 🤖 AI text below 🤖
  The new benchmark workflow/testbed behavior is documented in AGENTS.md and docs/, but not in README.md. The repository documentation policy requires README updates for workflow changes; please add a concise note or link to the Testbeds section so the top-level benchmark description reflects the optional RunsOn path.

- `.github/workflows/bench.yml`: the reusable build/benchmark/upload workflow; `bench_main.yml` and
  `bench_bare_metal.yml` are thin callers that only pick a runner, a testbed name and a label.

docs/content/docs/benchmarks.mdx:488

• 🤖 AI text below 🤖
  This adds a contributor-facing benchmark workflow, a repository variable gate, and a second testbed, but the README benchmark section still only describes a generic Bencher run. The repository documentation policy requires README updates when build/test workflows change; please add a short summary or link for the BENCH_BARE_METAL setup so the top-level documentation does not omit the new workflow.

#### Testbeds

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud