From 9f67dcb79c2a428bbe21d8c2be784f7208a9f3cd Mon Sep 17 00:00:00 2001
From: Aaron Miller <aaron.miller@algorithmiq.fi>
Date: Tue, 1 Sep 2026 17:57:50 +0100
Subject: [PATCH 1/5] =?UTF-8?q?chore(ci):=20=F0=9F=91=B7=20declare=20each?=
 =?UTF-8?q?=20Bencher=20rung's=20problem,=20shape=20and=20rounds?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

`monoprop_PARTITIONS` was unset everywhere in CI, so `resolve_partition_count_`'s
`ranks == 1 ? cores : 1` decided the shape: every point in the `ubuntu-26.04` and
`aws-c7i-4xlarge` series was one partition per core of whatever runner it landed on,
unrecorded, and a runner resize would have moved every series without moving a testbed
name. Nothing declared a problem either -- the lanes ran the suite's dataclass defaults.

A rung line now carries both: `<name> <ranks> <partitions> <rounds> | <pytest args>`.
L1 is benches/LADDER.md's own L1 rows; L2a and L2b share one problem sized to this
instance rather than to the 128-core node the ladder tables were measured on, and differ
only in shape. Measured on an 8-core mask of a Deucalion x86 node:

  L1   1x1  9.95M / 10.07M / 19.90M terms    2.60 GiB   3:53 at 3 rounds
  L2a  1x8  167,515,463 terms               19.11 GiB    135 s at 1 round
  L2b  4x2  167,515,463 terms         23.71 GiB/node      90 s at 1 round

`--obs-terms=2500000` is the largest measured point that fits: 3500000 reaches 35.49
GiB/node against the instance's 32 GiB, and a fit over 1.5M and 2.5M under-called it by
3.9 GiB, so the axis is read off measured points rather than interpolated. The term count
is identical at both shapes, which is the geometry-independence check.

Rounds is per rung because `pedantic` builds round k+1's propagator before releasing round
k's -- at these sizes that is a memory setting, not a statistics setting.

Placement is asserted rather than printed. `check_shape.py` fails a rung whose recorded
shape differs from the one asked for, or whose `single_cpu_threads_min` is zero while
threads were counted; an all-zero summary means /proc was unreadable, not that nothing was
pinned, so it does not fire there. Both L2 rungs sit exactly on the
`partitions > visible cores` boundary, where the only warning goes to C++ stderr and
pytest's capture eats it.

`--map-by slot:PE=$monoprop_PARTITIONS --bind-to core` moves into `bench-ci-mpi`, which can
no longer run unpinned and refuses without `monoprop_PARTITIONS`; `*ARGS` is freed for the
pytest args the rungs need. The testbed's CPU is slugified from the cpuinfo block
pytest-benchmark already writes into `time-<label>.json`.

bench_main.yml goes, and the README badge with it rather than pointing at a workflow that
is inert until a maintainer sets `vars.BENCH_BARE_METAL`: a GitHub-hosted runner has about
two physical cores, so L2a would be 1x2 and L2b 2x1 -- restatements of L1 through a noise
floor no threshold survives.

Assisted-by: ClaudeCode:claude-opus-5
---
 .github/runs-on.yml                      |   6 +-
 .github/workflows/bench.yml              |  87 +++++---------
 .github/workflows/bench_bare_metal.yml   |  32 +++--
 .github/workflows/bench_main.yml         |  28 -----
 .github/workflows/scripts/check_shape.py |  60 ++++++++++
 .github/workflows/scripts/cpu_slug.py    |  47 ++++++++
 README.md                                |   1 -
 justfile                                 | 145 +++++++++++++++++++++--
 pyproject.toml                           |   2 +
 9 files changed, 292 insertions(+), 116 deletions(-)
 delete mode 100644 .github/workflows/bench_main.yml
 create mode 100644 .github/workflows/scripts/check_shape.py
 create mode 100644 .github/workflows/scripts/cpu_slug.py

diff --git a/.github/runs-on.yml b/.github/runs-on.yml
index 4c6903d1..95eea8ae 100644
--- a/.github/runs-on.yml
+++ b/.github/runs-on.yml
@@ -6,10 +6,8 @@
 runners:
   # Used by .github/workflows/bench_bare_metal.yml.
   #
-  # Every field is pinned rather than ranged. Bencher compares each run against
-  # this testbed's own history, so a different instance type or core count is a
-  # step change in the series rather than a data point -- widening any of these
-  # means retiring the testbed and starting a new one.
+  # Pinned rather than ranged: the testbed is named after this CPU and its core count, so widening
+  # any field renames the testbed and stops the old series rather than corrupting it.
   bench:
     # Avoid expand to partial family
     family: ["c7i.4xlarge"]
diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml
index ee7d701f..0babd636 100644
--- a/.github/workflows/bench.yml
+++ b/.github/workflows/bench.yml
@@ -1,8 +1,8 @@
 name: Benchmark
 
-# Shared body of the Bencher continuous-benchmarking runs. Callers choose the
-# machine and the Bencher testbed; everything downstream is identical, so the
-# testbeds stay comparable in shape even though their absolute numbers are not.
+# Shared body of the Bencher runs; the caller names the machine and the benches/LADDER.md rungs.
+# Every rung runs on the same runner from the same MPI-enabled build and uploads to its own
+# testbed, because Bencher keys history on (branch, testbed, benchmark, measure).
 
 on:
   workflow_call:
@@ -11,24 +11,17 @@ on:
         description: Value for the job's `runs-on`.
         required: true
         type: string
-      testbed:
-        description: Bencher testbed. Must change whenever the hardware does.
-        required: true
-        type: string
       label:
-        description: Benchmark label, used to name the result artifacts.
+        description: Result-artifact prefix; each rung uploads as `<label>-<rung>`.
         required: true
         type: string
-      bench-markers:
-        description: pytest marker expression. Empty selects every benchmark.
-        required: false
-        type: string
-        default: not slow
-      bench-rounds:
-        description: Fixed timing rounds per benchmark.
-        required: false
+      rungs:
+        description: >-
+          One rung per line, `<name> <ranks> <partitions> <rounds> | <pytest args>`.
+          `partitions` is an integer or `per-rank` for (physical cores / ranks).
+          Blank lines and `#` comments are ignored.
+        required: true
         type: string
-        default: "5"
 
 env:
   FORCE_COLOR: 3
@@ -41,6 +34,7 @@ jobs:
     # Skips forks, which have no BENCHER_API_KEY.
     if: ${{ github.repository == 'Algorithmiq/monoprop' }}
     runs-on: ${{ inputs.runner }}
+    timeout-minutes: 120
 
     steps:
       - uses: actions/checkout@v7.0.1
@@ -52,24 +46,27 @@ jobs:
         with:
           cache-suffix: benchmark
 
+      # uv's `cache-keys` cover neither `SKBUILD_*` nor `monoprop_ENABLE_MPI`, so without the
+      # reinstall the warm cache above can hand back an MPI-off wheel and every rung still pass.
       - name: Install package
         env:
           SKBUILD_CMAKE_DEFINE: monoprop_ENABLE_CXX_UNIT_TESTS=OFF
+          monoprop_ENABLE_MPI: "ON"
         run: just build --group bench
 
-      - name: Run benchmarks
-        env:
-          BENCHER_LABEL: ${{ inputs.label }}
-          monoprop_BENCH_MARKERS: ${{ inputs.bench-markers }}
-          monoprop_BENCH_ROUNDS: ${{ inputs.bench-rounds }}
-        run: |
-          just bench-ci "$BENCHER_LABEL"
+      - name: Resolve the core count
+        run: just bench-ci-resolve-cores
 
-      - name: Convert results to Bencher Metric Format
+      - name: Run the ladder rungs
         env:
-          BENCHER_LABEL: ${{ inputs.label }}
-        run: |
-          just bench-bmf "$BENCHER_LABEL" > bmf.json
+          BENCH_LABEL: ${{ inputs.label }}
+          RUNGS: ${{ inputs.rungs }}
+          # Empty, so a rung's `-k` is the only selector and the slow fixed models are reachable.
+          monoprop_BENCH_MARKERS: ""
+          # The RunsOn images run as root, and OpenMPI refuses to launch there without these.
+          OMPI_ALLOW_RUN_AS_ROOT: "1"
+          OMPI_ALLOW_RUN_AS_ROOT_CONFIRM: "1"
+        run: just bench-ci-rungs "$BENCH_LABEL"
 
       - name: Upload the raw measurement
         if: always()
@@ -77,39 +74,19 @@ jobs:
         with:
           name: bench-${{ inputs.label }}
           path: |
-            bmf.json
+            bmf-*.json
             benches/results/*.json
           if-no-files-found: warn
 
       - uses: bencherdev/bencher@v0.6.12
 
-      # The project slug and API key go through the environment rather than the
-      # command line: `bencher` reads BENCHER_PROJECT / BENCHER_API_KEY natively,
-      # and it keeps a maintainer-set variable out of the shell's parsing.
+      # The project slug and API key go through the environment: `bencher` reads
+      # BENCHER_PROJECT / BENCHER_API_KEY natively, and it keeps a maintainer-set
+      # variable out of the shell's parsing.
       - name: Track benchmarks with Bencher
         env:
           BENCHER_PROJECT: ${{ vars.BENCHER_PROJECT || 'monoprop' }}
           BENCHER_API_KEY: ${{ secrets.BENCHER_API_KEY }}
-          BENCHER_TESTBED: ${{ inputs.testbed }}
-        run: |
-          bencher run \
-            --branch main \
-            --adapter json \
-            --file bmf.json \
-            --github-actions "${{ secrets.GITHUB_TOKEN }}" \
-            --thresholds-reset \
-            --threshold-measure latency \
-            --threshold-test t_test \
-            --threshold-max-sample-size 64 \
-            --threshold-lower-boundary _ \
-            --threshold-upper-boundary 0.99 \
-            --threshold-measure peak-memory \
-            --threshold-test percentage \
-            --threshold-max-sample-size 64 \
-            --threshold-lower-boundary _ \
-            --threshold-upper-boundary 0.10 \
-            --threshold-measure terms \
-            --threshold-test percentage \
-            --threshold-max-sample-size 1 \
-            --threshold-lower-boundary 0.0 \
-            --threshold-upper-boundary 0.0
+          BENCHER_BRANCH: ${{ github.ref_name }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: just bench-ci-track
diff --git a/.github/workflows/bench_bare_metal.yml b/.github/workflows/bench_bare_metal.yml
index a70a91a0..aacd7d02 100644
--- a/.github/workflows/bench_bare_metal.yml
+++ b/.github/workflows/bench_bare_metal.yml
@@ -1,12 +1,9 @@
 name: Track benchmarks (bare metal)
 
-# The same benchmark profile as `bench_main.yml`, on a dedicated EC2 node
-# provisioned per run by RunsOn. The point is the noise floor: nothing else is
-# scheduled on the machine, so this testbed can eventually carry thresholds
-# tight enough to gate on, which the shared GitHub runner cannot.
-#
-# The node exists only for the duration of the run, so cost tracks the number
-# of commits to main rather than wall-clock time.
+# monoprop's continuous benchmarking, on a dedicated EC2 node provisioned per run by RunsOn.
+# The point is the noise floor: nothing else is scheduled on the machine, so these testbeds can
+# carry thresholds tight enough to gate on. The node exists only for the run, so cost tracks the
+# number of commits to main rather than wall-clock time.
 
 on:
   workflow_dispatch:
@@ -20,22 +17,21 @@ concurrency:
 
 jobs:
   bare-metal:
-    # Inert until a maintainer sets the BENCH_BARE_METAL repository variable to
-    # 'true'. Without a RunsOn stack behind the label the job would sit queued
-    # until it timed out, once per commit.
+    # Inert until a maintainer sets BENCH_BARE_METAL to 'true': without a RunsOn stack behind the
+    # label the job would queue until it timed out, once per commit.
     if: ${{ vars.BENCH_BARE_METAL == 'true' }}
     uses: ./.github/workflows/bench.yml
     with:
-      # The `runner=bench` shape is defined in .github/runs-on.yml; run_id
-      # namespaces the runner so no other job can claim it.
+      # The `runner=bench` shape is defined in .github/runs-on.yml; run_id namespaces it.
       runner: runs-on=${{ github.run_id }}/runner=bench
-      # Names the hardware, not the provider: a different instance type is a
-      # different testbed, because its history cannot be compared to this one.
-      testbed: aws-c7i-4xlarge
       label: ci-bare-metal
-      # A dedicated node can afford the slow fixed models
-      bench-markers: ""
-      bench-rounds: "3"
+      # Three rungs of benches/LADDER.md on one node from one build. L1 is that document's own L1
+      # rows; L2a and L2b share one problem sized to ~25 GiB of the instance's 32 GiB and differ
+      # only in shape. See docs/content/docs/benchmarks.mdx for the tables and the 4x2 choice.
+      rungs: |
+        L1  1 1        3 | -k "(test_model_propagate and (hubbard or pauli)) or (test_random_gradient and heisenberg)" --hubbard-cutoff=10 --hubbard-lower-atol=4.2e-05 --pauli-cutoff=12 --pauli-lower-atol=1.22e-04 --num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=295000
+        L2a 1 per-rank 1 | -k "(test_random_build_graph or test_random_energy or test_random_gradient) and heisenberg" --num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=2500000
+        L2b 4 per-rank 1 | -k "(test_random_build_graph or test_random_energy or test_random_gradient) and heisenberg" --num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=2500000
     secrets: inherit
     permissions:
       contents: read
diff --git a/.github/workflows/bench_main.yml b/.github/workflows/bench_main.yml
deleted file mode 100644
index 1b4f203a..00000000
--- a/.github/workflows/bench_main.yml
+++ /dev/null
@@ -1,28 +0,0 @@
-name: Track benchmarks
-
-# Builds the historical baseline that Bencher's Perf page graphs and that pull
-# request runs are compared against. One data point per commit on main.
-# Alerts are recorded but do not fail the job for now.
-
-
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - main
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: false
-
-jobs:
-  github-hosted:
-    uses: ./.github/workflows/bench.yml
-    with:
-      runner: ubuntu-26.04
-      testbed: ubuntu-26.04
-      label: ci-linux
-    secrets: inherit
-    permissions:
-      contents: read
-      checks: write
diff --git a/.github/workflows/scripts/check_shape.py b/.github/workflows/scripts/check_shape.py
new file mode 100644
index 00000000..1a02b0e5
--- /dev/null
+++ b/.github/workflows/scripts/check_shape.py
@@ -0,0 +1,60 @@
+# Copyright 2026 Algorithmiq
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Fail a rung that did not run at the shape it was asked for.
+
+A knob that fails to reach the ranks does not fail the run: it measures one partition per
+rank at a plausible wall time. Neither does `partitions` above a rank's visible cores --
+that disables placement entirely, at 13.8x to 24.6x, warning only on C++ stderr, which
+pytest's capture eats.
+
+Usage: check_shape.py <results/<label>.json> <ranks> <partitions>
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+
+def main(argv: list[str]) -> str | None:
+    path, ranks, partitions = Path(argv[1]), int(argv[2]), argv[3]
+    meta = json.loads(path.read_text())["meta"]
+    pin = meta.get("pinning", {})
+    threads = pin.get("threads", 0)
+    placed = pin.get("single_cpu_threads_min", 0)
+
+    print(
+        f"  ranks={meta['ranks']} partitions_env={meta['partitions_env']} "
+        f"threads={meta['monoprop_threads']} pinned>={placed} of {threads} "
+        f"mask={pin.get('affinity_cpus_min')}..{pin.get('affinity_cpus_max')}"
+    )
+
+    if meta["ranks"] != ranks or meta["partitions_env"] != partitions:
+        return (
+            f"::error::{path.stem} ran at ranks={meta['ranks']} "
+            f"partitions_env={meta['partitions_env']}, asked for {ranks}x{partitions}"
+        )
+    # An all-zero summary means /proc was unreadable, not that nothing was pinned.
+    if int(partitions) > 1 and threads and not placed:
+        return (
+            f"::error::{path.stem} placed no thread on a CPU of its own. "
+            "partitions above the rank's visible cores returns an empty placement order."
+        )
+    return None
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))
diff --git a/.github/workflows/scripts/cpu_slug.py b/.github/workflows/scripts/cpu_slug.py
new file mode 100644
index 00000000..a540f8fe
--- /dev/null
+++ b/.github/workflows/scripts/cpu_slug.py
@@ -0,0 +1,47 @@
+# Copyright 2026 Algorithmiq
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Print this run's CPU as a Bencher testbed slug, e.g. `intel-xeon-platinum-8488c`.
+
+Naming a series after the silicon rather than a provider's instance label means the name
+changes exactly when the hardware does, which is the property Bencher needs. The brand comes
+from the cpuinfo block pytest-benchmark already writes into `time-<label>.json`.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+from pathlib import Path
+
+NOISE = re.compile(r"\((?:r|tm)\)|\bprocessor\b", re.IGNORECASE)
+SEPARATORS = re.compile(r"[^a-z0-9]+")
+
+
+def slugify(brand: str) -> str:
+    return SEPARATORS.sub("-", NOISE.sub(" ", brand).lower()).strip("-") or "unknown-cpu"
+
+
+def main() -> str | None:
+    paths = sorted(Path("benches/results").glob("time-*.json"))
+    if not paths:
+        return "::error::no timing artifact to read the CPU from"
+    brand = json.loads(paths[0].read_text())["machine_info"]["cpu"]["brand_raw"]
+    print(slugify(str(brand)))
+    return None
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/README.md b/README.md
index 0d14f744..0c6749eb 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,6 @@
 [![Documentation](https://github.com/Algorithmiq/monoprop/actions/workflows/docpages.yml/badge.svg)](https://docs.monoprop.algorithmiq.tech/)
 [![Test monoprop](https://github.com/Algorithmiq/monoprop/actions/workflows/test.yml/badge.svg)](https://github.com/Algorithmiq/monoprop/actions/workflows/test.yml)
 [![codecov](https://codecov.io/gh/Algorithmiq/monoprop/graph/badge.svg?token=1GgmPnNUxk)](https://codecov.io/gh/Algorithmiq/monoprop)
-[![Track benchmarks](https://github.com/Algorithmiq/monoprop/actions/workflows/bench_main.yml/badge.svg)](https://github.com/Algorithmiq/monoprop/actions/workflows/bench_main.yml)
 
 `monoprop` is a high-performance C++ library with Python bindings for **Majorana and
 Pauli propagation** — a backend for classically simulating and variationally
diff --git a/justfile b/justfile
index dc58779b..c6a4a71d 100644
--- a/justfile
+++ b/justfile
@@ -404,15 +404,10 @@ bench-smoke:
         -m "not slow" --num-generators 8 --num-modes 8 --cutoff 6 --obs-terms 16
     uv run --no-sync monoprop-bench-report "{{ bench_results }}"
 
-# Deliberately keeps the `bench` default sizes rather than inventing a second set
-# to keep in sync: they run in well under a minute yet leave the heavier
-# Schrödinger operations in the 50 ms - 1 s range, where runner noise does not
-# swamp the signal. Only the slow fixed models are dropped; they run nightly.
-# More rounds than a local run, because CI reports the mean.
-#
-# The marker expression and round count come from the environment so a caller can
-# pass values containing spaces, or an empty marker to select everything:
-#   monoprop_BENCH_MARKERS= monoprop_BENCH_ROUNDS=3 just bench-ci ci-bare-metal
+# Keeps the `bench` default sizes rather than a second set to keep in sync. The marker expression
+# and round count come from the environment so a caller can pass a value containing spaces, or an
+# empty marker to select everything:
+#   monoprop_BENCH_MARKERS= monoprop_BENCH_ROUNDS=3 just bench-ci ci-bare-metal-L1
 
 # Run the continuous-benchmarking profile tracked by Bencher.
 bench-ci LABEL *ARGS:
@@ -424,13 +419,143 @@ bench-ci LABEL *ARGS:
         -m "${monoprop_BENCH_MARKERS-not slow}" \
         --bench-rounds "${monoprop_BENCH_ROUNDS:-5}" "$@"
 
+# `bench-ci` on RANKS ranks. Needs an MPI build and `monoprop_PARTITIONS`, which also sizes the
+# pinning -- unpinned ranks vary by more than the effects being tracked. ARGS go to pytest:
+#   monoprop_PARTITIONS=2 monoprop_NUM_THREADS=2 just bench-ci-mpi ci-bare-metal-L2b 4 -k heisenberg
+
+# Run the continuous-benchmarking profile on RANKS ranks.
+bench-ci-mpi LABEL RANKS *ARGS:
+    uv run --no-sync python -c "import monoprop, sys; sys.exit(0 if monoprop.has_mpi else 'monoprop was built without MPI; run just bench-build-mpi first')"
+    @mkdir -p "{{ bench_results }}"
+    : "${monoprop_PARTITIONS:?bench-ci-mpi needs monoprop_PARTITIONS; it sizes --map-by slot:PE}"
+    label="$1"; ranks="$2"; shift 2; \
+    monoprop_BENCH_LABEL="$label" monoprop_BENCH_RESULTS="{{ bench_results }}" \
+        uv run --no-sync mpiexec -n "$ranks" \
+        --map-by "slot:PE=$monoprop_PARTITIONS" --bind-to core \
+        -x monoprop_BENCH_LABEL -x monoprop_BENCH_RESULTS \
+        -x monoprop_PARTITIONS -x monoprop_NUM_THREADS \
+        python -m pytest benches -o filterwarnings=default \
+        --benchmark-json="{{ bench_results }}/time-$label.json" \
+        -m "${monoprop_BENCH_MARKERS-not slow}" \
+        --bench-rounds "${monoprop_BENCH_ROUNDS:-5}" "$@"
+
 # Convert one LABEL's artifacts into Bencher Metric Format JSON on stdout, e.g.
-#   just bench-ci ci-linux && just bench-bmf ci-linux > bmf.json
+#   just bench-ci ci-bare-metal-L1 && just bench-bmf ci-bare-metal-L1 > bmf-L1.json
 
 # Emit LABEL's results as Bencher Metric Format JSON.
 bench-bmf LABEL:
     uv run --no-sync monoprop-bench-bmf "{{ bench_results }}" "$1"
 
+# Record the benchmark runner's physical core count for subsequent workflow steps.
+bench-ci-resolve-cores:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    uv run --no-sync python - <<'PY'
+    import os
+    import sys
+
+    import psutil
+
+    import monoprop
+
+    if not monoprop.has_mpi:
+        sys.exit("monoprop was built without MPI; every rung shares this binary")
+
+    # PHYSICAL cores: that is what the engine enumerates, and sizing from the logical
+    # count would cross the `partitions > visible cores` threshold on any SMT machine.
+    cores = psutil.cpu_count(logical=False)
+    if not cores:
+        sys.exit("cannot determine the physical core count on this runner")
+
+    print("variant ", monoprop.__variant__)
+    print("cores   ", cores, "physical,", psutil.cpu_count(logical=True), "logical")
+    print("affinity", len(os.sched_getaffinity(0)))
+    with open(os.environ["GITHUB_ENV"], "a") as env:
+        print(f"BENCH_CORES={cores}", file=env)
+    PY
+
+# Run every newline-delimited rung in RUNGS under the requested rank and partition shape.
+bench-ci-rungs LABEL:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    bench_label="$1"
+    echo "physical cores: $BENCH_CORES"
+    # fd 3, because mpiexec forwards its own stdin to rank 0 and would eat the rung list.
+    while IFS='|' read -r head args <&3; do
+      IFS=$' \t' read -r rung ranks partitions rounds <<<"$head"
+      case "${rung:-}" in '' | '#'*) continue ;; esac
+      # A missing field would otherwise fall through to a default and measure something else.
+      case "${rounds:-}" in '' | *[!0-9]*) echo "::error::rung $rung has no round count"; exit 1 ;; esac
+      if [[ "$partitions" == per-rank ]]; then
+        partitions=$((BENCH_CORES / ranks))
+      fi
+      # Oversubscription reports a plausible wall time and would silently corrupt the testbed.
+      if ((partitions < 1 || ranks * partitions > BENCH_CORES)); then
+        echo "::error::rung $rung wants ${ranks}x${partitions} on $BENCH_CORES physical cores"
+        exit 1
+      fi
+      # xargs honours quotes in `-k "a and b"` without evaluating the input.
+      argv=()
+      if [[ -n "${args//[[:space:]]/}" ]]; then
+        if ! parsed=$(printf '%s\n' "$args" | xargs printf '%s\n') || [[ -z "$parsed" ]]; then
+          echo "::error::rung $rung: could not parse its pytest args"
+          exit 1
+        fi
+        readarray -t argv <<< "$parsed"
+      fi
+      label="$bench_label-$rung"
+      echo "::group::$rung: ${ranks}x${partitions}, $rounds rounds -> $label"
+      export monoprop_PARTITIONS="$partitions" monoprop_NUM_THREADS="$partitions"
+      export monoprop_BENCH_ROUNDS="$rounds"
+      if ((ranks == 1)); then
+        just bench-ci "$label" "${argv[@]}"
+      else
+        just bench-ci-mpi "$label" "$ranks" "${argv[@]}"
+      fi
+      just bench-bmf "$label" > "bmf-$rung.json"
+      uv run --no-sync python .github/workflows/scripts/check_shape.py \
+        "benches/results/$label.json" "$ranks" "$partitions"
+      echo "::endgroup::"
+    done 3<<< "$RUNGS"
+
+# Upload every rung's BMF artifact to the silicon- and shape-specific Bencher testbed.
+bench-ci-track:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    shopt -s nullglob
+    files=(bmf-*.json)
+    if ((${#files[@]} == 0)); then
+      echo "::error::no rung produced a bmf-*.json"
+      exit 1
+    fi
+    slug=$(uv run --no-sync python .github/workflows/scripts/cpu_slug.py)
+    case "$slug" in '' | *[!a-z0-9-]*) echo "::error::bad cpu slug: '$slug'"; exit 1 ;; esac
+    echo "cpu slug: $slug"
+    for file in "${files[@]}"; do
+      rung="${file#bmf-}"
+      rung="${rung%.json}"
+      BENCHER_TESTBED="$slug-${BENCH_CORES}c-$rung" bencher run \
+        --adapter json \
+        --file "$file" \
+        --github-actions "$GITHUB_TOKEN" \
+        --thresholds-reset \
+        --threshold-measure latency \
+        --threshold-test t_test \
+        --threshold-max-sample-size 64 \
+        --threshold-lower-boundary _ \
+        --threshold-upper-boundary 0.99 \
+        --threshold-measure peak-memory \
+        --threshold-test percentage \
+        --threshold-max-sample-size 64 \
+        --threshold-lower-boundary _ \
+        --threshold-upper-boundary 0.10 \
+        --threshold-measure terms \
+        --threshold-test percentage \
+        --threshold-max-sample-size 1 \
+        --threshold-lower-boundary 0.0 \
+        --threshold-upper-boundary 0.0
+    done
+
 # Execute the tutorial notebooks and convert them to Markdown. Notebook
 
 # execution fails the build on any cell error -- this is the notebook doctest.
diff --git a/pyproject.toml b/pyproject.toml
index 018ce2f3..0f4861a0 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -236,6 +236,8 @@ exclude = [
   "packages/monoprop-bench-tools/src/monoprop_bench_tools/_version.py",
   "docs/conf.py",
   "performance_scripts/*.py",
+  # CI glue, not a package: prints to the workflow log and lives outside any module tree.
+  ".github/workflows/scripts/*.py",
 ]
 
 target-version = "py311"

From 58e7939f1c1bb3b1bc3770bdc36229ac2dfb6002 Mon Sep 17 00:00:00 2001
From: Aaron Miller <aaron.miller@algorithmiq.fi>
Date: Tue, 1 Sep 2026 17:57:51 +0100
Subject: [PATCH 2/5] =?UTF-8?q?docs(benches):=20=F0=9F=93=9D=20tabulate=20?=
 =?UTF-8?q?the=20rungs=20CI=20measures?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The continuous-benchmarking section described the lanes in prose and quoted no sizes,
because none had been measured. It now carries three tables -- the shapes, L1's rows and
the L2 rows -- and the prose that justified each decision is gone.

`~s` and `~GiB` are measured on an 8-core mask of a Deucalion x86 node (znver2, SMT off),
not on the runner. Term counts are deterministic at `--seed=0` and transfer exactly;
memory transfers well; the times are indicative for Sapphire Rapids.

The `bench_main.yml` references follow the file.

Assisted-by: ClaudeCode:claude-opus-5
---
 docs/content/docs/benchmarks.mdx        | 122 ++++++++++++++++++------
 packages/monoprop-bench-tools/README.md |   2 +-
 2 files changed, 94 insertions(+), 30 deletions(-)

diff --git a/docs/content/docs/benchmarks.mdx b/docs/content/docs/benchmarks.mdx
index d72f679b..d55c31ef 100644
--- a/docs/content/docs/benchmarks.mdx
+++ b/docs/content/docs/benchmarks.mdx
@@ -314,13 +314,18 @@ appears at one rung and not the one below it names its own cause.
 
 ### Continuous benchmarking
 
-The `bench_main.yml` workflow benchmarks every commit on `main`. [Bencher](https://bencher.dev/)
-records the time series and reports threshold violations. The workflow invokes the same recipes
-available locally:
+The `bench_bare_metal.yml` workflow benchmarks every commit on `main` on a dedicated
+[RunsOn](https://runs-on.com/) EC2 node, one per run. [Bencher](https://bencher.dev/) records the
+time series and reports threshold violations. The workflow invokes the same recipes available
+locally:
 
 ```bash
-just bench-ci ci-linux          # default sizes, more rounds, no slow models
-just bench-bmf ci-linux         # the same results as Bencher Metric Format JSON
+export monoprop_PARTITIONS=1 monoprop_NUM_THREADS=1     # the shape, declared
+just bench-ci ci-bare-metal-L1       # 1 rank, 1 partition
+just bench-bmf ci-bare-metal-L1      # the same results as Bencher Metric Format JSON
+
+export monoprop_PARTITIONS=2 monoprop_NUM_THREADS=2     # `bench-ci` under mpiexec; P sizes the pinning
+just bench-ci-mpi ci-bare-metal-L2b 4 -k heisenberg
 ```
 
 `monoprop-bench-bmf` combines both artifacts because each Bencher upload accepts one adapter. The
@@ -335,27 +340,86 @@ measures are recorded:
 
 For a fixed seed and problem size, `terms` is deterministic; any change therefore indicates an
 algorithmic change. It is tracked for that reason alone: it is the only check that a timing win is
-not an accuracy change. Timing alerts from shared runners are recorded but do not fail the build.
-
-#### Testbeds
-
-Bencher identifies each series by `(branch, testbed, measure)`. A testbed therefore denotes one
-fixed machine configuration; changing that configuration would introduce a discontinuity in every
-series.
-
-| Workflow | Runner | Testbed | Label | Profile |
-|---|---|---|---|---|
-| `bench_main.yml` | GitHub-hosted `ubuntu-26.04` | `ubuntu-26.04` | `ci-linux` | `not slow`, 5 rounds |
-| `bench_bare_metal.yml` | [RunsOn](https://runs-on.com/) EC2 node | `aws-c7i-4xlarge` | `ci-bare-metal` | everything, 3 rounds |
-
-The bare-metal workflow provisions a dedicated node for each run. Its lower measurement variance
-supports tighter timing thresholds and permits execution of the fixed models omitted on the shared
-runner.
-
-The `bench-markers` and `bench-rounds` inputs of the reusable `bench.yml` workflow select the
-benchmarks and repetition count. These values should remain fixed after establishing a testbed,
-because changing them changes the meaning of its time series:
-
-```bash
-monoprop_BENCH_MARKERS= monoprop_BENCH_ROUNDS=3 just bench-ci ci-bare-metal
-```
+not an accuracy change. Timing alerts are recorded but do not fail the build.
+
+#### The rungs
+
+Bencher keys history on `(branch, testbed, benchmark, measure)`, so a testbed denotes one machine
+*and* one shape and the name carries both: `<cpu>-<cores>c-<rung>`, where `<cpu>` is `brand_raw`
+slugified and `<cores>` the physical core count read off the machine. On the pinned `c7i.4xlarge` —
+16 vCPU / **8 physical cores**, 32 GiB nominal (~30 usable), no swap — that gives
+`intel-xeon-platinum-8488c-8c-L1` and `-L2a`, `-L2b`. All three rungs run on one node from one
+MPI-enabled build, so the edges between them move the shape and nothing else.
+
+| rung | `R` × `P` | launch | `monoprop_PARTITIONS` | rounds |
+|---|---|---|---:|---:|
+| L1 | 1 × 1 | `just bench-ci` | 1 | 3 |
+| L2a | 1 × 8 | `just bench-ci` | 8 | 1 |
+| L2b | 4 × 2 | `mpiexec -n 4 --map-by slot:PE=2 --bind-to core` | 2 | 1 |
+
+`R × P` = 8 at L2a and L2b. L2b is 4 × 2 because the per-rank cost is paid `R` times on one node,
+which makes it the memory-worst rung and so the one the size ceiling is set against; `P` = 2 keeps
+more than one partition inside each rank, which 8 × 1 would not.
+
+Rounds is 1 above L1 because `pedantic` builds round *k+1*'s propagator before releasing round
+*k*'s, so a second round holds two — a memory setting here, not a statistics setting.
+
+#### L1 — LADDER.md's own rows, one thread
+
+| row | flags | `-k` | terms | ~s | ~GiB |
+|---|---|---|---:|---:|---:|
+| hubbard `propagate` | `--hubbard-cutoff=10 --hubbard-lower-atol=4.2e-05` | `test_model_propagate and hubbard` | 9,953,109 | 25 | 0.87 |
+| pauli `propagate` | `--pauli-cutoff=12 --pauli-lower-atol=1.22e-04` | `test_model_propagate and pauli` | 10,069,308 | 21 | 1.06 |
+| random `gradient` | `--num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=295000` | `test_random_gradient and heisenberg` | 19,902,244 | 11 | 2.60 |
+
+One process, one selector: the three flag sets use disjoint options, so they union. Whole-rung peak
+2.60 GiB in 3:53 at three rounds. LADDER.md's fourth L1 row — the same gradient pared at `1e-10` —
+is not here, because `--pare-threshold` is session-wide and cannot share a process with the unpared
+row.
+
+#### L2a and L2b — one operator, three operations, ~25 GiB
+
+`build_graph` publishes the graph `energy` and `gradient` evaluate, so the three share one operator
+and the peak is the max over them. `propagate` is excluded: it holds its own. Both rungs take the
+same flags, so the term counts must match — that is the geometry-independence check, and the
+`terms` measure holds it to 0%.
+
+`-k "(test_random_build_graph or test_random_energy or test_random_gradient) and heisenberg"`
+`--num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=2500000`
+
+| row | terms | ~s | ~GiB L2a | ~s | ~GiB/node L2b | ~GiB worst rank |
+|---|---:|---:|---:|---:|---:|---:|
+| random `build_graph` | 167,515,463 | 83 | 19.11 | 68 | 23.71 | 6.14 |
+| random `energy` | 167,515,463 | 13 | 17.54 | 6 | 20.36 | 5.17 |
+| random `gradient` | 167,515,463 | 39 | 18.85 | 16 | 21.68 | 5.50 |
+
+Peak 19.11 GiB at L2a and 23.71 GiB/node at L2b, in 135 s and 90 s. `~GiB/node` is the sum over
+the node's ranks — it errs high, because a page shared between ranks is charged to each, so it is
+the number to provision against and the wrong number to divide.
+
+`--obs-terms=2500000` is **the largest measured point that fits**, not an interpolation: the next
+step tried, 3.5M, reaches 35.49 GiB/node, past the 32 GiB the instance has. A two-point fit over
+1.5M and 2.5M predicted 31.1 GiB there and under-called it by 3.9, so this axis is read off
+measured points rather than interpolated.
+
+The term count is identical at both shapes, which is the geometry-independence check. L2b's node
+sum is 1.24x L2a's peak at this size — the per-rank cost paid four times on one node — against
+1.29x at 1.5M, so the multiplier falls as the operator grows.
+
+#### Inputs
+
+`rungs` is one rung per line, `<name> <ranks> <partitions> <rounds> | <pytest args>`, where
+`partitions` is an integer or `per-rank` for one per physical core of the rank's share. A shape
+that does not fit is refused rather than oversubscribed, and every rung reads its `ranks`,
+`partitions_env` and thread placement back out of its result JSON before the next one starts. Once
+a testbed has history these values should stay fixed: a new shape is a new rung, so a new testbed.
+
+**No benchmark workflow gates a pull request.** It tracks `main`, and a threshold is a report, not
+a verdict — measuring a change is the author's job, per "Benchmarking ladder" above. Every run
+uploads each rung's `bmf-<rung>.json` and raw result JSON as an artifact, so a failed Bencher
+upload does not take the measurement with it.
+
+The workflow is inert until a maintainer sets the `BENCH_BARE_METAL` repository variable to
+`true`; without a RunsOn stack behind the label the job would queue until it timed out, once per
+commit. There is no shared-runner series: a GitHub-hosted runner has about two physical cores, so
+L2a would be 1 × 2 and L2b 2 × 1 — restatements of L1 through a noise floor no threshold survives.
diff --git a/packages/monoprop-bench-tools/README.md b/packages/monoprop-bench-tools/README.md
index a3a5da0a..2992cb55 100644
--- a/packages/monoprop-bench-tools/README.md
+++ b/packages/monoprop-bench-tools/README.md
@@ -55,7 +55,7 @@ Render the artifacts of a run:
 
 ```bash
 monoprop-bench-report benches/results             # writes REPORT.md
-monoprop-bench-bmf benches/results ci-linux       # BMF JSON on stdout
+monoprop-bench-bmf benches/results ci-bare-metal-L1   # BMF JSON on stdout
 ```
 
 Both read the two files a run leaves in the results directory:

From 2f06d390f45697253f0d1b50d24b107cf3bd0ccc Mon Sep 17 00:00:00 2001
From: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com>
Date: Thu, 3 Sep 2026 12:28:20 +0100
Subject: [PATCH 3/5] Update benchmarks.mdx
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com>
Signed-off-by: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com>
---
 docs/content/docs/benchmarks.mdx | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/docs/content/docs/benchmarks.mdx b/docs/content/docs/benchmarks.mdx
index d55c31ef..0c016596 100644
--- a/docs/content/docs/benchmarks.mdx
+++ b/docs/content/docs/benchmarks.mdx
@@ -418,8 +418,3 @@ a testbed has history these values should stay fixed: a new shape is a new rung,
 a verdict — measuring a change is the author's job, per "Benchmarking ladder" above. Every run
 uploads each rung's `bmf-<rung>.json` and raw result JSON as an artifact, so a failed Bencher
 upload does not take the measurement with it.
-
-The workflow is inert until a maintainer sets the `BENCH_BARE_METAL` repository variable to
-`true`; without a RunsOn stack behind the label the job would queue until it timed out, once per
-commit. There is no shared-runner series: a GitHub-hosted runner has about two physical cores, so
-L2a would be 1 × 2 and L2b 2 × 1 — restatements of L1 through a noise floor no threshold survives.

From 592bc5afc759841d4557d41b36269f05ed91efd5 Mon Sep 17 00:00:00 2001
From: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com>
Date: Thu, 3 Sep 2026 12:28:33 +0100
Subject: [PATCH 4/5] Update pyproject.toml
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com>
Signed-off-by: Aaron Miller <61472721+diagonal-hamiltonian@users.noreply.github.com>
---
 pyproject.toml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/pyproject.toml b/pyproject.toml
index 0f4861a0..5ca96222 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -236,7 +236,6 @@ exclude = [
   "packages/monoprop-bench-tools/src/monoprop_bench_tools/_version.py",
   "docs/conf.py",
   "performance_scripts/*.py",
-  # CI glue, not a package: prints to the workflow log and lives outside any module tree.
   ".github/workflows/scripts/*.py",
 ]
 

From ec86a2bb388cb7bb16c147e0d17e5e5bd95f1c11 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Roberto=20Di=20Remigio=20Eik=C3=A5s?=
 <robertodr@users.noreply.github.com>
Date: Thu, 3 Sep 2026 13:34:10 +0200
Subject: [PATCH 5/5] Apply batched suggestions from code review
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com>
Signed-off-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com>
---
 .github/workflows/bench.yml            | 4 ----
 .github/workflows/bench_bare_metal.yml | 6 ------
 docs/content/docs/benchmarks.mdx       | 5 +----
 3 files changed, 1 insertion(+), 14 deletions(-)

diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml
index 0babd636..3aa71c4a 100644
--- a/.github/workflows/bench.yml
+++ b/.github/workflows/bench.yml
@@ -1,8 +1,6 @@
 name: Benchmark
 
 # Shared body of the Bencher runs; the caller names the machine and the benches/LADDER.md rungs.
-# Every rung runs on the same runner from the same MPI-enabled build and uploads to its own
-# testbed, because Bencher keys history on (branch, testbed, benchmark, measure).
 
 on:
   workflow_call:
@@ -46,8 +44,6 @@ jobs:
         with:
           cache-suffix: benchmark
 
-      # uv's `cache-keys` cover neither `SKBUILD_*` nor `monoprop_ENABLE_MPI`, so without the
-      # reinstall the warm cache above can hand back an MPI-off wheel and every rung still pass.
       - name: Install package
         env:
           SKBUILD_CMAKE_DEFINE: monoprop_ENABLE_CXX_UNIT_TESTS=OFF
diff --git a/.github/workflows/bench_bare_metal.yml b/.github/workflows/bench_bare_metal.yml
index aacd7d02..57ef376a 100644
--- a/.github/workflows/bench_bare_metal.yml
+++ b/.github/workflows/bench_bare_metal.yml
@@ -1,9 +1,6 @@
 name: Track benchmarks (bare metal)
 
 # monoprop's continuous benchmarking, on a dedicated EC2 node provisioned per run by RunsOn.
-# The point is the noise floor: nothing else is scheduled on the machine, so these testbeds can
-# carry thresholds tight enough to gate on. The node exists only for the run, so cost tracks the
-# number of commits to main rather than wall-clock time.
 
 on:
   workflow_dispatch:
@@ -25,9 +22,6 @@ jobs:
       # The `runner=bench` shape is defined in .github/runs-on.yml; run_id namespaces it.
       runner: runs-on=${{ github.run_id }}/runner=bench
       label: ci-bare-metal
-      # Three rungs of benches/LADDER.md on one node from one build. L1 is that document's own L1
-      # rows; L2a and L2b share one problem sized to ~25 GiB of the instance's 32 GiB and differ
-      # only in shape. See docs/content/docs/benchmarks.mdx for the tables and the 4x2 choice.
       rungs: |
         L1  1 1        3 | -k "(test_model_propagate and (hubbard or pauli)) or (test_random_gradient and heisenberg)" --hubbard-cutoff=10 --hubbard-lower-atol=4.2e-05 --pauli-cutoff=12 --pauli-lower-atol=1.22e-04 --num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=295000
         L2a 1 per-rank 1 | -k "(test_random_build_graph or test_random_energy or test_random_gradient) and heisenberg" --num-generators=1000 --num-modes=142 --cutoff=6 --obs-terms=2500000
diff --git a/docs/content/docs/benchmarks.mdx b/docs/content/docs/benchmarks.mdx
index 0c016596..5a2a4fdd 100644
--- a/docs/content/docs/benchmarks.mdx
+++ b/docs/content/docs/benchmarks.mdx
@@ -397,10 +397,7 @@ Peak 19.11 GiB at L2a and 23.71 GiB/node at L2b, in 135 s and 90 s. `~GiB/node`
 the node's ranks — it errs high, because a page shared between ranks is charged to each, so it is
 the number to provision against and the wrong number to divide.
 
-`--obs-terms=2500000` is **the largest measured point that fits**, not an interpolation: the next
-step tried, 3.5M, reaches 35.49 GiB/node, past the 32 GiB the instance has. A two-point fit over
-1.5M and 2.5M predicted 31.1 GiB there and under-called it by 3.9, so this axis is read off
-measured points rather than interpolated.
+`--obs-terms=2500000` is **the largest measured point that fits**.
 
 The term count is identical at both shapes, which is the geometry-independence check. L2b's node
 sum is 1.24x L2a's peak at this size — the per-rank cost paid four times on one node — against