diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index cd7599c0..befcac60 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -78,6 +78,16 @@ jobs: run: | just bench-bmf "$BENCHER_LABEL" > bmf.json + - name: Upload the raw measurement + if: always() + uses: actions/upload-artifact@v7 + with: + name: bench-${{ inputs.label }} + path: | + bmf.json + benches/results/*.json + if-no-files-found: warn + - uses: bencherdev/bencher@v0.6.12 # The project slug and API key go through the environment rather than the @@ -105,11 +115,6 @@ jobs: --threshold-max-sample-size 64 \ --threshold-lower-boundary _ \ --threshold-upper-boundary 0.10 \ - --threshold-measure resting-memory \ - --threshold-test percentage \ - --threshold-max-sample-size 64 \ - --threshold-lower-boundary _ \ - --threshold-upper-boundary 0.10 \ --threshold-measure terms \ --threshold-test percentage \ --threshold-max-sample-size 1 \ diff --git a/AGENTS.md b/AGENTS.md index cf2f2fd0..2b16fc55 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,16 +48,19 @@ The root `pyproject.toml` declares a `[tool.uv.workspace]`, so one `uv.lock` cov - `monoprop` (repository root, `src/monoprop/`) — the library, built by scikit-build-core. - `packages/monoprop-bench-tools/` (`monoprop_bench_tools`) — the reusable half of the benchmark harness, published to PyPI: `memory.cpu` / `memory.gpu` (peak-footprint measurement), `models` - (the benchmarked problem builders), `report` and `bmf` (the artifact renderers). Pure Python, - built by hatchling, versioned off the same git tags as `monoprop`. It ships the console scripts - `monoprop-bench-report` and `monoprop-bench-bmf`. It contains **no benchmarks**. + (the benchmarked problem builders), `report` and `bmf` (the artifact renderers), and `rungs` + (the rung-table loader, runner and ladder collator). Pure Python, built by hatchling, versioned + off the same git tags as `monoprop`. It ships the console scripts `monoprop-bench-report`, + `monoprop-bench-bmf`, `monoprop-bench-rung` and `monoprop-bench-ladder`. It contains **no + benchmarks and no rung table** — those are data whose names campaigns already depend on. - `packages/bench-third-party/` — cross-engine comparison scripts. Listed in the workspace `exclude`: it pins a narrower `requires-python`, a git dependency and linux-x86_64-only CUDA wheels, so it is a standalone uv project with its own `uv.lock`. Run it with `cd packages/bench-third-party && uv sync`, never from the root environment. -- `benches/` — monoprop's own benchmark suite (`conftest.py`, `bench_*.py`, `results/`). It stays in - the repository and imports the tools package. Benchmark names are Bencher's history key, so they - must not move with a library release; that is why the suite is not in `monoprop-bench-tools`. +- `benches/` — monoprop's own benchmark suite (`conftest.py`, `bench_*.py`, `rungs.toml`, + `RUNGS.md`, `results/`). It stays in the repository and imports the tools package. Benchmark + names are Bencher's history key and rung ids name artifacts already written, so neither may move + with a library release; that is why the suite is not in `monoprop-bench-tools`. Dependency groups follow from that split: `test` is monoprop's own suite only (cibuildwheel installs it against a built wheel, so it must not reference a workspace member), `workspace-test` adds @@ -83,7 +86,14 @@ Key files: continuous-benchmarking workflow. Both read the schema written by `benches/conftest.py`, so a change to the recorded sections has to land on both sides of the package boundary. Benchmark names are Bencher's history key, so renaming or moving a `bench_*` test orphans its tracked - series. + series. Both bench modules measure the same four operations — `build_graph`, `propagate`, + `energy`, `gradient`. +- `benches/rungs.toml` and `monoprop_bench_tools.rungs`: the benchmark set at the sizes the + library is used at, as a table rather than a loop, so a campaign cannot quietly run a different + grid. Every row carries `expect_terms`, and a result missing it by >0.1% is refused: at a fixed + seed and tolerance the term count is reproducible to the digit, so the gate catches a mistyped + knob rather than tolerating noise. A row with `expect_terms = 0` has never been calibrated and + refuses to run. The scheduler that submits a rung is deliberately not in this repository. - **`rounds > 1` overlaps two rounds' live memory** (`setup=` runs before the prior round's teardown) — pin `--bench-rounds=1`; `record_memory` measures that construction transient, not per-op cost (use `op_memory`). diff --git a/benches/README.md b/benches/README.md index 713a33d4..807f614b 100644 --- a/benches/README.md +++ b/benches/README.md @@ -11,20 +11,58 @@ documentation for detailed instructions. ## What lives where This directory holds only monoprop's own benchmarks — `conftest.py` (the fixtures -and the results schema), `bench_random.py`, `bench_models.py`, and `results/`. +and the results schema), `bench_random.py`, `bench_models.py`, `rungs.toml` (the +benchmark set), [`RUNGS.md`](RUNGS.md) (how to run it) and `results/`. + +Both bench modules measure the same four operations — `build_graph`, `propagate`, +`energy` and `gradient` — so a number means the same thing whichever problem +produced it. Benchmark names are the key [Bencher](https://bencher.dev/) stores history under, so they stay here rather than moving with a library release. Everything reusable is in the `monoprop-bench-tools` package ([`../packages/monoprop-bench-tools`](../packages/monoprop-bench-tools)): the -memory instrumentation, the model builders, and the two renderers that turn a -run's artifacts into `REPORT.md` and Bencher Metric Format JSON. +memory instrumentation, the model builders, the two renderers that turn a run's +artifacts into `REPORT.md` and Bencher Metric Format JSON, and the runner that +executes one row of `rungs.toml` and gates the result on its term count. Cross-engine comparisons against other propagation libraries live in [`../packages/bench-third-party`](../packages/bench-third-party), a standalone uv project with its own lockfile. +## The rung ladder + +`rungs.toml` is the benchmark set: one row per cell, giving the picture, the model, +the operations, the geometry, the size knobs and the exact term count that +configuration produces. A row, not a loop, so a campaign cannot quietly run a +different grid and two campaigns' numbers are comparable row for row. + +```bash +monoprop-bench-rung benches/rungs.toml list # the set, and what each costs +monoprop-bench-rung benches/rungs.toml --dry-run # the plan, no allocation spent +monoprop-bench-rung benches/rungs.toml --rep 1 # one rep +monoprop-bench-ladder benches/rungs.toml benches/results # the block to paste into the PR +``` + +Nothing runs these for you. Run the rungs your change could plausibly move and put +the block in the pull request: it carries the timings, the peak memory, and the +resolved parameters of every problem measured. + +`expect_terms` is a gate, not documentation: a result missing it by more than 0.1% +is refused, so a mistyped tolerance fails the cell instead of measuring a different +problem under the right name. A row nobody has calibrated says so twice -- +`expect_terms = 0` and `TBD` on the unmeasured knob -- and refuses to run. + +`cost_seconds` and `cost_gib_per_node` are the opposite: what one rep last cost, so +you can see what a rung takes before you spend it. Documentation, never a gate. +The machine and scheduler that run a rung are not in this repository; the table, +the runner and the gate are. + +**[`RUNGS.md`](RUNGS.md)** is the guide: every model's parameters and what they do, +how the geometry maps onto a machine, example Slurm scripts, the pinning and +allocation-sizing traps, and how to calibrate a `TBD` row. + ## Fixed-model sizing At the default `--hubbard-lower-atol 1e-4`, both nominal Hubbard size axes are diff --git a/benches/RUNGS.md b/benches/RUNGS.md new file mode 100644 index 00000000..5caa2c7c --- /dev/null +++ b/benches/RUNGS.md @@ -0,0 +1,590 @@ +# Running the rungs + +`rungs.toml` is monoprop's benchmark set. This page is how to run it: what each model's +parameters mean, how the geometry maps onto a machine, and example Slurm scripts. + +Nothing here runs automatically. No CI workflow gates a pull request on these. Running the +rungs your change could plausibly move, and putting the numbers in the pull request, is your +job — and `monoprop-bench-ladder` prints the block to paste. + +## The three commands + +```bash +# What the set contains, and what one rep of each last cost. +monoprop-bench-rung benches/rungs.toml list + +# The exact pytest line, geometry and environment, without spending anything. +monoprop-bench-rung benches/rungs.toml n1-100m-hubbard-propagate --dry-run + +# One rep. Repeat with --rep 2, 3, ... for the rest. +monoprop-bench-rung benches/rungs.toml n1-100m-hubbard-propagate --rep 1 --results runs + +# Collate every gate-clean rep into the block to paste into the pull request. +monoprop-bench-ladder benches/rungs.toml runs +``` + +`list` marks an uncalibrated row with `*` and shows `?` where nobody has timed it: + +``` + n1-100m-hubbard-propagate size N=1 P=128 96,981,051 ? + weak-97m-n4 weak N=4 P=512 377,482,074 13.1s 11.5GiB/node +* st-10m-random-propagate size N=1 P=1 0 ? +``` + +## What a row declares + +| field | meaning | +| --- | --- | +| `id` | unique; names the artifacts, so it is stable once a rung has been run | +| `family` | `size`, `strong` or `weak` | +| `picture` | `heisenberg` or `schrodinger`; only the random model has this axis | +| `model` | `random`, `hubbard` or `pauli` | +| `ops` | any of `build_graph`, `propagate`, `energy`, `gradient` | +| `nodes`, `ranks_per_node`, `partitions` | the geometry; see below | +| `args` | `--