chore(benches): make the benchmark set a fixed, runnable table - #310
Draft
diagonal-hamiltonian wants to merge 6 commits into
Draft
chore(benches): make the benchmark set a fixed, runnable table#310diagonal-hamiltonian wants to merge 6 commits into
diagonal-hamiltonian wants to merge 6 commits into
Conversation
diagonal-hamiltonian
requested review from
fpietra,
ludmilaasb and
robertodr
as code owners
August 29, 2026 18:44
github-actions
Bot
added
documentation
|
Docs preview: https://pr-310.monoprop-docs.pages.dev |
diagonal-hamiltonian
force-pushed
the
chore/bench-rungs
branch
from
August 29, 2026 18:49
add8282 to
79f62a4
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #310 +/- ##
=======================================
Coverage 97.70% 97.70%
=======================================
Files 14 14
Lines 742 742
Branches 98 98
=======================================
Hits 725 725
Misses 12 12
Partials 5 5
Flags with carried forward coverage won't be shown. Click here to find out more. |
`benches/` carried three benchmarks nobody asked for and four measures where three are read, while the benchmark set that actually matters -- the 10M/100M/1B ladder and the strong/weak scaling rungs -- existed only as flags in a private shell script, so there was no fixed set a PR author could run and report against. Trim to four operations, uniform across both bench modules, both pictures and both fixed models: `build_graph`, `propagate`, `energy`, `gradient`. Drop `test_random_pare`, `test_random_inplace` (which is `propagate` followed by `energy`) and `test_model` (`test_model_propagate` plus a trailing `expectation_value`). These are deletions, never renames: a benchmark name is Bencher's history key. Drop `resting-memory` as a tracked measure, leaving latency, peak memory and an exact-match term count. The raw `memrest`/`membase` recordings stay in `conftest.py` -- unrendered here, but read by the out-of-tree A/B harness. Add `benches/rungs.toml`: one row per cell of the benchmark set, giving the picture, model, operations, geometry, size knobs, the exact term count that configuration produces, and what one rep last cost in wall seconds and GiB per node. `monoprop-bench-rung` runs one rep; `monoprop-bench-ladder` collates them into a markdown block -- the timings, the peak memory, and the resolved parameters of every problem measured -- to paste into a pull request. Nothing runs automatically and nothing gates. Running the rungs a change could plausibly move, and putting the numbers in the PR, is the author's job. The runner refuses a result whose term count, node count, rank count, partition count or round count disagrees with the row, so a mistyped knob fails the cell instead of quietly measuring a different problem. A row nobody has calibrated says so twice -- `expect_terms = 0` and `TBD` on the unmeasured knob -- and refuses to run; `TBD` rather than `0` because `lower_atol = 0` prunes nothing and is the largest problem the model can pose. `cost_seconds` and `cost_gib_per_node` are the opposite: documentation, never a gate. `propagate` carries the full ladder to a billion terms; `build_graph`, `energy` and `gradient` stop at one node, because `build_graph` extends the graph rather than replacing it and retains one layer-set per gate. `bench.yml` loses the `resting-memory` threshold and gains an artifact upload, so a failed Bencher upload no longer loses the measurement. No other CI change. Also fixes a pre-existing bug in `report.py`, where every `test_model_*[hubbard]` and `test_model_*[pauli]` row rendered under the same label. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
diagonal-hamiltonian
force-pushed
the
chore/bench-rungs
branch
from
August 29, 2026 19:02
79f62a4 to
cbb2821
Compare
diagonal-hamiltonian
changed the title
`rungs.toml` says what the benchmark set is; nothing said how to run it. A reader had the row's `args` but no way to learn what `--hubbard-observable-site` means or what a sensible `srun` line looks like, and the parameters were only discoverable by reading the model dataclasses. `benches/RUNGS.md` covers: the three commands; what each field of a row declares, and which of them gate and which only document; every parameter of all three models with its default and its effect; why `lower_atol` is the size knob and the other axes are saturated; how nodes, ranks and partitions map onto a machine and why P is the number that matters; example Slurm scripts for one node, many nodes and a ladder in one allocation; the pinning and allocation-sizing traps measured on Deucalion; and how to calibrate a TBD row off the gate's own refusal message. Every shell example is syntax-checked and every documented invocation was run. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
…pped rung The graph and eval rows stopped at a single `n1-cap-*` rung per picture, so the 100M, 1B and multinode-1B points the set is supposed to cover were absent for `build_graph`, `energy` and `gradient` — present only for `propagate`. That cap rested on an extrapolation from a 100-generator measurement (~1234 B per term for build_graph), applied to rows that run 1000 generators, where the retained layer count is an order of magnitude larger. It was a guess standing in for the measurement, and naming the rung `n1-cap` hid the question rather than answering it. Declare the graph and eval rows at the same 10M / 100M / 1B / multinode-1B points as propagate, in both pictures: 4 cap rows become 12, and the size family goes from 20 rungs to 28 (12 propagate + 16 graph/eval). Calibration now answers which of them fit. A row that does not fit is a result: record the node count it needed, or that it exceeded the machine, in its `note` and leave it uncalibrated. Also document two system-size ceilings that the option list does not show. `monoprop_MAX_NUM_MODES` defaults to 250, so `--num-modes` is capped at 250 and `--hubbard-num-sites` at 125. `--pauli-num-qubits` is effectively fixed at 127: the circuit is built over a hard-coded IBM Eagle heavy-hex map of 144 pairs whose highest index is 126, so a lower value fails and a higher one adds idle qubits. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
RUNGS.md showed one rung and one ladder within an allocation, but not how to run a family end to end: a family needs one allocation per node count, and nothing said where that job list comes from. Add a `Running everything` section that generates the job list from the table itself — the table is the only place that knows which rungs share a node count — then submits one `sbatch` per line and collates once over every job's artifacts. The generator's output is reproduced verbatim from a real run of it. Add a `What you get out` section showing the collated `strong` table and the `Problems measured` block. The medians are the shipped campaign's own, replayed through the collator, so `vs declared` reads `1.000x` by construction; that is stated inline so nobody reads it as a fresh measurement. The point is the shape, and the shape is the answer: `Mterms/s/node` goes 7.28 at one node, 6.86 at eight, 1.02 at sixty-four, so the ladder turns over well before 64 nodes. Every shell block is syntax-checked and the job-list generator was run. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
RUNGS.md carried the memory law but no way to answer "what will this cost me": the map from the size knob to terms, memory and time existed only inside the rung rows. Add the measured hubbard/propagate table -- eleven `lower_atol` values against terms, GiB/node, median seconds and Mterms/s/node, from the campaign the table ships. Two things fall out: the knob is steep, roughly doubling terms per halving, and throughput is nearly flat at ~7 Mterms/s/node across three orders of magnitude, which makes it a usable planning number. Add planning rules for node count and wall time, each checked against the measured rows and the check shown: 1569 Mterms predicts 1.0 nodes at 100 GiB against 1 measured, 24420 predicts 8.1 at 200 GiB against 8, and the time rule lands within ~15%. Refit the memory law on the shipped costs: GiB/node = 3.40 + 0.0634 x Mterms/node, marginal 68.0 B/term. Note that the constant dominates below about 1000 Mterms/node, where apparent bytes-per-term reads ~180 rather than ~68 -- a per-term figure taken from a small rung overestimates a large one badly. Replace the build_graph assertion with a measurement. At identical parameters (cutoff 6, lower_atol 1e-4) propagate runs 29 Trotter steps in 0.38 GiB while build_graph was killed above 20 GiB at 2 steps: at least fiftyfold, on the same problem, rather than an extrapolation from a 100-generator estimate. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
…ze knob The sizing data covered hubbard/propagate only -- all 38 shipped rungs are that one configuration. Nothing said what pauli costs, what the random model costs, or what any model costs at its defaults, which is the first thing anyone runs. Sweep 63 cells: three models, four operations, both pictures, each at its defaults and along a size-knob ladder. 42 produced numbers; 21 were killed at the login node's 20 GiB per-user cap, which is recorded as `> 20 GiB` -- a lower bound and a fact about that node, not about the model. Findings worth the table space: - At the defaults only `propagate` fits in 20 GiB on hubbard; `build_graph`, `energy` and `gradient` all hold a graph and all died. - The random model is 159x larger in the Schrodinger picture than the Heisenberg one at identical knobs, `schrodinger_cutoff` being `cutoff + 2`. - Marginal cost per term, fitted over each ladder so the ~380 MiB process floor drops out: hubbard/propagate 63.2 B/term, pauli/propagate 79.4, pauli/build_graph 134.0, random 134-150 across all four operations. - Hubbard's 63.2 B/term here against 68.0 fitted over 38 compute-node rungs is the only independent check available on any of these numbers: two machines, two partition counts, two harnesses, 7% apart. - `--obs-terms` maps to propagated terms at a flat x33 over the range measured, so a random rung can be calibrated by starting at `target / 33` instead of guessing. That starting point now sits in each of the 20 random rows' `note`, with the extrapolation flagged: the ratio was measured up to 0.66M terms and the largest target is 1B. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
|
diagonal-hamiltonian
marked this pull request as draft
August 30, 2026 13:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
What this is
benches/carried three benchmarks nobody asked for and four tracked measures where three areread. More importantly, the benchmark set that actually matters — the 10M/100M/1B ladder and the
strong/weak scaling rungs — existed only as flags in a shell script on one cluster, so there was
no fixed set a PR author could run and no standard to report against.
This makes the benchmark set a table in the repository, with a runner, a gate on what it
measured, and a paste-ready block of results. Nothing runs automatically and nothing gates a
PR. Running the rungs a change could plausibly move, and putting the numbers in the PR, is the
author's job.
Four operations, everywhere
build_graph,propagate,energy,gradient— uniform across both bench modules, bothpictures and both fixed models. Deleted:
test_random_paretest_random_inplacepropagatefollowed byenergy, both already benchmarkedtest_modeltest_model_propagateplus a trailingexpectation_valueDeletions, never renames. A benchmark name is Bencher's history key; the surviving names are
byte-identical to
main's, so no series is orphaned.Three measures
Tracked:
latency,peak-memory,terms(exact match).resting-memoryis dropped and itsreport section removed. The raw
memrest/membase/opbytes/opmemdelta/opmempeakrecordings stay in
conftest.py— unrendered here, but read by the out-of-tree A/B harness.benches/results/README.mdnow says so.The benchmark set —
benches/rungs.tomlmonoprop-bench-ladderprints the timings, the peak memory, and the resolved parameters ofevery problem measured — read back from the run rather than from the row's overrides, so a
reviewer can see what was measured without resolving flags against a model's defaults:
Its actual output, from the tiny two-rep run used to check this end to end:
What each row declares
expect_termsis a gate: a result missing it by more than 0.1% is refused (renamed.refused.json, not deleted), as is one whose node count, rank count, partition count or timinground count disagrees with the row. A mistyped knob fails the cell instead of quietly measuring a
different problem.
--bench-rounds=1is forced, not defaulted:pedanticbuilds round k+1's arguments beforereleasing round k's, so a second round holds two propagators and doubles peak RSS. Repetition
is repeated process launches, which is what
--repnames.cost_secondsandcost_gib_per_nodeare the opposite of a gate — what one rep last cost, soyou can see what a rung takes before you spend it, and so a re-run can be shown against the last
one. Documentation only: a timing is noisy and your machine is not the machine they were taken on.
The 38 scaling rungs carry the measured medians of the 2026-08-27 Deucalion campaign, along with
its provenance and the fitted memory law
GiB/node = 3.39 + 0.0634 × Mterms/node. Folding theminto the table replaced the separate baseline JSON, so there is one file to keep in sync instead
of two.
A row nobody has calibrated says so twice —
expect_terms = 0andTBDon every unmeasuredsize knob — and refuses to run.
TBDrather than0because0is not neutral:lower_atol = 0prunes nothing and is the largest problem the model can pose, so a placeholder0would invite exactly the allocation-burning run the gate exists to prevent.Why the graph rows stop at one node
build_graphextends the graph rather than replacing it, so it retains one layer-set per gatewhile
propagatereleases each layer as it contracts. Measured: Hubbard reaches 97M terms throughpropagatein well under 2 GiB, while four successivebuild_graphcalls on the same modelexceed a 242 GiB node. So
propagatecarries the full 10M/100M/1B ladder and the graph-holdingoperations cap at a single node. Adding a multi-node graph rung later is one row.
Running a rung needs a machine, a scheduler and an allocation, none of which belong here. The
table, the runner and the gate are in the repo; the launcher that submits them is not.
How to run them —
benches/RUNGS.mdA new guide covering: what each field of a row declares and which of them gate; every parameter
of all three models with its default and its effect; why
lower_atolis the size knob and theother axes are saturated; how nodes, ranks and partitions map onto a machine and why
Pis thenumber that matters; example Slurm scripts for one node, many nodes and a ladder in one
allocation; the pinning and allocation-sizing traps (
--cpu-bind=nonemeasured a 1.45x penalty;MALLOC_ARENA_MAXset to the partition count cost ~16% of wall); and how to calibrate aTBDrowoff the gate's own refusal message.
Every shell example in it is syntax-checked and every documented invocation was run.
The 66 rungs, with their model parameters
The
sizefamily is the requested matrix exactly: 10M single-thread, 100M and 1B on one node,1B across eight — for
propagateon all three models, and forbuild_graphandenergy+gradienton the random model at 1000 generators in both pictures. 12 propagate rows +16 graph/eval rows.
Which of the graph rungs actually fit is a question calibration answers.
build_graphretains onelayer-set per gate, and at 1000 generators that is an order of magnitude more than the 100-generator
measurement anyone has. A row that does not fit records the node count it needed, or that it
exceeded the machine, and stays uncalibrated — that is a measurement of the graph path's cost, not
a gap.
System-size ceilings, which the option list does not show:
monoprop_MAX_NUM_MODESdefaults to250, so
--num-modescaps at 250 and--hubbard-num-sitesat 125.--pauli-num-qubitsiseffectively fixed at 127 — the circuit is built over a hard-coded IBM Eagle heavy-hex map of 144
pairs whose highest index is 126, so a lower value fails and a higher one only adds idle qubits.
Size the Pauli model with
cutoff,num-layersandlower-atolinstead.cost/repis the last measured wall time and peak GiB/node;?means nobody has timed it yet.size — the 10M / 100M / 1B / multinode-1B ladder (28 rungs)
st-10m-hubbard-propagatehubbard-cutoff=10, hubbard-lower-atol=TBDst-10m-pauli-propagatepauli-cutoff=12, pauli-lower-atol=TBDst-10m-random-propagatenum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-100m-hubbard-propagatehubbard-cutoff=10, hubbard-lower-atol=1.25e-05n1-100m-pauli-propagatepauli-cutoff=14, pauli-lower-atol=5e-05n1-100m-random-propagatenum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-1b-hubbard-propagatehubbard-cutoff=10, hubbard-lower-atol=TBDn1-1b-pauli-propagatepauli-cutoff=14, pauli-lower-atol=TBDn1-1b-random-propagatenum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn8-1b-hubbard-propagatehubbard-cutoff=10, hubbard-lower-atol=TBDn8-1b-pauli-propagatepauli-cutoff=14, pauli-lower-atol=TBDn8-1b-random-propagatenum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDst-10m-random-graph-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDst-10m-random-eval-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDst-10m-random-graph-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDst-10m-random-eval-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-100m-random-graph-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-100m-random-graph-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-100m-random-eval-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-100m-random-eval-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-1b-random-graph-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-1b-random-graph-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-1b-random-eval-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn1-1b-random-eval-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn8-1b-random-graph-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn8-1b-random-graph-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn8-1b-random-eval-heisenbergnum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDn8-1b-random-eval-schrodingernum-generators=1000, num-modes=250, cutoff=6, obs-terms=TBDweak — fixed terms per node, 1 to 64 nodes (21 rungs)
weak-97m-n1hubbard-cutoff=10, hubbard-lower-atol=1.25e-05weak-97m-n2hubbard-cutoff=10, hubbard-lower-atol=8.8e-06weak-97m-n4hubbard-cutoff=10, hubbard-lower-atol=5.9e-06weak-97m-n8hubbard-cutoff=10, hubbard-lower-atol=3.9e-06weak-97m-n16hubbard-cutoff=10, hubbard-lower-atol=2.6e-06weak-97m-n32hubbard-cutoff=10, hubbard-lower-atol=1.73e-06weak-97m-n64hubbard-cutoff=10, hubbard-lower-atol=1.14e-06weak-385m-n1hubbard-cutoff=10, hubbard-lower-atol=5.9e-06weak-385m-n2hubbard-cutoff=10, hubbard-lower-atol=3.9e-06weak-385m-n4hubbard-cutoff=10, hubbard-lower-atol=2.6e-06weak-385m-n8hubbard-cutoff=10, hubbard-lower-atol=1.73e-06weak-385m-n16hubbard-cutoff=10, hubbard-lower-atol=1.14e-06weak-385m-n32hubbard-cutoff=10, hubbard-lower-atol=7.35e-07weak-385m-n64hubbard-cutoff=10, hubbard-lower-atol=4.68e-07weak-1529m-n1hubbard-cutoff=10, hubbard-lower-atol=2.6e-06weak-1529m-n2hubbard-cutoff=10, hubbard-lower-atol=1.73e-06weak-1529m-n4hubbard-cutoff=10, hubbard-lower-atol=1.14e-06weak-1529m-n8hubbard-cutoff=10, hubbard-lower-atol=7.35e-07weak-1529m-n16hubbard-cutoff=10, hubbard-lower-atol=4.68e-07weak-1529m-n32hubbard-cutoff=10, hubbard-lower-atol=2.94e-07weak-1529m-n64hubbard-cutoff=10, hubbard-lower-atol=1.82e-07strong — fixed total terms, 1 to 64 nodes (17 rungs)
strong-1569m-n1hubbard-cutoff=10, hubbard-lower-atol=2.6e-06strong-1569m-n2hubbard-cutoff=10, hubbard-lower-atol=2.6e-06strong-1569m-n4hubbard-cutoff=10, hubbard-lower-atol=2.6e-06strong-1569m-n8hubbard-cutoff=10, hubbard-lower-atol=2.6e-06strong-1569m-n16hubbard-cutoff=10, hubbard-lower-atol=2.6e-06strong-1569m-n32hubbard-cutoff=10, hubbard-lower-atol=2.6e-06strong-1569m-n64hubbard-cutoff=10, hubbard-lower-atol=2.6e-06strong-6126m-n2hubbard-cutoff=10, hubbard-lower-atol=1.14e-06strong-6126m-n4hubbard-cutoff=10, hubbard-lower-atol=1.14e-06strong-6126m-n8hubbard-cutoff=10, hubbard-lower-atol=1.14e-06strong-6126m-n16hubbard-cutoff=10, hubbard-lower-atol=1.14e-06strong-6126m-n32hubbard-cutoff=10, hubbard-lower-atol=1.14e-06strong-6126m-n64hubbard-cutoff=10, hubbard-lower-atol=1.14e-06strong-24420m-n8hubbard-cutoff=10, hubbard-lower-atol=4.68e-07strong-24420m-n16hubbard-cutoff=10, hubbard-lower-atol=4.68e-07strong-24420m-n32hubbard-cutoff=10, hubbard-lower-atol=4.68e-07strong-24420m-n64hubbard-cutoff=10, hubbard-lower-atol=4.68e-07CI
One workflow change, and it is not a gate:
bench.ymldrops theresting-memorythreshold andgains an artifact upload of
bmf.jsonand the raw results, so a failed Bencher upload no longerloses the measurement.
bench_main.ymlandbench_bare_metal.ymlare untouched and still justtrack the main branch.
Drive-by fix
report.py's_display_opcollapsed everytest_model_*[hubbard]andtest_model_*[pauli]rowto the same label, so the two models were indistinguishable in the report. A parameter that is not
a picture now names the model and replaces the group:
hubbard / propagate,pauli / build_graph.Regression test updated in both directions.
Verification
Run on a Deucalion login node against this branch:
uv run pytest— 662 passed, 8 skipped (the 8 need--with-mpi).prek run --from-ref origin/main --to-ref HEAD— every hook clean on the changed files.monoprop-bench-report— the resting-footprint section is gone; the hubbardand pauli rows are now distinct.
monoprop-bench-bmf— exactlylatency,peak-memory,terms; 20 benchmark names, allunchanged from
main.argsentry of all 58 rungs checked againstpytest --help's option list — no unknownflags.
expect_terms(137 measured vs 2,276 declared), then passed once corrected, and the ladder rendered both the
table and the resolved model parameters.
Not done here
The 18 uncalibrated rows need a cluster allocation to measure; they refuse to run until then. The
private harness is updated separately to call
monoprop-bench-runginstead of its own launchers,and keeps working unchanged until it is.