test(bench): record the geometry a scaling rung is identified by

[diagonal-hamiltonian]

What this is

Two commits that make a benchmark result identify the geometry it was measured at, plus the reader change that surfaces them. Nothing in the engine changes; both commits are Python, outside cpp/.

• benches/conftest.py records nodes, ranks_per_node, partitions_env and memhwm_max.
• report.py / test_report.py surface them.

Why

A scaling result is (nodes, ranks_per_node, partitions), and until now a recorded run carried none of the three. meta.ranks is the MPI world size, which is the product of two of them and cannot be factored back. That matters more than it sounds, because the engine's flat world is P = ranks_per_node × partitions × nodes and HybridComm::size() returns r_ * s_ — so 8 ranks × 16 partitions on 32 nodes and 1 rank × 128 partitions on 32 nodes are both P = 4096 and are not the same measurement. They differ by 2.06x in wall time (table below). Two runs that differ 2x were previously indistinguishable in the recorded JSON.

Three details worth review:

• nodes is collective. It is an allgather of socket.gethostname(), so pytest_configure had to call it before the rank-0 early return — every rank must enter it or it hangs. Serial short-circuits to (1, 1) without touching MPI.
• partitions_env is the environment variable, not a resolved count, and is labelled "Partitions (requested)" for that reason. No property exposes the count the engine actually chose. A bare Partitions column would have claimed otherwise.
• memhwm_max exists because memhwm is a sum over ranks. The two differ by the rank count. The old heading said neither, so this renames the existing table to "summed across ranks" and gives the max its own. On a 1024-rank run the sum is 460 GiB and the max is 0.48 GiB; a reader who guessed wrong about which one they were looking at would be off by 1000x.

world is deliberately not emitted: it is derivable from what is recorded, and a stored derived field is a second source of truth that can disagree.

What it was used for

A strong- and weak-scaling study of propagate on Hubbard at 1,569,152,761 terms (cutoff 10, 60 sites, lower_atol=2.6e-06), on Deucalion x86 (2x EPYC 7742, 128 cores/node, SMT off). Layout 8 ranks x 16 partitions per node, so the engine's flat world is P = 128 x nodes. Baseline is this branch's parent; the port arm is #263. 3 reps per cell, A/B interleaved inside one allocation with the arm order flipped per rep, ratios taken as the median of per-rep paired ratios.

Strong scaling — the problem is fixed, only the hardware grows

N	P	propagate s	speedup	efficiency	ledger coverage
1	128	281.0	1.00	100%	91.0%
2	256	152.6	1.84	92%	85.4%
4	512	103.2	2.72	68%	76.5%
8	1024	82.2	3.42	43%	65.2%
16	2048	69.4	4.05	25%	49.9%
32	4096	88.5	3.18	10%	40.4%

Scaling reverses between 16 and 32 nodes. Reproduced in two independent passes (a 1-rep pass gave 4.07 → 3.21, a 3-rep pass 4.05 → 3.18). Weak efficiency falls 100.0 / 81.6 / 56.2 / 38.5 / 27.1 / 16.0% over N = 1…32, from main-arm walls of 18.7 / 21.7 / 32.4 / 49.0 / 69.8 / 116.6 s. It is normalised by the measured terms/node recorded in each cell's ledger (92.06–98.07M across the six rungs), not by an assumed-flat target — the rungs are close enough that the distinction is small, and far enough apart that assuming it would have been wrong.

Instrument self-check, and the reason the nodes key earns its place: strong/16 and weak/16 are the identical cell (1.569G terms at N=16) run in two separate allocations, and they agree to 0.56% (base) and 0.26% (port). Before this commit the two were indistinguishable in the recorded JSON and could not have been cross-checked at all.

Where the time goes — this is what time -v says once User and System are separated

Same runs, main arm, median rep, summed over ranks:

N	ranks	user core-s	system core-s	system share	user vs N=1	system vs N=1
1	8	29,008	6,262	17.8%	1.00x	1.00x
4	32	30,316	21,382	41.4%	1.05x	3.41x
8	64	38,402	45,287	54.1%	1.32x	7.23x
16	128	56,555	83,546	59.6%	1.95x	13.34x
32	256	117,014	241,152	67.3%	4.03x	38.51x

Total CPU consumed grows 10.14x for a problem that does not change, against a 3.18x speedup — so ~90% of the CPU burned at N=32 does not exist at N=1. It is overwhelmingly system time, and system time is very nearly linear in MPI rank count (exponent log 38.51 / log 32 = 1.05).

Layout at fixed flat world — six geometries, same P

At N=32, P = 4096 is reachable at six splits of 128 cores/node, with the MPI world size spanning 32x. This is the measurement ranks_per_node exists to make legible:

layout	partitions/rank	MPI ranks	propagate s	vs 8x16	threads busy	peak RSS sum
1x128	128	32	212.8	2.24x	95%	173.3 GiB
2x64	64	64	122.5	1.29x	70%	177.6 GiB
4x32	32	128	94.4	0.99x	78%	190.6 GiB
8x16	16	256	95.0	1.00x	58%	218.7 GiB
16x8	8	512	127.3	1.34x	66%	278.6 GiB
32x4	4	1024	300.2	3.16x	62%	460.4 GiB

Three things worth having in the record:

• A U with its minimum at the current default. 8x16 is within 0.2% of the best cell, so there is no free win in the geometry, and the 3.18x spread across the row is entirely invisible to a JSON that records only ranks.
• Thread occupancy is anti-correlated with speed. 1x128 saturates its cores at 95% busy and is the slowest cell; 8x16 runs 58% busy and is the fastest. Busy is not productive.
• Peak RSS is linear in MPI ranks at ~0.29 GiB per process (173.3 GiB at 32 ranks to 460.4 GiB at 1024, so +287 GiB over +992 ranks). This is what memhwm_max beside memhwm is for: the summed figure spans 173→460 GiB across the row while the per-rank max falls 5.55→0.48 GiB. A reader who mistook one for the other would be wrong by three orders of magnitude.

#263, for reference

Its paired ratio is ~0.8 at narrow widths and 0.96 at N=32, and it tracks the paired CPU-seconds ratio to within +0.014/−0.025 at every rung — i.e. predominantly work removal, diluted as that work becomes a smaller share of a growing overhead. Per layout at N=32 it runs 0.95 / 0.85 / 0.80 / 0.91 / 0.97 / 0.73 for 1x128 / 2x64 / 4x32 / 8x16 / 16x8 / 32x4. The last is not quotable and is shown only for completeness: 32x4's three paired ratios are 0.975 / 0.727 / 0.541, and its main arm alone spans 1.93x within the cell (300.2 / 332.1 / 579.2 s) where every other cell holds inside 6%. An earlier revision of this body printed 0.975 for it — that was rep 1 read while the cell was still filling, which is exactly the failure mode a 1024-rank cell invites. Any claim at this width needs far more than 3 reps.

Every ratio above is descriptive, not resolved. Three paired reps floor the uncorrected two-sided sign test at p=0.25, so no 3-rep cell in this comment carries statistical significance; the ladder buys the shape. The layout comparison is additionally unpaired — six allocations on six node sets — so no sign test applies to it at all.

Resolved at 10 reps, at the two rungs where the answer was in doubt

N	reps	port/main median	agree	sign p	system ratio
16	10	0.691x	10/10	0.0020	0.633x
32	10	0.950x	6/10	0.754	0.938x

Per-rep ratios, N=16: 0.671 / 0.677 / 0.680 / 0.689 / 0.689 / 0.693 / 0.715 / 0.723 / 0.755 / 0.858. N=32: 0.890 / 0.894 / 0.897 / 0.917 / 0.942 / 0.959 / 1.004 / 1.005 / 1.007 / 1.019.

The two distributions do not overlap at all — every N=16 ratio is below every N=32 ratio. An exact two-sided Mann-Whitney (the appropriate test, because comparing two rungs is unpaired: separate allocations, separate node sets) gives U = 0, p = 1.08e-5, which is the floor for a 10-vs-10 design. So this is not "0.69 versus 0.95, probably different" — it is the strongest separation the experiment can express.

Conclusion: #263 buys a resolved ~1.45x at 16 nodes and is indistinguishable from no change at 32. Quoting a single number for "#263's improvement" is therefore wrong; the benefit is scale-dependent and disappears at the flat world size where propagate itself reverses (P = 4096). Two independent ladders agree on this: the weak ladder's 32-node rung is likewise its weakest cell.

A calibration warning that applies to every 3-rep figure above, including mine. The same comparison, same two binaries (installed _core.so md5 edb00c44… vs 7dbc7de1…, neither rebuilt between runs — both .so files predate both jobs), byte-identical configuration, differing only in rep count, measured 0.857x at 3 reps and 0.691x at 10 on two different 16-node allocations. The 3-rep median landed at the very top of the 10-rep per-rep range. So a 3-rep ratio here can be ~24% away from the resolved value even with paired, order-flipped interleaving inside one allocation — far beyond the ~1.7% between-job drift seen elsewhere. Read the 3-rep tables above for shape only, never for magnitude.

A large part of this wall was our own benchmark environment, not the engine

Reported here because it changes how the walls above should be read, and because it is a trap any
HPC harness for a multi-threaded engine can fall into.

The harness that produced every number above exported MALLOC_ARENA_MAX=$PARTITIONS = 16. The engine
runs 19 threads per rank (16 partition masters, the main thread, the OFI async progress thread,
and one more Python thread), so late claimants share an arena and every malloc/free in the
local-work region becomes a futex sleep. Unsetting it, one binary, 10 paired reps per rung with the
arm order flipped every rep inside a single allocation:

N	unset / =16	reps agreeing	sign p	median wall	saving	peak RSS
16	0.7732x	10/10	0.0020	69.02 → 53.34 s	15.67 s (22.7%)	+0.080 GiB/node
32	0.8304x	10/10	0.0020	94.52 → 76.32 s	18.19 s (19.2%)	+0.107 GiB/node

Do not subtract these savings from the walls in the tables above. The =16 arms here read
94.52 s at N=32 where the strong-scaling table reports 88.5 s — a 6.8% gap — while a separate,
independently-instrumented probe of the same configuration read 94.50 s, agreeing with these runs to
0.02%. Two runs agreeing to two parts in ten thousand means 94.5 s is the figure for this
provisioning and the 88.5 s cell was provisioned differently. So "88.5 − 18.19 = 70.3 s" is a
configuration mix, and it looks entirely reasonable on the page. The paired ratios are safe precisely
because each lives inside one allocation; only the cross-quoting of absolute seconds is not.

MALLOC_ARENA_MAX is a cap, not an allocation: glibc creates an arena only when a thread finds
the existing ones contended, at most one per thread, so unsetting it yields ~19 arenas rather than the
8 × ncores = 1024 the documentation's formula suggests. That is why the memory cost is ~0.1 GiB/node
against ~245 GiB of headroom, and why there is no trade to weigh here.

This is a wall win at every width, not a scaling fix, and the distinction is easy to miss. Both
walls fall substantially and the reversal gets worse: 1.370x → 1.431x, because the relative
saving is larger at the smaller rung. Removing a cost that is roughly rung-independent always shrinks
the smaller wall proportionally more. A reader checking only that both numbers improved would conclude
the opposite of what happened.

Two consequences for the tables above, stated rather than quietly corrected:

• The A/B ratios stand. Both arms of every comparison ran under the same contended allocator, and
  the comparisons are paired within one allocation. The perf(evolution): ⚡ carry cross-rank queries as a width-adaptive record #263 result (0.691x at N=16, 0.950x at N=32) is
  unaffected.
• The attribution shares are distorted, because arena contention is charged to the barrier-wait
  and system-time terms. The system share column in particular — 17.8% → 67.3% across the ladder —
  is measuring the engine and this misconfiguration together, and no split between the two is offered
  here.

One diagnostic worth recording, because it is what ruled out every topology explanation: which
partition indices are penalised changes between runs of the identical configuration on the same nodes.
A core, cache, NUMA, or IRQ-affinity story predicts a stable set. Only a lazy, racy arena assignment
reproduces that. voluntary_ctxt_switches on an affected thread tracked the penalty exactly (557,537
vs 68) while minor faults stayed flat, so it is lock contention rather than page-fault churn.

Reproducibility

The numbers above are not reproducible from this diff. They were produced by a private harness (allocation management, A/B interleaving, /usr/bin/time -v collection, collation) that is not in this repo and is not proposed for it. This PR contains only the recording and reporting change that makes such a campaign interpretable. Every figure is quoted against the commit that measured it.

[github-actions]

Docs preview: https://pr-287.monoprop-docs.pages.dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (55c4c0b) to head (99a3aa5).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #287   +/-   ##
=======================================
  Coverage   97.70%   97.70%           
=======================================
  Files          14       14           
  Lines         742      742           
  Branches       98       98           
=======================================
  Hits          725      725           
  Misses         12       12           
  Partials        5        5           

Flag	Coverage Δ
cpp	97.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud