perf(mpi): 🧭 GF(2)-linear rank routing — messages per rank flat in R

[diagonal-hamiltonian]

Implements a GF(2)-linear rank hash, in the same spirit as arXiv:2506.13241 but not the same construction — see Relation to prior work below.

Stacked on #263 (base 9ff9d732). Merging #263 retargets this to main unchanged.

Updated 2026-08-29 — ten further commits (a610f048..2f8d3dc2) collapse the d dial, take the
emit path off the hash, size the per-generator bookkeeping to the reachable window, overlap the
count round, and carry the lever onto propagate()/energy()/gradient(). Every timing table
below is labelled with the commit it was measured on. The Leonardo ladder and the
#263-composition table predate these commits and are kept as the record of the routing change in
isolation; the three Deucalion campaigns are current.

What this does

main's MPI cost obeys

MPI time  =  collectives per rank (FIXED by the algorithm)  ×  R  ×  (R−1)  ×  cost per peer-message

#263 attacks the last factor. This attacks (R−1), which is not physics — it is a consequence of one
line. The destination of a query was monomial_hash(M ⊕ G) % R, and monomial_hash is splitmix,
whose full avalanche is exactly what makes the exchange dense. Every earlier attempt at a sparse
transport failed because the routing stayed dense: a rank pair is non-empty if any of its S²
partition pairs has data, so at 8.7% partition-level occupancy the chance a rank pair is empty is ~2e-8.

Make the rank-level hash GF(2)-linear instead:

h(M) = XOR of v_i over support(M)      ⇒      h(M ⊕ G) = h(M) ⊕ h(G)

A rank owns M, so every query it emits for generator G lands on my_rank ⊕ Δ(G) — fanout 1.
XOR is an involution, so that peer sends back on the same round. Nothing about the term set, the
volume moved, or the gate structure changes; only which rank a term lives on.

Routing is two-level, because the two levels have different costs: GF(2)-linear across MPI ranks,
where message count is the cost, and full-avalanche splitmix across partitions within a rank, where
fanout is free (shared memory) and only balance matters.

flat_dest(M) = rank_linear(M) * S  +  splitmix(M) % S

monoprop_ROUTING selects the arm: linear (default, fanout 1) or splitmix (the previous
behaviour, bit-identical to the base). Power-of-two R is required and a non-power-of-two R
now throws UnroutableGeometry rather than silently falling back — a silent fallback was a way to
measure the wrong arm without noticing. R == 1 takes no rank bit and so is the dense case by
construction, which is why the collective path stays.

Commits

Routing (22ad7049 and below):

route through one Router	one routing::Router owns the term→slot map, replacing two inline copies of hash % P. mpi::geometry() is new: size() cannot tell an inter-rank message from an inter-partition one.
skip the exchange for identity generators	a generator with empty support anticommutes with nothing, so both its exchange passes carry zero queries — but collectives fire regardless of payload. 60 of the 60-site Hubbard's 476 generators per Trotter layer are these, i.e. 12.6% of all MPI calls doing nothing.
exchange point-to-point over the peers routing can reach	mpi::PeerPlan threads the fanout to HybridComm; every O(R·S²) sweep that runs serially on partition 0 shrinks to O(f·S²). check_routing_agreement() allreduces the configuration once at construction, because a rank that missed the env var would hang, not answer differently.
route GF(2)-linear by default	flips the default; splitmix is the explicit opt-out.

Follow-through (a610f048..2f8d3dc2, new):

collapse the linear-bit dial to a boolean	d was an integer dial where a boolean does: the shipped configuration is d = log2 R and d = 0 is the control, and the intermediate values existed only to be tested. PeerPlan::bits becomes bool sparse; monoprop_ROUTE_LINEAR_BITS, hi_mask_, the hi term and a 64-bit division on a runtime divisor all go. −139 lines, and one whole dimension out of the test matrix.
route the emit path from the generator's shift	dest(M ⊕ G) was recomputing a per-generator constant per emitted term. rank(M ⊕ G) = my_rank ⊕ Δ(G) is an identity, and Δ(G) is already computed once per generator for the PeerPlan. Removes the parity-plane fold (28 loads + 28 AND + 28 XOR + 7 popcount per query at production widths) from the hot loop.
pack the send side under the count round	the count block is S² ints — eager, and fully overlappable with packing. Split post/wait and pack between them.
replay the layer exchange point-to-point + gate the pairwise replay on routing	begin_flat_exchange posted a dense Ialltoallv with no PeerPlan, so the graph-replay path got zero benefit. Now gated on a rank-uniform wire_bits derived from the routing configuration — the obvious gate (each rank counts its own non-zero legs) is rank-varying and deadlocks when the decision straddles.
name the slot window a peer plan can reach + size the query path to the peer window + narrow the staging sweeps	at fanout 1 only S of the P = R·S flat slots can receive a query, yet the bookkeeping was sized and swept over P — ~18 sweeps of length 2048 per generator per pass at R=128, S=16. SlotWindow{base, count} is contiguous (that is what the boolean buys), and the dense case is the count == P value of the same path, not a second implementation.

Measured

Balance — settled by measurement, not by the dial

One instrumented run of the production point (60 sites, cutoff 10, lower_atol=2.6e-06 →
1,569,152,761 terms) recorded a joint histogram over every owned term at ten layers, so every
(R, S) is a marginal of one table and d=0 is its own control:

layer	terms	rank max/mean d=0	fanout 1	ranks used
3	21,148	1.265	1.204	128/128
5	346,334	1.047	1.051	128/128
12	151,921,485	1.003	1.002	128/128
29	1,569,152,761	1.001	1.001	128/128

Imbalance is confined to layers 1–3, which hold ≤21k of 1.57e9 terms (0.0013%). This is a property of
the map, not of the transport, so it is unchanged by the follow-through commits.

Campaign A — the full branch against #263's head

22ad7049 → 2f8d3dc2, two binaries, md5-gated. Deucalion x86, layout 8×16, 10 reps per cell
interleaved inside one allocation with the arm order flipped per rep, paired per-rep ratios then
median. All 15 timing tests survive Holm at family size 15 (p = 0.00195 or 0.0215 against a
0.05/15 = 0.0033 floor).

operation	N=4 (R=32)	N=8 (R=64)	N=16 (R=128)
propagate[hubbard]	1.32× (0.76)	1.56× (0.64)	1.99× (0.50)
build_graph[pauli]	1.23× (0.81)	1.34× (0.75)	1.53× (0.65)
propagate[pauli]	1.26× (0.79)	1.52× (0.66)	1.84× (0.54)
energy[pauli]	1.28× (0.78)	1.39× (0.72)	4.96× (0.20)
gradient[pauli]	1.20× (0.84)	1.46× (0.68)	3.05× (0.33)

The gain grows with R in every row, which is the signature the work removed is proportional to P.

Campaign B — the lever itself, re-baselined on the current tip

monoprop_ROUTING=splitmix → linear on one binary (633df79e, a bench-only tree), so the arms
differ in nothing but the routing arm, read back per-rep from the in-process config. The harness
refuses same-md5 arms and printed its usual banner; the readback (splitmix on all 20 main files,
linear on all 20 port files) is what makes the cells valid. All 15 tests survive Holm.

operation	N=4 (R=32)	N=8 (R=64)	N=16 (R=128)
propagate[hubbard]	2.89× (0.35)	4.23× (0.24)	8.19× (0.12)
build_graph[pauli]	2.83× (0.35)	3.94× (0.25)	5.86× (0.17)
propagate[pauli]	2.82× (0.35)	4.49× (0.22)	7.20× (0.14)
energy[pauli]	1.56× (0.64)	1.72× (0.58)	1.49× (0.67)
gradient[pauli]	1.47× (0.68)	2.34× (0.43)	2.66× (0.38)

This is a per-operation figure, not the whole-run wall the 0.32× headline below measures — the two
are not directly comparable. The engine's byte ledger also shows the graph shrinking 2.06× at N=16
(14.19 → 6.87 GiB) — expected, since the retained graph is quadratic in the flat world and fanout 1
collapses the peer set — but /usr/bin/time -v did not run in these cells, so that is a ledger
number (capacity, not resident bytes) and not a peak-RSS claim.

Campaign C — attribution

3e6ab805 → 2f8d3dc2 isolates the windowing commits. Only 4 of 12 tests survive Holm at family
size 12, so read the rest as unresolved:

operation	N=4	N=8	N=16	Holm
propagate[pauli]	1.24× (0.80)	1.48× (0.68)	1.90× (0.53)	survives at all three
build_graph[pauli]	1.17× (0.85)	1.29× (0.78)	1.43× (0.70)	survives at N=8 only
energy[pauli]	0.97×	1.02×	1.04×	none
gradient[pauli]	0.94×	0.99×	1.00×	none

Subtracting C from A: propagate's gain is the windowing, and energy/gradient's 4.96×/3.05×
at N=16 is not — it comes from the commits below 3e6ab805, of which the point-to-point graph
replay is the only one touching that path. This also explains the apparent energy 1.04× in C: the
replay commit had already taken that cell from 4589 ms to 883 ms, leaving the windowing nothing to win.

Strong ladder (Leonardo DCGP, measured at 27da5fad — routing only, no follow-through)

3 arms × 2 layouts × 5 rungs × 2 reps, interleaved inside one allocation per rung, ITAC on every cell:

N	R	main	this	ratio	main msgs/rank/layer	this
1	8	200.8 s	188.8 s	0.94	19,992	1,274
2	16	113.6 s	98.7 s	0.87	42,840	1,350
4	32	67.6 s	49.8 s	0.74	88,536	1,388
8	64	47.9 s	25.3 s	0.53	179,928	1,395
16	128	46.5 s	15.0 s	0.32	362,712	1,397

Strong efficiency 27% → 79%; on 4×28, 31% → 82%. Messages per rank per layer are flat in R
(1,274 → 1,397, +10% across a 16× increase in R) against main's +18×, i.e. 260× fewer at R=128.
The residual +10% is the self-peer effect (shift 0 ⇒ the peer is me ⇒ memcpy, no message), which
saturates. main's reversal is gone: 4×28 turns over at n8→n16 (45.6 → 48.0 s) while this keeps
falling (28.2 → 17.2 s). Term count was 1,569,152,761 on every arm at every flat world.

The same lever applies to the Pauli path with no code change (a Pauli in symplectic form is a vector
in F₂^2n and multiplication is XOR of those vectors up to phase). On the 127-qubit heavy-hex kicked
Ising, fanout 1 gives 1.039 max/mean at R=128 against splitmix's 1.052. The identity-generator skip
does not transfer: those were an artifact of Hubbard's interaction gates.

Correctness

Routing changes which rank owns a term, so cross-rank miss indices mint in a different order and
reductions reassociate. The bar is term count exact, expectation value within a few ULP, with
bit-identity required on the splitmix arm.

• splitmix is bit-identical to the base at every (world, S) pair tested — same term count and
  the same 64 bits of expectation value.
• Linear routing: term count exact in every configuration, worst deviation 1 ULP.
• 309/309 ctest at 2f8d3dc2, and 593/593 MPI tests at flat worlds 2, 32 (two layouts) and
  256. routing_tests is 17 cases, hybrid_comm_tests 20, plus a new flat_exchange_tests (6) for
  the point-to-point replay — including the shift identity route(M⊕G) == route(M) ⊕ Δ(G), fanout
  exactly 1, window() agreeing with peer()/contains() in both the sparse and dense cases, the
  non-power-of-two throw, and the derived wire plan over an empty partition-0 row.

Collapsing d removes tests by construction, so the surviving cases were checked to still reach
both routers rather than assumed to.

Scope

Routing/transport only — no gate fusion; gates stay atomic and there is still exactly one exchange
per generator per pass. The dense path stays, because R == 1 routes through it.

Net +1,694 / −664 over the ten new commits and +3,231 / −430 for the branch, of which
HybridComm.h is +331 / −127. The earlier claim of "~120 lines" was measured before the
follow-through; the windowing replaced the parallel O(P) indexing but did not shrink the file, and
the only net deletion in the stack is the d collapse at −139. What actually got simpler is the
per-exchange serial work (O(R·S²) → O(f·S²)), the per-generator bookkeeping (O(P) → O(S)), one
Router where routing was two inline expressions, and a boolean where there was a four-valued dial.

Composing with #263 (measured at 27da5fad, before the follow-through)

4 arms × 8×14 × {4, 8, 16} nodes × 2 reps, interleaved in one allocation per rung:

N	R	main	#263 alone	this alone	both	vs this alone	msgs/rank/layer	KB/rank/layer
4	32	67.0 s	53.2 s	50.1 s	40.7 s	-19%	88,536 -> 1,388	852k -> 368k
8	64	52.1 s	54.7 s*	26.2 s	22.5 s	-14%	179,928 -> 1,395	433k -> 187k
16	128	44.7 s	40.9 s	14.4 s	13.8 s	-4%	362,712 -> 1,397	221k -> 96k

* one cold rep (42.0 / 67.4 s); the #263-vs-main read at n8 is not usable.

The two levers compose in the mechanism: the stacked arm carries this branch's message count
unchanged and roughly half its wire volume at every rung. Routing moves messages, #263 moves
bytes, and neither undoes the other. The wall gain is sub-additive and shrinks with R, which is what
fanout 1 implies: per-peer cost is multiplied by 1 rather than by (R−1).

At 27da5fad the residual was 90–91% MPI_Waitall at a thread spread of 1.02–1.04 — latency
everyone pays alike — which is what the follow-through commits attack.

Two questions this raises, answered

Should the splitmix arm be removed? No. It is the bit-identity control for a change that moves
term ownership, it is the opt-out arm every ratio in Campaign B is measured against, and it is the
recovery path when the generator shifts span only ρ < d dimensions and 2^d − 2^ρ ranks would sit
empty (which report_routing_coverage_ warns about). Removing it would also not remove splitmix:
SplitmixHash is the hash for OperatorIndex and MonomialMap and is untouched by routing.

Should the linear hash be used elsewhere — e.g. for OperatorIndex? No, and this is the one place
it must not go. Table is open-addressing with linear probing. A hash of the form ⊕_{i∈supp} v_i
is exactly 2-independent, not 3-independent: if M₃ = M₁ ⊕ M₂ then h(M₃) is determined by the
other two. Pătraşcu–Thorup showed 2-independence is insufficient for linear probing, and monoprop's
key set is precisely XOR-generated — the scan inserts M ⊕ G for a fixed small generator set — so
the dependent triples are the workload, not a corner case. Keep fold_hash/spread as they are. (For
the same reason, an XOR set checksum cancels duplicates and is not a safe agreement digest.)

One free variable is left unused: linear_basis is drawn at random, but the generator list is
replicated and known at construction, so the basis could be fitted so the shifts provably span
F₂^d — turning the coverage warning into a guarantee. Tracked as a follow-up, not done here.

Relation to prior work

The PR description points at arXiv:2506.13241 (Broers, Sun &
Yunoki, Scalable Simulation of Quantum Many-Body Dynamics with Or-Represented Quantum Algebra;
Phys. Rev. Applied 26, 024046). Worth being precise, because it is not the same hash, and the
difference is the whole result.

ORQA's distribution map (their eq. 12) is

f(I) = [ Σ_j I^(j,k) ] mod N

— the 2n-bit multi-index cut into k-bit blocks, each read as an integer, summed over ℤ_N. That is
additive over the integers mod N. The gate acts by XOR, and addition mod N carries, so the two
operations do not commute. Their eq. 13 makes the consequence explicit:

f(I⊻J) = m + Σ_{j=1..2|J|} ±2^(J_j mod k)   mod N

The signs depend on the current bits of I, so the destination is not determined by J alone: they
bound it at 2^(2|J|+1) processes (17 in their setting). That is a large reduction in fanout, and
it is why the paper needs a stochastic perturbation (their eq. 14) to recover load balance afterwards.

This PR takes the hash linear over the same group the gate acts by. Monomials under a gate form
(F₂^{2n}, ⊕), and h(M) = ⊕_{i ∈ supp(M)} v_i is a homomorphism of it, so h(M ⊕ G) = h(M) ⊕ h(G)
is an identity, not a bound. Three things follow that eq. 13 does not give:

• Fanout is exactly 1, independent of |supp G| and of R — not 2^(2|J|+1), and not a function
  of the gate weight at all.
• The pairing is an involution, so a fixed generator partitions the ranks into disjoint pairs that
  each side derives with no communication. The response retraces the query for free.
• Balance is structural: the fibres h⁻¹(r) are cosets of ker h, all of size 2^(2n-d), so a
  uniformly drawn monomial is balanced by construction. No perturbation term is needed — measured rank
  max/mean 1.001 at R=128 from layer 5 on.

Two further differences: our routing is two-level where ORQA is flat; and they move updates with
one-sided RMA (MPI_Put) rather than the Isend/Irecv pairs here. Two things they have that we do
not, both tracked as follow-ups: the RMA transport, and scale — they report strong scaling to 2^17
processes on Fugaku against the R=128 measured above.

🤖 Generated with Claude Code

[github-actions]

Docs preview: https://pr-296.monoprop-docs.pages.dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (9ff9d73) to head (27da5fa).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@                Coverage Diff                @@
##           pr/query-wire-v3     #296   +/-   ##
=================================================
  Coverage             97.70%   97.70%           
=================================================
  Files                    14       14           
  Lines                   742      742           
  Branches                 98       98           
=================================================
  Hits                    725      725           
  Misses                   12       12           
  Partials                  5        5           

Flag	Coverage Δ
cpp	97.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[Copilot]

Pull request overview

This PR improves MPI scalability by changing term routing so that (when the MPI rank count is a power of two) each generator’s cross-rank queries route to a predictable XOR-derived peer set, enabling sparse point-to-point exchanges and keeping messages-per-rank approximately flat in R. It also adds correctness/consistency checks and new test coverage to ensure routing and exchange behavior stays aligned across the codebase.

Changes:

• Introduce routing::Router for a two-level routing scheme (GF(2)-linear across MPI ranks, splitmix within-rank partitions) and route all ownership decisions through it.
• Add mpi::PeerPlan and integrate sparse peer exchange into HybridComm / MPICompat alltoallv paths, plus skip exchange work for identity generators.
• Expand tests to pin routing identities, scan/find-rank agreement under both routers, and correctness of sparse peer delivery.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
cpp/monoprop/detail/mpi/Routing.h	New routing::Router implementing two-level (linear+splitmix) routing with env-configured defaults.
cpp/monoprop/detail/mpi/MPIUtils.h	Route ownership via Router, add router_for(), and add a cross-rank routing-agreement check.
cpp/monoprop/detail/mpi/Comm.h	Add mpi::PeerPlan to describe sparse reachable peers for point-to-point exchange.
cpp/monoprop/detail/mpi/MPICompat.h	Thread PeerPlan through alltoallv helpers and add sparse point-to-point fallback for S==1 MPI-only comms.
cpp/monoprop/detail/mpi/MPICompat.cpp	Extend alltoall_counts to support sparse plans, including MPI-only point-to-point counts exchange.
cpp/monoprop/detail/mpi/HybridComm.h	Add sparse-plan support to count/payload phases (plan-aware packing, exchange, and scatter).
cpp/monoprop/detail/mpi/MPICompat.h / .cpp	Add geometry() split (ranks vs partitions) and use it to drive routing/plan decisions.
cpp/monoprop/detail/mpi/CMakeLists.txt	Register new Routing.h header in build sources.
cpp/monoprop/detail/evolution/layer_build/Scan.h	Emit query destinations via Router (single source of truth for ownership).
cpp/monoprop/detail/evolution/layer_build/Engine.h	Derive PeerPlan per generator from Router and use it in query/response exchanges; skip exchanges for identity generators.
cpp/monoprop/detail/monomial_propagator/MonomialPropagator.inl	Seed initial ownership using Router and enforce routing agreement at construction.
cpp/tests/routing_tests.cpp	New tests pinning routing invariants (bit-identical d=0, GF(2) linearity, shift identity, fanout, defaults).
cpp/tests/mpi_utils_tests.cpp	Update scan/find-rank agreement test to validate both routing modes and pass Router through.
cpp/tests/hybrid_comm_tests.cpp	Add tests ensuring sparse plans deliver only to plan peers on both Hybrid and plain-MPI paths.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[diagonal-hamiltonian]

🤖 AI text below 🤖

Correct, this is a real defect — the comment was written before route GF(2)-linear by default (40041c1) flipped the default and was not updated. As written it contradicts the PR title, the commit, and routing_tests.cpp:219, which pins the shipped default at fanout 1.

Fixed the comment rather than the code: linear is the intended default, monoprop_ROUTING=splitmix is the explicit opt-out.

The comment was also the only documentation these three knobs had. They are now documented on the parallelism page in the docs, and their parsing has moved into EnvConfig.h so that an unparseable value raises instead of silently selecting d = 0 — previously monoprop_ROUTE_LINEAR_BITS=abc turned the whole feature off with no diagnostic.

[diagonal-hamiltonian]

🤖 AI text below 🤖

Pushing back on this one. The assertion is seed-robust by construction: it draws 200 shifts and checks they span $\mathbb{F}_2^{7}$, and 200 random vectors fail to span a 7-dimensional space with probability around $2^{-194}$ — for any seed, not just the default. monoprop_ROUTE_SEED changes which basis, not whether a large random sample spans.

Skipping under an override would also remove the coverage in exactly the situation the comment names: someone debugging balance with a custom seed is precisely who wants to know the shifts still reach every rank.

Added a comment stating why it holds for any seed, so the next reader does not have to re-derive it.

For contrast, the neighbouring routing_default_is_linear_where_the_geometry_allows_it does skip on an override, and correctly so — there the environment default genuinely is the thing under test.

[sonarqubecloud]

Quality Gate passed

Issues
15 New issues
0 Accepted issues

Measures
0 Security Hotspots
58.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud