petsc_interpolate: the DMLocatePoints call count is rank-local (cache key + unreduced location capability) — hangs at np>=4

[lmoresi]

Summary

tests/parallel/test_0760_swarm_cache_migration.py hangs at mpirun -np 4 on current development (pre-existing — verified on a pristine baseline during the gating of #313, by stashing that branch's src changes and rebuilding).

Mechanism

With rank-biased particle coordinates, global_evaluate leaves some ranks with zero interior points. Those ranks diverge inside the DMInterpolation machinery in petsc_interpolate while the remaining ranks block in its collectives — an MPI deadlock. It passes at np2, which is why CI (test.sh --p 2) never sees it.

Per-rank stack traces were captured during #313's gating and confirm the hang is inside the DMInterpolation path; the swarm-cache fixes in #313 enter and exit their collective hooks symmetrically before the hang (i.e. #313 neither causes nor masks this).

Relation to the 2026-07 quality audit

Corresponds to the provisional findings SWARM-15/16 in docs/reviews/2026-07/SWARM-SUBSYSTEM-REVIEW.md (parallel-evaluation empty-rank asymmetry family; see also BF-16 in the remediation worklist — substep advection evaluating on the wrong rank).

Reproduction

mpirun -np 4 python -m pytest tests/parallel/test_0760_swarm_cache_migration.py on development — hangs. Same at np2 passes.

Underworld development team with AI support from Claude Code

[lmoresi]

Root cause located, and #513 consolidated in here — one defect, two triggers.

The number of DMLocatePoints calls on the world mesh DM inside petsc_interpolate is a rank-local function of that rank's coordinate array. Two rank-local switches produce it:

• the DMInterpolationCache is keyed on a per-rank coords hash (function/_function.pyx:1293, function/dminterpolation_cache.py:124-132) — a hit skips create_structure → DMLocatePoints entirely, and hit-vs-miss is decided per rank;
• _location_capability() is deliberately not reduced (discretisation/discretisation_mesh.py:6334-6339), and the policy it yields is itself part of the cache key, so a capability split forces a hit on some ranks and a miss on others.

The entry guards added earlier (_function.pyx:1166-1170, :1012-1016) correctly get every rank into petsc_interpolate; nothing keeps them in lockstep once inside.

Both triggers reach it: a rank with zero interior points (this issue's original) and rank-distinct partially-off-domain points (#513's). #513's puzzling observation — that its non-empty control also hung — is the evidence for the shared mechanism rather than against it: what has to diverge is the per-rank coords array (coords_array[in_or_not], _function.pyx:1027), and the off-domain fraction is simply another way to make it diverge.

Separation check before fixing (one line): if a hung rank in the tests/parallel/test_0760_swarm_cache_migration.py reproducer owns zero cells rather than merely zero interior points, the symptom is #405 wearing this issue's clothes — test mesh.dm.getHeightStratum(0) empty on the hung rank.

Fix order for whoever picks this up: #405 first (it raises before this path is even reached), then this, then #512's item 1 falls out.

Underworld development team with AI support from Claude Code

[lmoresi]

Superseded by #551 — the mechanism is item 4 of #551 and it is the same locator seen through its cache: the DMLocatePoints call count inside petsc_interpolate is a rank-local function of the rank's coords, via the per-rank coords-hash cache key plus the unreduced _location_capability() that is itself part of that key. #513 was consolidated here first and travels with it.

We are bundling these because they are not independent: point location is a bespoke kd-tree + half-space walk with no cheap rejection, re-run several times per call, not shared between the classifier and the interpolator, and bypassed on the one path where its answer is trusted. #551 carries the full evidence (including a previously unfiled finding — global_evaluate at np=8 is 2.83x slower than serial for identical work, efficiency 0.044, because a point a rank does not own costs 51x the containment work of one it does) and the six-item fix shape. Nothing recorded here is lost; the mechanism and the reproducer travel with it.

Underworld development team with AI support from Claude Code