The co-located multigrid tail fails at np=3 on the 3-D width network: a Galerkin coarse operator with a missing diagonal, and a 10% slip error

[lmoresi]

Found while measuring the placement gather (#670) on the crossing-patches fixture (tests/parallel/ptest_0863, split realisation, linear viscous + contact). np=2 and np=4 reproduce the serial slips to five digits; np=3 does not, in two different ways depending on the partition.

With the placement gather as on development (the wide mark of #670), np=3 completes the solve but the smallest piece is 10% low:

AssertionError: Cross_2: parallel peak 0.0027 vs serial 0.00295

Main and Cross_1 pass. The velocity block is custom-FMG, as asserted.

With the tightened gather mark (branch feature/placement-local-gather, which moves about half as many cells and so changes the partition of the fine level), np=3 fails in the multigrid setup on rank 2, reproducibly:

[2]PETSC ERROR: Argument out of range
[2]PETSC ERROR: New nonzero at (1497,1497) caused a malloc.
#3 MatDiagonalSet_Default ... #8 MatGalerkin ... #9 PCSetUp_MG

A Galerkin coarse operator with no diagonal entry means a coarse degree of freedom that no fine row maps to, i.e. a zero column in the transfer that the parallel repair (_repair_zero_columns_parallel) did not catch, or a coarse row that is owned on a rank with no fine image on that rank. The serial path repairs such columns; the parallel guard is the piece the design note (fault-patch-multigrid-2026-08.md, parallel rule 5) lists as needing a parallel form.

Both symptoms are in the co-located tail (#662) at three ranks, not in the placement. np=3 was the rank count the design rule said to test, and neither ptest_0863 nor the other network ptests run at np=3 in CI (they are not in scripts/test.sh's globs at all).

Underworld development team with AI support from Claude Code

[lmoresi]

Second occurrence, np=4, on a different layout: two faults in one network (split realisation) on a [-0.5, 1.5]^3 base, band gathered onto rank 2 with 12,803 of 28,656 cells. Same signature, on rank 0 this time:

[0]PETSC ERROR: New nonzero at (1599,1599) caused a malloc.
#3 MatDiagonalSet_Default ... #6 MatDiagonalSet_MPIAIJ

The crossing-patches fixture passes at np=4 with the band on rank 3, so the defect is partition-dependent rather than tied to np=3. The placement and the split complete; the failure is in the multigrid setup of the co-located tail on the placed mesh.

Underworld development team with AI support from Claude Code

[lmoresi]

Third occurrence, and the clearest: two faults a domain apart in one network at np=2 on a [-0.5, 1.5]^3 base, with the per-region placement of #672. Both regions were already interior to their ranks, nothing was moved, the mesh is balanced (14,339 / 14,332 cells) and the band is genuinely distributed (347 / 345 band cells). The contact solve then fails in PCSetUp_MG with the same signature (New nonzero at (564,564) in MatDiagonalSet under MatGalerkin).

So the defect is not np=3, nor a rank-count: the co-located tail assumes the band's fine cells sit on one rank, and any layout where the placed region spans ranks produces a coarse degree of freedom with no fine image on its owner. That is now the blocking item for a balanced parallel fault network.

Underworld development team with AI support from Claude Code

[lmoresi]

Diagnosis and partial fix, on PR #672 (fourth commit):

The crash (New nonzero at (i,i) caused a malloc in MatDiagonalSet under MatGalerkin / PCSetUp_MG) is PETSc repairing a zero row of the Galerkin coarse operator with an identity diagonal that was never allocated. The zero row came from the rotated prolongation zeroing the wrong rows: _build_rotated_custom_Pl enumerated this rank's velocity-IS entries and passed the LOCAL block indices to zeroRows, which takes global ones. Every rank but the first zeroed rows belonging to rank 0. Fixed by adding the ownership offset.

The non-convergence that followed (velocity KSP reason -11 at iteration 0, |r| = inf) was the cross_partition="auto" rule: on a band spanning two ranks the co-partitioned transfer between the base and the placed mesh had 3 orphan columns, the cross-partition rebuild 16,791 of 96,009 (43,047 at np=4), and the repair injected nearest fine DOFs into all of them. The rule now keeps whichever transfer has fewer orphans and warns when the rebuild would have been worse. The cross-partition builder itself, which loses 17 to 45 percent of the coarse columns on a pair that is actually co-partitioned, is still defective and not fixed.

Still open: the np=3 answer on the crossing fixture (Cross_2 slip 0.00266 vs 0.00295 serial, 10% low). It no longer crashes; it is a wrong number, present before and after both fixes.

Underworld development team with AI support from Claude Code