Document why the multigrid coarse level must be small (#644)

[lmoresi]

Closes the documentation gap agreed on #644 (and its superseded twin #643).

The gap

bknight1 hit an OOM at Pawsey: FMG's coarse solve is redundant+LU, which replicates the coarse factorization on every rank, and his coarsest level still carried 14k velocity DOFs — ~300 MB/rank replicated, killed at the 900 MB/task cap. The conclusion on the issue was that this is not a bug, it needs documenting.

The premise the page never stated is the one that matters: multigrid is cheap because the bottom of the hierarchy is solved exactly at negligible cost, and that is only true if the bottom is genuinely tiny. Nothing in the method makes it so, and nothing warns you when it is not.

So a 14,000-DOF coarsest level is not a coarse-solver problem — it means the hierarchy was built from too fine a base with too few levels, and the "solve the coarse problem for free" premise never held. The fix is another level underneath, not a different bottom solve. Doubling the base cell size and adding one refinement level lands on the same finest grid with every coarser level ~4x smaller in 2-D (8x in 3-D). bknight1 measured exactly that: 683 → 416 MB/rank for +11% linear-solve time, finest grid and SNES iteration sequence unchanged.

This is PETSc's choice, not ours

Worth stating plainly, because the issue reads as though Underworld picked redundant and thereby caused the OOM. PCSetUp_MG selects PCREDUNDANT when the communicator has more than one rank and PCLU when it does not — its own source comment reads "coarse solve is (redundant) LU by default" (src/ksp/pc/impls/mg/mg.c). Confirmed by inspection at np = 1, 2 and 4 on PETSc 3.25.

We set it explicitly only so a bundle applied after another cannot inherit a sibling's leftover options (#468). Deleting the keys and letting PETSc decide would change nothing — which is the useful part, because it puts attention back on the hierarchy.

Two corrections to what was there

• The page claimed the default coarse solve was a bare lu, in both the bundle listing and the "Parallel coarse solve" section, and told users to set redundant themselves for large partitions. It has been the default all along, and a bare lu cannot factor a distributed matrix — it fails at np > 1.
• The SVD path is now documented as what it actually is. It keys on verified rotation null modes, not on a problem being "rotated": a rotated problem with Dirichlet walls or a split-fault contact box has none, and a blanket dense SVD coarse solve accounted for most of ~0.8 s per V-cycle application in Custom-P V-cycles across a cut level cost more than they save (7x slower than GAMG at 11x fewer iterations) #622. Advising svd for anything rotated would have been an expensive mistake.

Also

A snippet to check the hierarchy you actually built (mesh.dm_hierarchy[0] is the coarsest), verified to run, and a rule of thumb — low thousands of DOFs at the bottom; if it reaches five figures, add a level.

docs-build succeeds with no warnings on this page.

Not done here

The size-aware default from the issue's option 1 (warn or switch when the coarsest level is large). That is a code change, and the call on #644 was documentation. It would be a reasonable follow-up: the check is cheap, and the failure it prevents currently presents as an OOM kill with no hint of the cause.

Underworld development team with AI support from Claude Code

[Copilot]

🟢 Approval recommended

The change is documentation-only, technically consistent with the codebase’s stated multigrid settings, and addresses the gap described in #644 without introducing functional risk.

Pull request overview

Updates the advanced multigrid preconditioning documentation to explain why FMG’s coarse level must remain genuinely small in parallel runs, and clarifies PETSc’s default coarse-solver behavior to prevent misdiagnosis of OOM failures as “coarse solver bugs.”

Changes:

• Corrects the documented FMG coarse-solver defaults to mg_coarse_pc_type="redundant" with mg_coarse_redundant_pc_type="lu", and explains why this is PETSc’s default choice in parallel.
• Adds a detailed explanation of the replicated-memory failure mode for large coarse levels (with the Setonix/#644 measurements) and emphasizes that the fix is to deepen the hierarchy, not swap the coarse PC.
• Clarifies when SVD is required (null-space robustness via verified rotation modes), and adds a snippet to inspect the built DM hierarchy.

File summaries

File	Description
docs/advanced/multigrid-preconditioning.md	Documents coarse-level sizing constraints and correct PETSc/Underworld FMG coarse-solver defaults to prevent OOM surprises at scale.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[lmoresi]

Adversarial review

Copilot approved this, but only on the grounds that it is documentation-only. Documentation-only is not the same as harmless: a doc that tells you to run something that cannot work costs more than silence, because you spend the afternoon believing the tool is broken. We reviewed it by running its claims rather than re-reading them. Five findings, two of them wrong advice. All fixed in ffd8920f.

1. The telescope recommendation cannot work at all — and we had promoted it

The page offered mg_coarse_pc_type = "telescope" for large partitions, and our rewrite gave it its own subsection, making it more prominent than before.

PCTELESCOPE has no DMPlex support, and every Underworld mesh is a DMPlex. Running the snippet as written, at np=2:

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Support for DMPLEX is currently not available

PETSc error 56, solve aborted. This was pre-existing text we inherited and endorsed. It is now a warning that the option is unavailable — which actually strengthens the page's thesis, since it means there is no bottom-solve escape hatch from an over-large coarse level. The hierarchy is the only lever.

2. The SVD snippet was inert

We showed:

geometric_mg_bundle(coarse="svd")

That constructs an MGSettings namedtuple and discards it. It applies nothing. It read as a user action and did nothing at all. Removed — the solver already selects SVD from the verified rotation-mode count, which is the accurate story.

3. The "override it yourself" promise is false on exactly the path that needs it

The page promised you can keep preconditioner = "auto" and override the coarse solver through petsc_options. We verified that on the standard path: fieldsplit_velocity_mg_coarse_pc_type = svd is still set after the solve. Good.

It is false on the rotated path — the one path where an SVD coarse solve is the whole point. That route writes its bundle under the velocity sub-PC's own prefix, applies it, then deletes the keys again; anything set through solver.petsc_options[...] lives under the SNES prefix and never reaches it. Now scoped to the standard path, with the rotated exception called out.

4. "Lands on the same finest grid" overclaimed

Measured on a unit box: cellSize=0.10, refinement=2 gives 3,872 finest cells; cellSize=0.20, refinement=3 gives 4,224 — 9% more, not equal, while the coarsest level drops to 27%. #644's production numbers go the other way (227k vs 233k). The base mesh is unstructured and gmsh does not halve cleanly. Now stated as a few percent either side, with both measurements, and a note to check rather than assume.

5. The rule of thumb omitted the two variables that set it

"Low thousands of DOFs" has no rank count and no memory budget in it, yet those are precisely what decide the threshold — the replicated factor is paid per rank against a per-rank cap. Rewritten around that arithmetic and calibrated on #644's own numbers (~14k DOFs → ~300 MB/rank at 5.16x fill).

What survived

Stated here so the review is not only negative — each of these was run, not assumed:

• The mesh.dm_hierarchy diagnostic works on a bare mesh, with no solver or MeshVariable constructed first. The .pyx warns the hierarchy "may not be populated this early", so this was worth checking.
• Per-level growth is exactly 4x in 2-D and 8.00x in 3-D — 66 / 264 / 1056 and 184 / 1472 / 11776 on a refinement=2 box.
• PETSc's own source selects PCREDUNDANT above one rank and PCLU at one, confirmed at np = 1, 2, 4.

Underworld development team with AI support from Claude Code