FMG + redundant coarse solver: replicated memory scales with base mesh size, causing OOM at HPC scale

[bknight1]

Problem

When using FMG with the default redundant coarse solver (pc_type=redundant + pc_factor_mat_solver_type=lu), the coarse-grid LU factorization is replicated on every MPI rank. For models with large base meshes (e.g., 10 km cells, 800x230 km domain), the coarsest level still has ~14k velocity DOFs, causing:

• ~300 MB/rank replicated memory from the coarse LU factorization (fill ratio 5.16× on 14k DOFs)
• Total ~9 GB equivalent replicated across 30 ranks
• OOM kills at 900 MB/rank cap on Setonix/Pawsey

Root Cause

The multigrid_options.py default geometric_mg_bundle() sets mg_coarse_pc_type=redundant (see multigrid_options.py:262). This is documented as "(small) coarse system" but fails when base mesh resolution is high.

Related: PR #213 (_use_iterative_solver) already avoids MUMPS in parallel for the same reason, using redundant+svd instead — but svd is infeasible at this scale (dense 14k×14k).

Solution Found

Using a finer base mesh with more refinement levels keeps the finest grid identical while shrinking the coarsest level:

# Original (OOM)
-uw_csize_km 10 -uw_refinement 2   # 14k coarse DOFs → 300 MB/rank

# Fixed (fits in 900 MB cap)  
-uw_csize_km 20 -uw_refinement 3   # 1.9k coarse DOFs → 100 MB/rank

Config	Coarsest vDOFs	Linear solve time (serial)	Peak RSS/rank (np=30)
csize10 ref2	14,195	~100 s	683 MB ❌
csize20 ref3	~1,900	~111 s (+11%)	416 MB ✅

Finest grid DOFs nearly identical (233k vs 227k). Linear solve time increases ~11%.

Proposed Fix Options

1. Size-aware default in multigrid_options.py (preferred):

   • Query coarsest-level DOFs from mesh.dm_hierarchy
   • Use redundant+lu only if coarsest DOFs < threshold (~5k)
   • Above threshold: gamg or asm (distributed, no replication)

2. Document the workaround (immediate):

   • Add csize20 ref3 pattern to docs/examples
   • Warn about redundant memory scaling in docstrings

3. Expose threshold via parameter (user control):

   mg_coarse_pc_type = "redundant" if coarse_dofs < 5000 else "gamg"

Related Issues

• PR Parallel adaptive seam-spike fix: mover heap corruption + locator hardening + remesh field-transfer redesign (Phase 1 + 2) #213: Fixed _use_iterative_solver to avoid MUMPS in parallel (same root cause)

Verification

Confirmed on Setonix (Pawsey):

• Control run (csize10 ref2): 683 MB/rank, OOM at 900 MB cap
• Treatment (csize20 ref3): 416 MB/rank, 576 MB peak, clean completion

Fine grid DOFs: 233k vs 227k (identical for science). Linear solve time: 100s vs 111s (+11%). Memory: 683 MB → 416 MB/rank.

[bknight1]

Superseded by #644 (same content, corrected timing table — the original body here mixed np=2 and serial timings). Please use #644.