Fix BdIntegral parallel hang: coordinate DM label contamination by lmoresi · Pull Request #115 · underworldcode/underworld3

lmoresi · 2026-04-15T15:07:48Z

Summary

Fixes #96 — BdIntegral.evaluate() followed by solver.solve() deadlocks in parallel.

Root cause: DMSetCoordinateDisc (called during mesh init) clears the coordinate field cache. When DMPlexComputeBdIntegral later lazily recreates the coordinate field via DMClone, it copies ALL labels from mesh.dm to the coordinate DM — including boundary labels. DMCompleteBCLabels_Internal then fails with MPI errors on these inherited labels.
Fix: New C wrapper UW_DMForceCoordinateField forces coordinate field creation and strips non-essential labels from the coordinate DM after createCoordinateSpace. Also replaces setCoordinateDisc with createCoordinateSpace (avoids a separate PETSc null-subspace bug).
PETSc upstream bug filed: DMSetCoordinateDisc with user FE leaves broken dual space subspaces. Pure C reproducer in petsc-custom/bug-reports/.

Changes

petsc_compat.h: Add UW_DMForceCoordinateField — forces coord field creation + strips boundary labels from coord DM
petsc_maths.pyx: Python wrapper dm_force_coordinate_field()
discretisation_mesh.py:
- _from_plexh5: Use createFromFile instead of topologyLoad+coordinatesLoad+labelsLoad
- nuke_coords_and_rebuild: createCoordinateSpace instead of setCoordinateDisc, call dm_force_coordinate_field after
- Skip copyDS self-copy when hierarchy is trivial

Test plan

Serial: 21/21 test_0502_boundary_integrals.py pass
Parallel (np=2): 3/3 test_0765_internal_boundary_integral_mpi.py pass
Parallel: solve + BdIntegral + solve + BdIntegral + solve (simplex and quad meshes)
Full test suite (./uw test)
Stokes solver with BdIntegral

Underworld development team with AI support from Claude Code

…ble installs Build system hardening after editable install contamination caused hours of debugging (debug PETSc .so files leaked into optimised environment via shared source tree and stale .pth files). uw (build script): - Use pip --target to install where Python actually looks, not where pip resolves through worktree symlinks - Post-install verification: confirm import underworld3 works after build - Clean stale editable .pth files and source-tree .so before each build - Remove .pixi symlink fallback in worktree create — fail hard instead of creating a broken shared environment pixi.toml: - Add amr-debug environment (petsc-4-uw-openmpi-debug) for isolated debug PETSc investigation without contaminating the optimised build CLAUDE.md: - Add "NEVER Use Editable Installs" section with recovery instructions - Update worktree docs: each worktree has own pixi env (not shared) Underworld development team with AI support from Claude Code

Root cause: DMClone (inside createCoordinateSpace) copies ALL labels from mesh.dm to the coordinate DM — including boundary labels added by UW's mesh init. When DMPlexComputeBdIntegral later lazily recreates the coordinate field, DMCompleteBCLabels_Internal fails with MPI errors on these inherited boundary labels, causing a deadlock or segfault. Fix: - Add UW_DMForceCoordinateField (petsc_compat.h): forces coordinate field creation and strips non-essential labels from the coordinate DM. Called after createCoordinateSpace in nuke_coords_and_rebuild so the coordinate field is cached clean before boundary labels can contaminate lazy recreation. - Replace setCoordinateDisc with createCoordinateSpace: the former leaves the coordinate dual space without point subspaces (separate PETSc bug causing null pointer in PetscFECreatePointTrace). - Use createFromFile instead of topologyLoad+coordinatesLoad+labelsLoad in _from_plexh5 for more robust HDF5 mesh loading. - Skip copyDS self-copy when dm_hierarchy is trivial. Also found (for PETSc upstream report): - DMSetCoordinateDisc with user FE: broken dual space subspaces - DMCompleteBCLabels_Internal: fails on coordinate DMs with inherited boundary labels from DMClone - Pure C reproducer: /tmp/petsc_bdint_coorddisc.c + issue96_box.msh Closes #96 Underworld development team with AI support from Claude Code

Copilot

Pull request overview

Fixes the parallel deadlock triggered by calling BdIntegral.evaluate() before a subsequent solver.solve() by preventing PETSc’s lazy coordinate-field recreation from inheriting boundary labels onto the coordinate DM (which later trips DMCompleteBCLabels_Internal in MPI).

Changes:

Add a PETSc C helper (UW_DMForceCoordinateField) and Python wrapper to force coordinate field creation and remove non-essential labels from the coordinate DM.
Update mesh coordinate rebuild to use createCoordinateSpace(...) and immediately “sanitize” the coordinate DM via the new helper; also avoid a trivial copyDS self-copy.
Adjust tooling/environment management (worktree isolation behavior, build install target selection) and add an amr-debug Pixi environment.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`uw`	Installs into the Python-discovered site-packages path and cleans editable-install artifacts to improve worktree/env isolation.
`src/underworld3/discretisation/discretisation_mesh.py`	Switch HDF5 load path to `createFromFile`; rebuild coordinate space via `createCoordinateSpace` and force coordinate-field creation/label stripping; skip trivial `copyDS` self-copy.
`src/underworld3/cython/petsc_maths.pyx`	Adds `dm_force_coordinate_field(dm)` Python wrapper calling the new PETSc helper.
`src/underworld3/cython/petsc_extras.pxi`	Exposes `UW_DMForceCoordinateField` to Cython.
`src/underworld3/cython/petsc_compat.h`	Implements `UW_DMForceCoordinateField`: forces coord-field creation and removes most labels from the coordinate DM.
`pixi.toml`	Adds `amr-debug` feature/environment configuration for a debug PETSc build.
`pixi.lock`	Lockfile updated to include the new environment and associated resolution changes.
`CLAUDE.md`	Updates worktree guidance to reflect per-worktree Pixi environments and warns against editable installs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-15T15:13:38Z

        return h5plex
    else:
-        return sf0, h5plex
+        return None, h5plex


_from_plexh5(..., return_sf=True) now returns (None, h5plex). Since Mesh.__init__ stores this into self.sf0, this effectively makes return_sf a no-op and can break any code expecting a valid SF from the HDF5 load path. Consider either (a) returning an appropriate SF from the new createFromFile pathway (if one exists), or (b) removing return_sf and updating callers to not expect an SF value.

Suggested change

return None, h5plex

sf = h5plex.getPointSF()

return sf, h5plex

Copilot · 2026-04-15T15:13:38Z

+            # Use createCoordinateSpace rather than setCoordinateDisc.
+            # setCoordinateDisc with a user-created FE leaves the coordinate
+            # dual space without proper point subspaces, causing
+            # DMPlexComputeBdIntegral to segfault/deadlock (issue #96).
+            # createCoordinateSpace builds the FE internally with correct
+            # subspace initialisation.
+            self.dm.createCoordinateSpace(self.degree, False, True)
+


In the PETSc >3.20.5 path, createCoordinateSpace(...) is used and the previously-created self.petsc_fe is no longer referenced in this method. If it is truly only needed for the <=3.20.5 projectCoordinates() branch, consider creating it conditionally (or documenting why it must still be constructed) to avoid unnecessary FE creation and reduce confusion about which coordinate discretisation is actually in use.

…_plexh5 - Add hasattr guard for createCoordinateSpace (not in all petsc4py versions) - Fall back to setCoordinateDisc for PETSc builds without createCoordinateSpace - Return h5plex.getPointSF() instead of None from _from_plexh5 (Copilot review) Underworld development team with AI support from Claude Code

gthyagi · 2026-04-16T03:31:06Z

@lmoresi, I’m also observing significant runtime (or apparent stalling) in boundary integral computations for spherical benchmarks at 1000+ CPUs. The Stokes solve completes in ~20 minutes, but the total script runtime exceeds 3 hours (mostly computing integrals).

Since you’re already looking into this issue, it would be very helpful to also identify and expose more efficient and scalable approaches for computing boundary integrals.

Thanks.

lmoresi · 2026-04-16T19:24:12Z

@gthyagi ... thank you for reporting this. The fix in this PR addresses a correctness bug (the parallel deadlock), not the performance issue you're seeing, so the scaling problem is a separate concern.

To help us diagnose where the time is going at 1000+ CPUs, here are some profiling options:

1. UW3 built-in timing

import underworld3 as uw
uw.timing.start()

# ... your simulation with BdIntegral calls ...

uw.timing.print_table()             # Full PETSc log view
uw.timing.print_summary()           # UW3-focused summary
uw.timing.print_table("timing.csv") # Save for analysis

This will show BdIntegral.evaluate and BdIntegral.__init__ as named events, plus all PETSc operations underneath (VecScatter, MatAssembly, etc.).

2. PETSc command-line profiling (more MPI detail)

mpirun -np 1024 python script.py -log_view

This gives per-event breakdown including DMPlexIntegralFEM and all MPI communication stats (message counts, volumes, wait times). Save to file with:

mpirun -np 1024 python script.py -log_view :timing.csv:ascii_csv

3. Isolate BdIntegral into its own PETSc stage

from petsc4py import PETSc
stage = PETSc.Log.Stage("BdIntegral")

stage.push()
result = bd_integral.evaluate()
stage.pop()

This separates BdIntegral timing into its own stage in -log_view output so it doesn't mix with solver stats.

What to look for

The most likely bottleneck at 1000+ CPUs is MPI communication: DMPlexComputeBdIntegral does a DMGlobalToLocal internally, and our wrapper adds an Allreduce. The -log_view output will show how much time is in communication vs computation — that will tell us whether this is a PETSc scalability issue or something on our side.

The JIT compilation is cached across calls, so repeated evaluate() with the same expression doesn't recompile. The actual integration + MPI reduction must run every call since field data changes.

Could you raise this as a new issue with the timing output attached? That way we can track the performance investigation separately from this correctness fix.

jcgraciosa · 2026-04-17T14:12:29Z

@lmoresi, I've tested the fix by running the test code with two processors, but I now get some PETSc-related errors in addition to hanging:

[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: General MPI error
[1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer
[1]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
[1]PETSC ERROR:   Option left: name:-dm_plex_gmsh_mark_vertices value: true source: code
[1]PETSC ERROR:   Option left: name:-dm_plex_gmsh_multiple_tags value: true source: code
[1]PETSC ERROR:   Option left: name:-dm_plex_gmsh_use_regions (no value) source: code
[1]PETSC ERROR:   Option left: name:-dm_plex_hash_location (no value) source: code
[1]PETSC ERROR:   Option left: name:-options_left value: 0 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_pressure_ksp_rtol value: 1e-07 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_pressure_ksp_type value: fgmres source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_pressure_pc_type value: gasm source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_ksp_rtol value: 3.3e-08 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_ksp_type value: cg source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_mg_levels_ksp_converged_maxits (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_mg_levels_ksp_max_it value: 3 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_gamg_agg_nsmooths value: 2 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_gamg_repartition value: true source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_gamg_type value: agg source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_mg_type value: additive source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_type value: gamg source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_ksp_atol value: 1e-12 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_diag_use_amat (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_off_diag_use_amat (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_schur_fact_type value: full source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_schur_precondition value: a11 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_type value: schur source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_type value: fieldsplit source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_converged_reason (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_ksp_ew (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_ksp_ew_version value: 3 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_rtol value: 1e-06 source: code
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: PETSc Release Version 3.25.0, unknown
[1]PETSC ERROR: /Users/jgra0019/Documents/codes/uw3-dev/Conv-TALA-EBA-benchmark/src-retest-2026/boundary_integral_simple.py with 2 MPI process(es) and PETSC_ARCH petsc-4-uw-openmpi on Juans-MacBook-Pro.local by jgra0019 Sat Apr 18 00:07:01 2026
[1]PETSC ERROR: Configure options: --with-petsc-arch=petsc-4-uw-openmpi --with-debugging=0 --with-pragmatic=1 --with-x=0 --download-eigen=1 --download-metis=1 --download-mmg=1 --download-mmg-cmake-arguments="-DMMG_INSTALL_PRIVATE_HEADERS=ON -DUSE_SCOTCH=OFF" --download-mumps=1 --download-parmetis=1 --download-parmmg=1 --download-pragmatic=1 --download-ptscotch=/Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/patches/scotch-7.0.10-c23-fix.tar.gz --download-scalapack=1 --download-slepc=1 --with-mpi-dir=/Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/.pixi/envs/amr-dev --with-hdf5=1 --with-hdf5-dir=/Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/.pixi/envs/amr-dev --download-hdf5=0 --download-mpich=0 --download-openmpi=0 --download-mpi4py=0 --download-bison --with-petsc4py=0
[1]PETSC ERROR: #1 PetscCommBuildTwoSided_Allreduce() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/sys/utils/mpits.c:166
[1]PETSC ERROR: #2 PetscCommBuildTwoSided() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/sys/utils/mpits.c:273
[1]PETSC ERROR: #3 PetscSFSetUp_Basic() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/impls/basic/sfbasic.c:203
[1]PETSC ERROR: #4 PetscSFSetUp() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:292
[1]PETSC ERROR: #5 PetscSFReduceBegin() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:1589
[1]PETSC ERROR: #6 PetscSFGetMultiSF() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:1291
[1]PETSC ERROR: #7 PetscSFGatherBegin() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:1908
[1]PETSC ERROR: #8 DMLabelGather() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/label/dmlabel.c:2152
[1]PETSC ERROR: #9 DMPlexLabelComplete_Internal() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/impls/plex/plexsubmesh.c:251
[1]PETSC ERROR: #10 DMPlexLabelComplete() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/impls/plex/plexsubmesh.c:301
[1]PETSC ERROR: #11 DMAddBoundary() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/interface/dm.c:8138

I'm not entirely sure what is happening.
I'll try to create a pure petsc4py/PETSc version of the test code (BD integral then Stokes solve) to investigate.

lmoresi · 2026-04-17T17:10:52Z

@jcgraciosa — your error trace:

DMAddBoundary → DMPlexLabelComplete → DMLabelGather → PetscSFGatherBegin → MPI_ERR_BUFFER

is exactly the coordinate DM label contamination that this PR fixes. Without this fix, the coordinate DM inherits boundary labels from DMClone during lazy coordinate field recreation. When PETSc then calls DMPlexLabelComplete (inside DMAddBoundary during solver setup), those inherited labels reference mesh points that the coordinate DM doesn't own, causing the SF to fail.

This is not PETSc 3.25.0-specific — the same issue exists on 3.24.x but manifests differently (deadlock vs MPI error depending on debug mode and timing).

Could you try again once this PR is merged? The fix should resolve both the hang and the MPI error you're seeing.

lmoresi · 2026-04-17T21:13:08Z

@jcgraciosa — the fix for your issue is now on development. Two PRs were needed:

Fix BdIntegral parallel hang: coordinate DM label contamination #115 (this PR): Fixes the coordinate DM label contamination that causes the MPI error in DMAddBoundary / DMPlexLabelComplete
Build cache coherence across PETSc version switches #120 (just merged): Fixes the build system so that switching PETSc versions actually takes effect — without this, setup.py silently compiled UW3 against the old conda PETSc even when PETSC_DIR/PETSC_ARCH pointed to a custom build

To update, from your underworld3 repo:

# 1. Pull latest development
git checkout development
git pull

# 2. If switching PETSc versions (e.g. to 3.25.0):
./uw petsc switch v3.25.0    # checks out tag, clears stale build artifacts
./uw petsc build              # builds PETSc + petsc4py for the new version

# 3. Clean rebuild of underworld3
rm -rf build/                 # ensure no stale .so from previous PETSc
./uw build

# 4. Verify the correct PETSc is linked
pixi run -e amr-dev python -c "
from petsc4py import PETSc
print(f'PETSc: {PETSc.Sys.getVersion()}')
"

The key verification step: check that the compiled .so links to the right libpetsc:

otool -L .pixi/envs/amr-dev/lib/python3.12/site-packages/underworld3/cython/petsc_maths.cpython-312-darwin.so | grep petsc

This should show the PETSc version you built (e.g. libpetsc.3.25.dylib), not the conda one (libpetsc.3.24.dylib). If it shows the wrong version, the stale build cache wasn't cleared — rm -rf build/ and ./uw build again.

We've verified this end-to-end: Stokes solve + BdIntegral + second Stokes solve passes in parallel (mpirun -np 2) on PETSc 3.25.0.

jcgraciosa · 2026-04-21T03:33:38Z

I followed the steps outlined above, but I still can't verify the fix. :(

Verification of development version

$ git checkout development
$ git pull
$ git log --oneline -5 
403dd4fb (HEAD -> development, origin/development) Merge pull request #120 from underworldcode/feature/pixi-petsc-arch-static
15f8e1d8 Address Copilot review feedback on PR #120
61675b59 setup.py: prefer PETSC_DIR/PETSC_ARCH env vars over petsc4py.get_config()
9b7c359a Nuke stale build artifacts and petsc4py on ./uw petsc switch
499f8132 Make pixi.toml static; derive PETSC_ARCH via activation script

Verification using ./uw doctor

$./uw doctor
(base) Juans-MacBook-Pro:underworld3 jgra0019$ ./uw doctor

============================================================
  Underworld3 Diagnostics
============================================================

Environment: amr-dev

  Checking pixi environment... [OK] amr-dev installed
  Checking custom PETSc... [OK] Custom PETSc built
  Checking petsc4py... [OK] petsc4py available
  Checking underworld3... [OK] underworld3 importable

  Running detailed diagnostics...
  
  ============================================================
    Underworld3 Diagnostics
  ============================================================
  
    Checking PETSc version... [OK] PETSc 3.25.0
    Checking PETSc version match... [OK] Compile/runtime versions match (3.25)
    Checking extension modules... [OK] All 6 extensions loaded
    Checking MPI configuration... [OK] MPI working (rank 0 of 1)
    Checking environment variables... [INFO] 4 relevant variables set
    Checking PETSc library match... [OK] PETSc libraries match
    Testing function evaluation... Structured box element resolution 2 2
  [OK] Basic evaluation working
  
  ------------------------------------------------------------
    [OK] All checks passed!
  ------------------------------------------------------------
  

------------------------------------------------------------
  [OK] All checks passed
------------------------------------------------------------

Verify that correct PETSc is linked

 pixi run -e amr-dev python -c "
> from petsc4py import PETSc
> print(f'PETSc: {PETSc.Sys.getVersion()}')
> "
PETSc: (3, 25, 0)

check that the compiled .so links to the right libpetsc:

$ otool -L .pixi/envs/amr-dev/lib/python3.12/site-packages/underworld3/cython/petsc_maths.cpython-312-darwin.so | grep petsc
.pixi/envs/amr-dev/lib/python3.12/site-packages/underworld3/cython/petsc_maths.cpython-312-darwin.so:
	/Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/petsc-325-uw-openmpi/lib/libpetsc.3.25.dylib (compatibility version 3.25.0, current version 3.25.0)

Even with the rebuild, I still get PETSc errors and solver hang when doing a Stokes solve() after calculating a boundary integral. The log is shown below:

[0] Initial boundary integral ...
[0] boundary integral result: 0.9999999999999997
[0] Solving stokes ...
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: General MPI error
[1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer
[1]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
[1]PETSC ERROR:   Option left: name:-dm_plex_gmsh_mark_vertices value: true source: code
[1]PETSC ERROR:   Option left: name:-dm_plex_gmsh_multiple_tags value: true source: code
[1]PETSC ERROR:   Option left: name:-dm_plex_gmsh_use_regions (no value) source: code
[1]PETSC ERROR:   Option left: name:-dm_plex_hash_location (no value) source: code
[1]PETSC ERROR:   Option left: name:-options_left value: 0 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_pressure_ksp_rtol value: 1e-07 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_pressure_ksp_type value: fgmres source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_pressure_pc_type value: gasm source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_ksp_rtol value: 3.3e-08 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_ksp_type value: cg source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_mg_levels_ksp_converged_maxits (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_mg_levels_ksp_max_it value: 3 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_gamg_agg_nsmooths value: 2 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_gamg_repartition value: true source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_gamg_type value: agg source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_mg_type value: additive source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_fieldsplit_velocity_pc_type value: gamg source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_ksp_atol value: 1e-12 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_diag_use_amat (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_off_diag_use_amat (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_schur_fact_type value: full source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_schur_precondition value: a11 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_fieldsplit_type value: schur source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_pc_type value: fieldsplit source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_converged_reason (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_ksp_ew (no value) source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_ksp_ew_version value: 3 source: code
[1]PETSC ERROR:   Option left: name:-Solver_6_snes_rtol value: 1e-06 source: code
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: PETSc Release Version 3.25.0, unknown
[1]PETSC ERROR: boundary_integral_simple.py with 2 MPI process(es) and PETSC_ARCH petsc-325-uw-openmpi on Juans-MacBook-Pro.local by jgra0019 Tue Apr 21 13:06:33 2026
[1]PETSC ERROR: Configure options: --with-petsc-arch=petsc-325-uw-openmpi --with-debugging=0 --with-pragmatic=1 --with-x=0 --download-eigen=1 --download-metis=1 --download-mmg=1 --download-mmg-cmake-arguments="-DMMG_INSTALL_PRIVATE_HEADERS=ON -DUSE_SCOTCH=OFF" --download-mumps=1 --download-parmetis=1 --download-parmmg=1 --download-pragmatic=1 --download-ptscotch=/Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/patches/scotch-7.0.10-c23-fix.tar.gz --download-scalapack=1 --download-slepc=1 --with-mpi-dir=/Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/.pixi/envs/amr-dev --with-hdf5=1 --with-hdf5-dir=/Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/.pixi/envs/amr-dev --download-hdf5=0 --download-mpich=0 --download-openmpi=0 --download-mpi4py=0 --download-bison --with-petsc4py=0
[1]PETSC ERROR: #1 PetscCommBuildTwoSided_Allreduce() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/sys/utils/mpits.c:166
[1]PETSC ERROR: #2 PetscCommBuildTwoSided() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/sys/utils/mpits.c:273
[1]PETSC ERROR: #3 PetscSFSetUp_Basic() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/impls/basic/sfbasic.c:203
[1]PETSC ERROR: #4 PetscSFSetUp() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:292
[1]PETSC ERROR: #5 PetscSFReduceBegin() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:1589
[1]PETSC ERROR: #6 PetscSFGetMultiSF() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:1291
[1]PETSC ERROR: #7 PetscSFGatherBegin() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/vec/is/sf/interface/sf.c:1908
[1]PETSC ERROR: #8 DMLabelGather() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/label/dmlabel.c:2152
[1]PETSC ERROR: #9 DMPlexLabelComplete_Internal() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/impls/plex/plexsubmesh.c:251
[1]PETSC ERROR: #10 DMPlexLabelComplete() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/impls/plex/plexsubmesh.c:301
[1]PETSC ERROR: #11 DMAddBoundary() at /Users/jgra0019/Documents/codes/uw3-dev/uw3-code-dev/uw3-dev-2026/underworld3/petsc-custom/petsc/src/dm/interface/dm.c:8138
^C^CAbort is in progress...hit ctrl-c again to forcibly terminate

Perhaps @NengLu or @gthyagi can also try to verify the fix. Please.

lmoresi · 2026-04-21T05:41:29Z

@jcgraciosa — thanks for the thorough verification. The build is correct (3.25.0 linked properly), so this is a genuine bug that our simple test case doesn't trigger.

Your error trace is in DMAddBoundary during solver setup after the BdIntegral call — same call chain as before, but something about your mesh or solver configuration exposes a code path our UnstructuredSimplexBox test doesn't hit.

Could you share the test script (boundary_integral_simple.py) you're running? We need to reproduce it on our end. Even a minimal version that loads the mesh and sets up the Stokes solver + BdIntegral would help.

Key questions:

Is the mesh loaded from a .msh file (gmsh) or generated by PETSc?
Does the error happen on the first stokes.solve() after BdIntegral.evaluate(), or only on a subsequent one?
Does it fail with mpirun -np 2 on a simple PETSc-generated mesh (e.g. uw.meshing.UnstructuredSimplexBox)?

That last point would tell us whether the issue is mesh-specific or general.

lmoresi · 2026-04-21T06:54:05Z

@jcgraciosa — we've identified the root cause and have a fix in PR #125.

The issue is deeper than the coordinate DM labels we fixed in #115. DMPlexComputeBdIntegral lazily initialises height-trace FE caches on the coordinate DMField, which is shared by refcount between mesh.dm and all solver DMs (via DMClone). This corrupts the solver's Jacobian assembly. The bug is mesh-partition dependent — it only triggers when boundary faces span multiple MPI ranks, which is why our small test mesh (cellSize=0.1) passed but yours failed.

The fix: BdIntegral.evaluate() now runs the PETSc integral on a disposable sandbox DM with its own coordinate field caches, leaving the original mesh DM clean.

Could you test PR #125 on your benchmark?

git fetch origin
git checkout bugfix/bdintegral-sandbox-dm
rm -rf build/
./uw build
pixi run -e amr-dev mpirun -np 2 python boundary_integral_simple.py

jcgraciosa · 2026-04-22T01:09:04Z

@lmoresi, thank you for the fix!
I've tested PR #125 on a cartesian and annulus mesh and can confirm that it is fixed (solver does not hang and no PETSc errors).
I've also tested a longer sequence (i.e., bdIntegral -> Stokes solve -> bdIntegral -> Stoke solve) together with higher resolution and procs up to 8, and did not encounter any hang or PETSc error.

lmoresi added 2 commits April 13, 2026 08:21

lmoresi requested review from Copilot and jcgraciosa April 15, 2026 15:07

Copilot started reviewing on behalf of lmoresi April 15, 2026 15:08 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

lmoresi mentioned this pull request Apr 16, 2026

Parallel bug - Calculating a boundary integral causes the next Stokes solve to hang #96

Closed

lmoresi mentioned this pull request Apr 17, 2026

Add PETSc version switching infrastructure #117

Merged

4 tasks

lmoresi merged commit f3debde into development Apr 17, 2026
1 check passed

lmoresi mentioned this pull request Apr 17, 2026

Build cache coherence across PETSc version switches #120

Merged

7 tasks

lmoresi mentioned this pull request Apr 21, 2026

BdIntegral: sandbox DM to prevent shared DM_Plex cache corruption #125

Merged

4 tasks

lmoresi deleted the bugfix/issue-96-bdintegral-parallel-hang branch June 13, 2026 00:48

Navigation Menu

Uh oh!

Fix BdIntegral parallel hang: coordinate DM label contamination - #115

Fix BdIntegral parallel hang: coordinate DM label contamination#115
lmoresi merged 3 commits into
developmentfrom
bugfix/issue-96-bdintegral-parallel-hang

lmoresi commented Apr 15, 2026 •

edited by jcgraciosa

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

gthyagi commented Apr 16, 2026

Uh oh!

lmoresi commented Apr 16, 2026 •

edited

Loading

Uh oh!

jcgraciosa commented Apr 17, 2026

Uh oh!

lmoresi commented Apr 17, 2026

Uh oh!

Uh oh!

lmoresi commented Apr 17, 2026

Uh oh!

jcgraciosa commented Apr 21, 2026

Uh oh!

lmoresi commented Apr 21, 2026

Uh oh!

lmoresi commented Apr 21, 2026

Uh oh!

jcgraciosa commented Apr 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	return None, h5plex
	sf = h5plex.getPointSF()
	return sf, h5plex

Uh oh!

Conversation

lmoresi commented Apr 15, 2026 • edited by jcgraciosa Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

gthyagi commented Apr 16, 2026

Uh oh!

lmoresi commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. UW3 built-in timing

2. PETSc command-line profiling (more MPI detail)

3. Isolate BdIntegral into its own PETSc stage

What to look for

Uh oh!

jcgraciosa commented Apr 17, 2026

Uh oh!

lmoresi commented Apr 17, 2026

Uh oh!

Uh oh!

lmoresi commented Apr 17, 2026

Uh oh!

jcgraciosa commented Apr 21, 2026

Uh oh!

lmoresi commented Apr 21, 2026

Uh oh!

lmoresi commented Apr 21, 2026

Uh oh!

jcgraciosa commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lmoresi commented Apr 15, 2026 •

edited by jcgraciosa

Loading

lmoresi commented Apr 16, 2026 •

edited

Loading

jcgraciosa commented Apr 22, 2026 •

edited

Loading