Skip to content

petsc-custom: add internal-boundary ownership patch, mpi test, and docs - #78

Merged
lmoresi merged 2 commits into
underworldcode:feature/boundary-integralsfrom
gthyagi:feature/boundary-integrals
Mar 12, 2026
Merged

petsc-custom: add internal-boundary ownership patch, mpi test, and docs#78
lmoresi merged 2 commits into
underworldcode:feature/boundary-integralsfrom
gthyagi:feature/boundary-integrals

Conversation

@gthyagi

@gthyagi gthyagi commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Problem

Internal-boundary assembly in parallel was rank-dependent for PETSc DMPlex boundary workflows used by UW3 (natural BC / boundary-integral paths). This showed up as branch-split benchmark errors (notably at higher MPI counts) even for identical physics/input.

Root Cause

  • Boundary facet ownership was not handled consistently across all boundary assembly paths.
  • DMPlexComputeBdResidualSingleByKey used support[0] unconditionally instead of support[key.part].
  • For part > 0, points with insufficient support were not filtered/guarded consistently.

What This PR Adds

  • New PETSc patch:
    • petsc-custom/patches/plexfem-internal-boundary-ownership-fix.patch
  • Patch documentation:
    • petsc-custom/patches/PETSC_MR_DESCRIPTION_INTERNAL_BOUNDARY_OWNERSHIP.md
  • PETSc build integration:
    • petsc-custom/build-petsc.sh now prefers the new patch, with legacy fallback to the older patch if the new file is absent.
  • Regression test:
    • tests/parallel/test_0765_internal_boundary_integral_mpi.py
  • Documentation updates:
    • petsc-custom/README.md
    • CHANGES.md

PETSc Fix Summary

In src/dm/impls/plex/plexfem.c:

  • Filter SF leaf (ghost) facets in:
    • DMPlexComputeBdIntegral
    • DMPlexComputeBdResidual_Internal
    • DMPlexComputeBdJacobian_Internal
  • In DMPlexComputeBdResidualSingleByKey:
    • filter invalid points for key.part > 0
    • use support[key.part] for closure gather/scatter
    • add support-size guards before part access

Validation

  • MPI regression test passes:
    • mpirun -np 4 python -m pytest --with-mpi tests/parallel/test_0765_internal_boundary_integral_mpi.py -q
  • Benchmark validation (Ex_Stokes_Kramer_latest.py, case1, natural) is stable across np=1,2,4,8.

Patch Application Guidance

  • Use one patch only.
  • Preferred: plexfem-internal-boundary-ownership-fix.patch
  • Legacy fallback: plexfem-ghost-facet-fix.patch
  • Do not apply both together (overlapping hunks / duplicate logic).

@gthyagi
gthyagi requested a review from lmoresi as a code owner March 12, 2026 01:19
@gthyagi
gthyagi force-pushed the feature/boundary-integrals branch from 6fca5e6 to 0561eae Compare March 12, 2026 01:25
@gthyagi

gthyagi commented Mar 12, 2026

Copy link
Copy Markdown
Contributor Author

@lmoresi quick clarification: this PR’s plexfem-internal-boundary-ownership-fix.patch supersedes the earlier plexfem-ghost-facet-fix.patch used in this branch. The newer patch includes the ghost-facet fix plus additional part-consistent boundary assembly updates.

I’m happy to leave consolidation of patch history and documentation to your preferred approach.

I will leave it up to you to create a new PETSc MR or add this to your existing PETSc MR.

@lmoresi
lmoresi merged commit c98affe into underworldcode:feature/boundary-integrals Mar 12, 2026
1 check passed
lmoresi added a commit that referenced this pull request Mar 13, 2026
Document boundary integral feature (PR #70), PETSc ownership patch
(PR #78), binder fix (#71), and worktree symlink safety in both
CHANGES.md and docs/developer/CHANGELOG.md.

Underworld development team with AI support from Claude Code
@gthyagi
gthyagi deleted the feature/boundary-integrals branch April 8, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants