Build cache coherence across PETSc version switches by lmoresi · Pull Request #120 · underworldcode/underworld3

lmoresi · 2026-04-17T19:31:32Z

Why

Issue #96 (BdIntegral parallel hang) was fixed by a PETSc version bump that ships the upstream ownership fix. That merged in #115, but after switching PETSc the build system was leaving stale compiled code in place: users who pulled the fix and did ./uw petsc switch + ./uw build still loaded extensions linked against the previous PETSc. The #96 fix wasn't actually reaching their installs, and code like jcgraciosa's that depended on it stayed broken.

Three places were silently assuming PETSc never changes during the life of an install. Fixing only one or two still leaves a stale-load path open; this PR fixes all three so the chain from ./uw petsc switch → ./uw build → import underworld3 is consistent.

The three fixes

1. `setup.py` — honour `PETSC_DIR` / `PETSC_ARCH` env vars first

setup.py::configure() was trying import petsc (the conda pip package) first, which returns the conda PETSc path even when $PETSC_DIR/$PETSC_ARCH point at a custom build — so the custom build was silently ignored and underworld3 compiled against the wrong libpetsc. It also only consulted env vars as a corner case inside the conda fallback.

Explicit priority chain now:

$PETSC_DIR / $PETSC_ARCH from the environment (pixi activation, shell, HPC module)
petsc4py.get_config() — authoritative match for the installed petsc4py
conda petsc pip package
CONDA_PREFIX/lib/.../site-packages/petsc fallback

Env vars are the most specific signal and should win.

2. `./uw petsc switch` — nuke stale wheels and `petsc4py`

./uw build already has target-change detection via build/.petsc_target, but it reads petsc4py.get_config() to identify the "current" target. Right after a switch, petsc4py is still the old one and reports the old arch, so the check doesn't fire, pip's wheel cache (version 0.0.0 collides forever) and Cython's build/lib.*/build/temp.* both hand back the previous build.

./uw petsc switch now forces the clean rebuild path at switch time:

remove build/lib.*, build/temp.*, build/bdist.*, and build/.petsc_target
pip cache purge
uninstall petsc4py in AMR envs (rebuilt by the next ./uw build against the new PETSc)

3. `pixi.toml` → activation script — make `PETSC_ARCH` actually change when PETSc changes

[feature.*.activation.env] PETSC_ARCH = "petsc-*-uw-*" was pinned in tracked pixi.toml. Since pixi caches its solve against pixi.toml content, $PETSC_ARCH inside a pixi shell could disagree with petsc-custom/.petsc-version whenever the latter changed.

Replace the five hardcoded entries (features amr [osx-arm64 / linux-64], amr-mpich, amr-openmpi, amr-debug) with a single activation script wired in per-feature:

[feature.amr.activation]
scripts = ["petsc-custom/activate-petsc-arch.sh"]

The script (sourced on every pixi shell / pixi run) reads petsc-custom/.petsc-version plus $PIXI_ENVIRONMENT_NAME (and uname for the generic amr env) and exports PETSC_ARCH=petsc-{ver}-uw-{mpi}{suffix}. Non-AMR envs match no case and leave PETSC_ARCH unset — no-op for default/runtime/dev/mpich/openmpi.

pixi.toml now stays static across PETSc versions; switching touches only the gitignored .petsc-version.

How the three fit together

./uw petsc switch v3.25.0
 │
 ├─ .petsc-version       ← updated
 ├─ build/.petsc_target  ← deleted
 ├─ build/lib.* temp.*   ← deleted
 ├─ pip cache            ← purged
 └─ petsc4py             ← uninstalled (AMR envs)

./uw build
 │
 ├─ pixi shell           → activate-petsc-arch.sh runs, exports new PETSC_ARCH
 ├─ petsc4py rebuild     → against new PETSc
 └─ setup.py             → honours PETSC_DIR/PETSC_ARCH env vars (not the
                           stale petsc4py.get_config()) → links against new PETSc

Any single one of these in isolation still leaves a stale-load path; the three together close the loop.

Test plan

From a clean build, ./uw petsc switch v3.25.0 (or any other tag) then ./uw petsc build && ./uw build produces an underworld3 importable against the new arch.
Link-time verification: after a switch + rebuild, confirm the installed .so is linked against the new PETSc, not the old one:
```
otool -L .pixi/envs/amr-dev/lib/python3.12/site-packages/underworld3/cython/petsc_maths.cpython-312-darwin.so | grep petsc
```
Should resolve to petsc-{new-ver}-uw-openmpi/lib/libpetsc.{new-ver}.dylib, not the previous petsc-{old-ver}-*.
import petsc4py; petsc4py.get_config() post-switch reports the new PETSC_DIR/PETSC_ARCH.
git diff pixi.toml after a switch is empty.
pixi run -e amr-dev env | grep PETSC_ARCH matches ./uw petsc active.
pixi run -e default env | grep -c PETSC_ARCH is 0 (script is a no-op off-AMR).
Parallel reproducer for Parallel bug - Calculating a boundary integral causes the next Stokes solve to hang #96 stops hanging against a freshly-switched PETSc — confirms the fix actually reaches the user-facing install (jcgraciosa's code path was blocked here).

Notes for reviewers

Three commits, reviewable independently.
After merge, contributors on older checkouts should pixi install -e <env> once to pick up the per-feature activation-script wiring.
./uw still contains petsc_arch_for_env() for internal diagnostics; it reads the same .petsc-version so its output stays consistent with the activation script.

Underworld development team with AI support from Claude Code

Previously ./uw petsc switch rewrote PETSC_ARCH values in pixi.toml via sed, which (1) made every switch a tracked file change, (2) forced contributors on different PETSc versions to carry diverging pixi.toml in their working tree, and (3) did not reliably propagate to pixi's activated environment because pixi caches resolved state. Replace the five hardcoded PETSC_ARCH = "petsc-*-uw-*" entries (in features amr [osx-arm64 and linux-64], amr-mpich, amr-openmpi, and amr-debug) with per-feature activation scripts that source the new petsc-custom/activate-petsc-arch.sh. The script reads the active version from petsc-custom/.petsc-version and combines it with the active $PIXI_ENVIRONMENT_NAME (and uname on Darwin/Linux for the plain amr env) to construct PETSC_ARCH = petsc-{ver}-uw-{mpi}{suffix}. Non-AMR environments match no case and leave PETSC_ARCH unset, so the script is safe as a no-op for default/runtime/dev/mpich/openmpi environments. Switching PETSc versions now only updates .petsc-version (one untracked, gitignored file); pixi.toml is not modified. Underworld development team with AI support from Claude Code

./uw build has target-change detection (build/.petsc_target) that cleans build/lib.* when petsc4py.get_config() reports a new arch. After ./uw petsc switch, though, the still-installed petsc4py is linked to the OLD arch and continues to report it, so the check does not fire and pip/Cython silently reuse stale wheels against the new PETSc. Force the full rebuild path at switch time: remove build/lib.*, build/temp.*, build/bdist.*, the .petsc_target marker, purge pip's wheel cache, and uninstall petsc4py (AMR envs only). The next ./uw build then rebuilds petsc4py against the new PETSc and recompiles every Cython extension. Underworld development team with AI support from Claude Code

Copilot

Pull request overview

This PR aims to make PETSc version switching non-invasive to tracked configuration by keeping pixi.toml static (no more PETSC_ARCH rewrites) and ensuring PETSc switches don’t leave behind stale build artifacts/wheels that cause underworld3/petsc4py to link against the wrong PETSc.

Changes:

Add a pixi activation script to dynamically export PETSC_ARCH for AMR environments based on petsc-custom/.petsc-version and the active pixi environment.
Remove hardcoded PETSC_ARCH entries from pixi.toml AMR features and wire in the activation script per feature.
On ./uw petsc switch, proactively remove build artifacts/markers and uninstall petsc4py (AMR envs) to force a clean rebuild on the next ./uw build.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
`uw`	Adds cleanup steps on `petsc switch` to prevent stale build reuse after PETSc changes.
`pixi.toml`	Removes hardcoded `PETSC_ARCH` values and attaches an activation script for AMR features.
`petsc-custom/activate-petsc-arch.sh`	New activation script that derives and exports `PETSC_ARCH` at environment activation time.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-17T19:35:54Z

+if [ -n "$_uw_mpi" ]; then
+    _uw_ver="4"
+    _uw_ver_file="${PIXI_PROJECT_ROOT:-.}/petsc-custom/.petsc-version"
+    [ -f "$_uw_ver_file" ] && _uw_ver=$(cat "$_uw_ver_file" 2>/dev/null || echo "4")
+
+    export PETSC_ARCH="petsc-${_uw_ver}-uw-${_uw_mpi}${_uw_suffix}"


activate-petsc-arch.sh defaults _uw_ver to 4 and reads .petsc-version without trimming whitespace. Elsewhere (uw and petsc-custom/build-petsc.sh) the version defaults to 324 and .petsc-version is expected to contain the short numeric form (e.g. 325). On a fresh checkout where .petsc-version is missing (or contains whitespace), this can export a legacy/invalid PETSC_ARCH and break downstream builds (notably pip install of petsc4py which relies on PETSC_ARCH). Align the default/version-parsing logic with petsc_version_short() (default 324, strip whitespace, and ideally fall back to tag detection when the file is absent).

Copilot · 2026-04-17T19:35:55Z

+[feature.amr-debug.activation]
+scripts = ["petsc-custom/activate-petsc-arch.sh"]
+
 [feature.amr-debug.activation.env]
 PETSC_DIR = "$PIXI_PROJECT_ROOT/petsc-custom/petsc"


The AMR debug setup comment block still hardcodes petsc-4-uw-openmpi-debug, but PETSC_ARCH is now being set via the activation script (and will be version-derived from petsc-custom/.petsc-version). This documentation is now misleading for anyone following the debug setup steps; update it to reflect the new arch naming scheme and clarify whether debug builds should be legacy (petsc-4-...-debug) or versioned (petsc-<ver>-...-debug).

Copilot · 2026-04-17T19:35:55Z

+            $PIXI run -e "$(get_env)" pip cache purge 2>/dev/null || true
+            if is_amr_env "$(get_env)"; then
+                echo "Uninstalling old petsc4py (linked to previous PETSc build)..."
+                $PIXI run -e "$(get_env)" pip uninstall -y petsc4py 2>/dev/null || true
+            fi


pip cache purge clears the entire pip cache (not just PETSc/petsc4py artifacts). If the goal is specifically to prevent reuse of cached petsc4py/UW wheels across PETSc switches, consider a more targeted approach (e.g. removing only the relevant cached wheels, or ensuring the subsequent installs use --no-cache-dir/--force-reinstall) to avoid unnecessarily slowing unrelated installs.

Copilot · 2026-04-17T19:35:55Z

+        esac ;;
+    amr-mpich|amr-mpich-dev)     _uw_mpi="mpich" ;;
+    amr-openmpi|amr-openmpi-dev) _uw_mpi="openmpi" ;;
+    amr-debug)                   _uw_mpi="openmpi"; _uw_suffix="-debug" ;;


In amr-debug the script hardcodes _uw_mpi="openmpi", but on linux-64 the amr feature dependencies use MPICH (see pixi.toml target deps). That makes PETSC_ARCH inconsistent with the MPI actually present in the environment and can lead to PETSc/petsc4py being built/loaded against the wrong MPI. Consider selecting MPI for amr-debug the same way as amr (platform-default), or provide explicit amr-debug-mpich / amr-debug-openmpi envs/features to avoid cross-platform mismatches.

Suggested change

amr-debug) _uw_mpi="openmpi"; _uw_suffix="-debug" ;;

amr-debug)

case "$(uname -s)" in

Darwin) _uw_mpi="openmpi" ;;

*) _uw_mpi="mpich" ;;

esac

_uw_suffix="-debug" ;;

…ig() When pixi activation exports PETSC_DIR and PETSC_ARCH for a custom build, setup.py was ignoring them and reading petsc4py.get_config() instead. That reports the arch petsc4py was BUILT against — so immediately after a PETSc version switch, setup.py would silently link the rebuild against the previous PETSc, not the new one. Re-order the PETSc discovery into an explicit priority chain: 1. PETSC_DIR / PETSC_ARCH from the environment (pixi activation, shell export, HPC module load) 2. petsc4py.get_config() — authoritative match for installed petsc4py 3. conda `petsc` pip package 4. CONDA_PREFIX/lib/.../site-packages/petsc fallback The previous code only consulted env vars as a last-resort branch of the conda fallback, which was exactly backwards — env vars are the most specific signal and should win. Underworld development team with AI support from Claude Code

- activate-petsc-arch.sh: platform-aware MPI for amr-debug (was hardcoded openmpi, wrong on linux-64 where feature.amr pulls mpich). Version discovery now matches ./uw petsc_version_short: strip whitespace from .petsc-version, default to 324 (not "4", which would produce a legacy arch name that no current build tree uses). - pixi.toml: update the amr-debug setup comment to refer to "$PETSC_ARCH" (now exported by the activation script) instead of the stale hardcoded "petsc-4-uw-openmpi-debug" string. - uw: drop the global `pip cache purge` on ./uw petsc switch — it clears every project's pip wheel cache, which is overkill. The two installs that matter already avoid the cache: run_build passes --no-cache-dir to `pip install .`, and petsc4py's local-path install now does the same. Underworld development team with AI support from Claude Code

lmoresi added 2 commits April 17, 2026 12:30

Copilot AI review requested due to automatic review settings April 17, 2026 19:31

Copilot started reviewing on behalf of lmoresi April 17, 2026 19:32 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

lmoresi changed the title ~~Static pixi.toml: derive PETSC_ARCH from activation script; nuke stale build on petsc switch~~ Build cache coherence across PETSc version switches Apr 17, 2026

lmoresi merged commit 403dd4f into development Apr 17, 2026
1 check passed

lmoresi deleted the feature/pixi-petsc-arch-static branch April 17, 2026 21:11

lmoresi mentioned this pull request Apr 17, 2026

Fix BdIntegral parallel hang: coordinate DM label contamination #115

Merged

5 tasks

Navigation Menu

Uh oh!

Build cache coherence across PETSc version switches - #120

Build cache coherence across PETSc version switches#120
lmoresi merged 4 commits into
developmentfrom
feature/pixi-petsc-arch-static

lmoresi commented Apr 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lmoresi commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

The three fixes

1. setup.py — honour PETSC_DIR / PETSC_ARCH env vars first

2. ./uw petsc switch — nuke stale wheels and petsc4py

3. pixi.toml → activation script — make PETSC_ARCH actually change when PETSc changes

How the three fit together

Test plan

Notes for reviewers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lmoresi commented Apr 17, 2026 •

edited

Loading

1. `setup.py` — honour `PETSC_DIR` / `PETSC_ARCH` env vars first

2. `./uw petsc switch` — nuke stale wheels and `petsc4py`

3. `pixi.toml` → activation script — make `PETSC_ARCH` actually change when PETSc changes