Add Kaiju and Gadi cluster support: pixi env, PETSc build script, and documentation

[jcgraciosa]

Summary

• Adds kaiju pixi feature and environment to pixi.toml (linux-64, conda-forge Python packages — sympy, scipy, pint, pydantic, gmsh, etc.)
• Adds petsc-custom/build-petsc-kaiju.sh — PETSc configure script for Kaiju (spack OpenMPI, AMR tools, petsc4py)
• Adds docs/developer/guides/kaiju-cluster-setup.md — full installation and usage guide for the Kaiju HPC cluster
• Updates docs/developer/index.md to include the new guide in the toctree

Background
Kaiju is a Rocky Linux 8.10 HPC cluster (Slurm + spack) used by the Underworld development team. MPI-dependent packages (mpi4py, PETSc+AMR+petsc4py, h5py) must be built from source against spack's OpenMPI to be compatible with Slurm's parallel interconnect.

The architecture is:

pixi kaiju env → Python 3.12, sympy, scipy, pint, ... (conda-forge, no MPI)
spack → openmpi@4.1.6 (cluster MPI)
source build → mpi4py, PETSc+AMR+petsc4py, h5py (linked to spack MPI)

Key fixes included

• build-petsc-kaiju.sh: -DUSE_SCOTCH=OFF in MMG cmake arguments — fixes PARMMG configure failure with pixi's conda ld 14.x (pixi's stricter linker requires transitive shared library deps to be explicit; libmmg.so built with SCOTCH causes MMG_WORKS link test to fail)
• load_env(): builds LD_LIBRARY_PATH from all spack transitive dep prefixes via CMAKE_PREFIX_PATH — required for pixi's ld at link time
• pixi shell-hook (not pixi shell) used for Slurm batch job compatibility

Test plan

• Per-user install verified on Kaiju head node (verify_install passed)
• Shared install deployed to /opt/cluster/software/underworld3, accessible via module load underworld3/development-12Mar26
• Slurm job script verified (multi-node MPI run)
• Docs build (pixi run docs-build) — verify new page appears in developer guides

Notes
Install scripts and Slurm job templates are maintained in the kaiju-admin-notes repo (admin/cluster-specific tooling kept separate from the framework).

Underworld development team with AI support from Claude Code

[lmoresi]

Review: Design Feedback

Thanks for this — getting UW3 running on Kaiju and Gadi is important, and the documentation is excellent. The main feedback is about the architecture: per-cluster proliferation of build scripts and pixi features won't scale well. Here's a suggested path forward.

1. Single feature.hpc instead of per-cluster features

The feature.kaiju and feature.gadi pixi environments are nearly empty — they're just platform markers. A single HPC feature covers both (and future clusters):

# ============================================
# HPC CLUSTER FEATURE
# ============================================
# Pure Python from pixi; MPI/PETSc/h5py built from source
# against the cluster's MPI (spack, modules, etc.)
# See: docs/developer/guides/hpc-cluster-setup.md

[feature.hpc]
platforms = ["linux-64"]

[feature.hpc.dependencies]
# patchelf needed on some clusters (e.g. Gadi) to fix h5py RPATH
patchelf = "*"

# --- HPC Cluster Track (linux-64 only) ---
[environments]
hpc = { features = ["hpc"], solve-group = "hpc" }

Adding patchelf unconditionally is harmless — it's tiny and only used when needed.

2. Parameterise the PETSc build script

The two build scripts are ~90% identical. The differences are:

• Whether to --download-hdf5 vs --with-hdf5-dir
• Whether to --download-fblaslapack vs system BLAS
• Whether to --download-cmake vs system cmake
• PETSC_ARCH naming

Instead of build-petsc-kaiju.sh and build-petsc-gadi.sh, extend the existing build-petsc.sh with a --cluster flag or auto-detection. The cluster-specific bits can be a config block at the top:

# In build-petsc.sh, detect cluster from hostname or explicit flag:
case "${UW_CLUSTER:-auto}" in
  kaiju|*.kaiju.*)
    EXTRA_PETSC_OPTS="--download-hdf5 --download-fblaslapack --download-cmake"
    PETSC_ARCH="petsc-4-uw-openmpi"
    ;;
  gadi|*.gadi.nci.*)
    EXTRA_PETSC_OPTS="--with-hdf5-dir=${HDF5_DIR}"
    PETSC_ARCH="petsc-4-uw-openmpi"
    ;;
  *)
    # Default: local dev build (existing logic)
    ;;
esac

This keeps one script to maintain, and PETSC_ARCH follows the existing petsc-4-uw-{mpi} convention.

3. If you'd like to use Claude Code to do this refactoring

Here's a prompt that should work well:

Refactor the HPC cluster support in PR #79. Read petsc-custom/build-petsc.sh, petsc-custom/build-petsc-kaiju.sh, and petsc-custom/build-petsc-gadi.sh. Extract the cluster-specific differences (HDF5, BLAS, cmake, PETSC_ARCH) into a configuration block in build-petsc.sh keyed by UW_CLUSTER env var or hostname auto-detection. Remove the separate cluster scripts. In pixi.toml, replace feature.kaiju and feature.gadi with a single feature.hpc. Update the kaiju docs to reference the unified script.

4. Minor: Gadi is out of scope

The PR title is "Add Kaiju cluster support" but also adds a full Gadi build script and pixi feature. I'd suggest either:

• Rename the PR to cover both, or
• Split Gadi into a follow-up (cleaner if you're refactoring to the unified approach anyway)

Summary

The substance is all good — the PETSc configure flags, the MMG/SCOTCH fix, the spack LD_LIBRARY_PATH handling, and the documentation are solid. It's just the packaging that needs consolidation before we end up with build-petsc-{kaiju,gadi,setonix,magnus,...}.sh.

[jcgraciosa]

Update: latest changes

1. Replaced kaiju-cluster-setup.md with hpc-cluster-setup.md, which now covers both Kaiju and Gadi in a single document (architecture overview, hardware, prerequisites, install steps, job submission, shared install, and troubleshooting for each cluster)
2. Merged build-petsc-kaiju.sh and build-petsc-gadi.sh into a single build-petsc.sh with cluster auto-detection (UW_CLUSTER=kaiju|gadi or hostname); replaced feature.kaiju/feature.gadi in pixi.toml with unified feature.hpc

[lmoresi]

Follow-up on consolidated PR — looks great

The refactoring is clean and well-structured. The detect_cluster() + case-block approach in build-petsc.sh is exactly what we were after. A few items:

Bug: toctree reference (please fix)

docs/developer/index.md line 117 still references guides/kaiju-cluster-setup but the file is now hpc-cluster-setup.md. This will break the docs build:

-guides/kaiju-cluster-setup
+guides/hpc-cluster-setup

Suggestion: PETSC_ARCH naming consistency

Not blocking, but worth considering for future maintainability:

• Kaiju uses petsc-4-uw (no MPI suffix) — since Kaiju only has OpenMPI, this is fine functionally, but petsc-4-uw-openmpi would match the local convention and make it obvious which MPI was used if someone inspects the directory later.

• Gadi uses arch-linux-c-opt (PETSc default) — same idea: petsc-4-uw-openmpi would be consistent. No strong feelings here since these are cluster-specific builds that won't coexist with other archs.

Both are minor — happy to merge once the toctree fix is in.

[jcgraciosa]

Thanks for the checks.
I've fixed the toctree reference and renamed PETSC_ARCH to petsc-4-uw-openmpi.

[lmoresi]

Thanks for the quick turnaround on both fixes. Toctree and PETSC_ARCH naming both look good. Approving — ready to merge.