Add Gadi Singularity container build files for UW3 - #133
Conversation
lmoresi
left a comment
lmoresi
left a comment
There was a problem hiding this comment.
Thanks @jcgraciosa — multi-stage build is clean and the Gadi-specific environment touches (`OMPI_MCA_io=ompio`, `OPENBLAS_NUM_THREADS=1`, scratch redirect for the Singularity cache) are exactly right. Nice touch on the aarch64 graceful skips for gmsh/vtk-osmesa so the recipe still tests locally on Apple Silicon.
One change required, plus a few optional polish items.
Required: drop `pykdtree` from `underworld3.rhel`
Line ~143 of `underworld3.rhel` lists `pykdtree` in the pip install. UW3 removed this dependency in the Aug 2025 KDTree backend switch — `uw.kdtree` is now backed by ckdtree/nanoflann. Beyond being unused, pykdtree's OpenMP integration causes:
- Fatal crashes on macOS when loaded alongside PETSc/numpy/scipy (double `libomp.dylib` initialisation — C-level abort, not catchable by Python).
- Hangs under MPI during KDTree queries due to thread contention between OpenMP and the MPI processes.
The second one is the concern for this container specifically — Gadi runs are MPI by construction. Please drop `pykdtree` from the pip install list.
(Background and justification are in the planning entry titled "Remove pykdtree dependency", 2026-02-13.)
Optional polish
These are not blockers — happy to land the must-fix without them, and follow up later.
-
Default `PETSC_IMAGE` points at a personal namespace. `underworld3.rhel:24` defaults to `ghcr.io/jcgraciosa/petsc:3.25.0-ompi`. The README correctly tells users to override with `--build-arg`, but a downstream user copy-pasting the build command without reading the README would pull from your personal namespace. Either point the default at an org-level image (e.g. `ghcr.io/underworldcode/petsc:3.25.0-ompi`, once published) or leave the default empty so the build fails loudly instead of silently using the wrong image.
-
`--with-cxx-dialect=C++11` for PETSc 3.25. PETSc 3.25 requires C++14 as a minimum. The flag may still work for backwards compatibility but it's understating what PETSc actually needs — easier to drop it entirely (PETSc auto-detects) or set `C++14` explicitly.
-
`--download-fblaslapack=1` while runtime has `openblas`. The runtime layer installs system `openblas`, but PETSc downloads f2c BLAS/LAPACK separately. Using `--with-blaslapack-dir` against the system openblas would shrink the image a bit and speed up the configure stage. Optimisation, not correctness.
Cosmetic
- Containerfile build-command comments reference `./docs/development/gadi_singularity/` (in two places) while the actual path is `./docs/developer/` — two-letter typo.
- `petsc.rhel` ends without a trailing newline.
Looks good
- Multi-stage `runtime → builder → final` keeps the final image small.
- Patch application has a graceful "already-merged-upstream" fallback.
- petsc4py/slepc4py install failure dumps the build log on exit — good debuggability.
- `mpi4py` forced `--no-binary` against openmpi and `h5py` rebuilt `HDF5_MPI=ON` against PETSc's HDF5 — both essential and easy to get wrong.
Happy to approve and merge once `pykdtree` is removed. The polish items can roll into this PR or a follow-up — whichever you prefer.
Underworld development team with AI support from Claude Code
986513b to
06813bd
Compare
Since 2026-04-30 every push to development and every PR has been failing
CI with the runner timing out:
Resolving Environment ⧖ Starting # 02:51:02
##[error]The runner has received a shutdown signal. # 03:57:25
That's 66 minutes spent in micromamba's conda-forge solve before the
test step starts, after which GitHub kills the runner. environment.yaml
has loose constraints (python <= 3.11, an exact pin on petsc=3.21.5
that's now a year-old in conda-forge, and several unpinned packages
including pykdtree which UW3 doesn't actually use any more) — that
combination, plus recent conda-forge package state shifts, has put the
solver into deep backtracking.
The pixi.lock committed in the repo already captures the exact same
dependency set we use locally for development. Using
prefix-dev/setup-pixi with frozen: true (refuses to re-solve) gives a
deterministic, fast install matching local dev state. Bonus: the build
step now uses `pixi run -e dev build` (= `pip install . --no-build-isolation`
from pixi.toml), avoiding the editable install the previous CI was doing
in violation of project policy (CLAUDE.md "NEVER use pip install -e .").
Replaces the workflow file in place; keeping environment.yaml in the
repo for now because PR underworldcode#133 (Gadi Singularity container) consumes it,
and removing it would create a cross-PR ordering hazard.
Test plan: this commit's own CI run is the test plan — if it goes
green in <10 min instead of red in 90 min, we have our answer.
Underworld development team with AI support from Claude Code
Addresses the required review change on underworldcode#133: - remove pykdtree from the pip install (dependency removed in the Aug 2025 KDTree backend switch to ckdtree/nanoflann; its OpenMP integration hangs under MPI — the relevant failure mode for a Gadi container) - fix build-command comment path docs/development -> docs/developer (both files) - add missing trailing newline to petsc.rhel Optional polish (personal PETSC_IMAGE default, C++11 vs C++14 dialect, --download-fblaslapack vs system openblas) left for a follow-up. Underworld development team with AI support from Claude Code
lmoresi
left a comment
lmoresi
left a comment
There was a problem hiding this comment.
Approving — I pushed the one required change (drop pykdtree) plus the two cosmetic fixes (doc-path typo, trailing newline) directly to the branch (commit 9f29538), since maintainer-edit was enabled and the PR had been idle.
Thanks @jcgraciosa — the multi-stage build, the MPI-correct mpi4py --no-binary / h5py HDF5_MPI=ON rebuilds, and the Gadi env touches are all solid. The optional polish (personal PETSC_IMAGE default → org namespace, C++11→C++14 dialect for PETSc 3.25, --download-fblaslapack vs system openblas) can roll into a follow-up. Merging for the release.
Underworld development team with AI support from Claude Code
Adds two workflows for docs/developer/gadi_singularity/, which has had no CI since underworldcode#133 and has only ever been built by hand: - gadi-uw3-image.yml builds underworld3.rhel on pushes to main/development, v* tags and releases, publishing underworld3-gadi:<branch|tag>. - gadi-petsc-image.yml builds petsc.rhel, triggered only by that file and petsc-custom/patches/**, publishing petsc-gadi:<version>-ompi. Both are amd64-only and publish Docker v2 media types with buildx attestations disabled, which is what Gadi's Singularity pulls reliably. Image names derive from github.repository_owner so the files work unchanged in a fork. Each build is smoke-tested, then old untagged versions are pruned (keeping 3, tagged exempt) only if that test passed. Fixes five pre-existing problems the automation exposed, none of them visible when building by hand on macOS: - underworld3.rhel omitted requests, which utilities/_utils.py imports unguarded at module level, so import underworld3 raised ModuleNotFoundError. - underworld3.rhel:22 was missing a leading '#'. podman continues a comment across a trailing backslash; BuildKit read it as an instruction. - setuptools-scm was absent from the builder, so with --no-build-isolation the dynamic version never resolved and the image reported 0.0.0. - petsc.rhel pinned cython>=3.1 unbounded. petsc4py 3.25.0's PC.pyx:1256 does not cythonize under 3.3.0; bisected, 3.2.9 is the last that builds. - ARGs re-declared after FROM carried no defaults, so PYVER could expand empty and yum would look for python-pip. Also adds ARG UW3_REPO, since the clone URL was hardcoded and a fork could not build its own branches, and ARG PETSC_MAKE_NP. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 participants
Adds two Containerfiles and a README for building and running UW3 as a Singularity container on Gadi (NCI), modeled on the UW2 gadi_singularity setup.
Files added
docs/developer/gadi_singularity/petsc.rhel— builds PETSc 3.25.0 with full AMR support (petsc4py, slepc4py, mmg, parmmg, ptscotch, hypre, etc.) on Rocky Linux 8.10docs/developer/gadi_singularity/underworld3.rhel— builds UW3 on top of the PETSc imagedocs/developer/gadi_singularity/README.md— build and deployment instructionsTested
singularity pullNotes
COPY petsc-custom/patches/...)--platform linux/amd64required when building on Apple SiliconSINGULARITY_CACHEDIRmust point to scratch on Gadi (home quota too small)Future work
--mca btl ^openib)Underworld development team with AI support from Claude Code