From 2d1efae419c10c8f7cdb4f03fdf10706fbe09736 Mon Sep 17 00:00:00 2001 From: Henry Fricke Date: Mon, 3 Aug 2026 20:39:58 -0600 Subject: [PATCH] Rewrite mpiCASA.md as a real, tested CASA tutorial for Easley Previous content was raw, unfinished draft notes: a fake placeholder path (/users/sbruzew/xena-scratch/casa-blah-blah/bin/casa), the dead PBS variable $PBS_NODEFILE, a nonexistent "singleGPU" partition, and unresolved rhetorical questions to self. Investigated live on Easley: `module avail casa` / `module spider casa` both claim casa/6.5.2 is loadable, but `module load casa/6.5.2` actually fails - Lmod's cached spider index is stale and falsely advertises a module whose modulefile no longer exists (confirmed with `module --ignore_cache avail casa`, which finds nothing). This is a real CARC module-system gap, not something fixable from the doc side; flagged for help@carc.unm.edu in the doc itself. The underlying CASA 6.5.2 software is still installed and world-executable at a fixed path, so the tutorial documents using that direct path with an alias instead. Tested end-to-end on Easley via srun: both plain `casa --nogui --log2term -c script.py` (1 task) and the MPI-parallel `mpicasa -n 2 casa ...` wrapper (2 tasks, landed on 2 separate nodes) ran a real Python script inside CASA and printed correct output. Replaced the fake singleGPU partition with Easley's real partitions (general/bigmem/h100/l40s/interactive/debug/scavenger, verified via sinfo) and PBS_NODEFILE with plain Slurm --ntasks. Co-Authored-By: Claude Sonnet 5 --- mpiCASA.md | 111 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 22 deletions(-) diff --git a/mpiCASA.md b/mpiCASA.md index 4aba14a2..b5d42256 100644 --- a/mpiCASA.md +++ b/mpiCASA.md @@ -1,41 +1,108 @@ -# Using CASA on CARC +# Using CASA on Easley ### A Bit About CASA -[CASA](https://casa.nrao.edu/) is the premier software for reducing radio data coming off of a variety of telescopes around the world, including the Jansky Very Large Array (VLA) and Atacama Large Millimeter Array (ALMA). +[CASA](https://casa.nrao.edu/) (Common Astronomy Software Applications) is the primary software for reducing radio interferometry data from telescopes such as the Jansky Very Large Array (VLA) and the Atacama Large Millimeter/submillimeter Array (ALMA). -### Getting Some Data to Play With +### Known Issue: `module load casa` currently does not work -Going to use the new [Archive](data.nrao.edu). Find something small and have them download it manually, like a few GB at most. +Running `module avail casa` on Easley (or Hopper) will show: -### Getting Set Up +``` +----------------------------- /opt/local/modules ------------------------------ + casa/6.5.2 +``` + +and `module spider casa/6.5.2` even says "This module can be loaded directly." **Don't trust this** — it's a stale entry in Lmod's cached module index. Actually running `module load casa/6.5.2` fails: + +``` +Lmod has detected the following error: These module(s) or extension(s) exist +but cannot be loaded as requested: "casa/6.5.2" +``` + +and re-checking with the cache disabled (`module --ignore_cache avail casa`) confirms there is currently no live modulefile for CASA at all — only the stale cached listing. This is a real gap in CARC's module system, not something a doc edit can fix; if you hit this, let help@carc.unm.edu know the `casa/6.5.2` module needs its modulefile restored. + +The good news: the actual CASA 6.5.2 software is still installed on disk and runs fine — it's just not wired up to `module load`. Use the direct path shown below until CARC fixes the module. + +```bash +CASA_BIN=/opt/local/casa/6.5.2/casa-6.5.2-26-py3.8/bin/casa +``` + +You can confirm this path is valid at any time with: + +```bash +$CASA_BIN --version +# CASA 6.5.2.26 +``` + +### Getting Compute Resources with Slurm + +Easley uses Slurm, not PBS — there is no `$PBS_NODEFILE` and no `singleGPU` partition (that partition does not exist; CASA doesn't need a GPU anyway). Easley's real partitions, per `sinfo`, are `general`, `bigmem`, `h100`, `l40s`, `interactive`, `debug`, and `scavenger`. For an ordinary CASA reduction script, request the `general` (or `debug`, for short test runs) partition — you don't need `bigmem` or a GPU partition unless your dataset specifically demands it. + +Request an interactive session to test in: + +```bash +srun --partition debug --time=00:30:00 --ntasks=1 --cpus-per-task=2 --pty bash +``` -First off get some nodes +Set up a convenience alias once you're on the node (do this per-session, or add it to your `~/.bashrc`): ```bash -srun --partition singleGPU --nodes 2 --tasks-per-node 2 --pty bash +CASA_DIR=/opt/local/casa/6.5.2/casa-6.5.2-26-py3.8 +alias casa="$CASA_DIR/bin/casa" +alias mpicasa="$CASA_DIR/bin/mpicasa" ``` -Do we want singleGPU? Can request more nodes obviously. tasks-per-node we don't need to set unless CASA demands slots +### Running CASA Non-Interactively -Might want to create aliases for casa and mpi casa, just to make things quick +CASA scripts are plain Python. Save a script, e.g. `hello.py`: + +```python +print("Hello from inside CASA") +print(2 + 2) +``` + +Run it with `-c`, `--nogui`, and `--log2term` (so output goes to your terminal/log instead of a GUI logger window): ```bash -alias casa='/users/sbruzew/xena-scratch/casa-blah-blah/bin/casa' -alias mpicasa='/users/sbruzew/xena-scratch/casa-blah-blah/bin/mpicasa' +$CASA_DIR/bin/casa --nologger --nogui --log2term -c hello.py +``` + +This was tested end-to-end on Easley (`srun --partition debug`, single task) and produces output including: + +``` +CASA 6.5.2.26 -- Common Astronomy Software Applications [6.5.2.26] +Hello from inside CASA +4 +``` + +### Running MPI-Parallel CASA with `mpicasa` + +CASA ships its own `mpicasa` wrapper (a thin layer over the OpenMPI build bundled with CASA) for running CASA in parallel across multiple MPI ranks/nodes. Request more than one task from Slurm and pass the rank count to `mpicasa` with `-n`: + +```bash +srun --partition debug --time=00:30:00 --ntasks=2 --cpus-per-task=2 --pty bash + +$CASA_DIR/bin/mpicasa -n 2 $CASA_DIR/bin/casa --nologger --nogui --log2term -c hello.py +``` + +This was also tested end-to-end on Easley across 2 tasks (Slurm placed them on two separate nodes) and each rank correctly ran the script and printed: + +``` +Hello from inside CASA +4 +``` + +**Note:** at the end of the run you'll likely see a message like: + +``` +mpirun has exited due to process rank 0 with PID ... exiting improperly... ``` -Actually it doesn't like the alias when you run the command +This is expected/benign — it happens because CASA's script mode doesn't call `MPI_Finalize()` cleanly on exit, not because anything failed. As long as your script's own output appears for every rank (as it does above), the run succeeded. -2) It will create a nodefile for us at $PBS_NODEFILE - If CASA doesn't need slots, we can use this, and all the nodes - If CASA does need slots, we'll need to make a script that can reference $PBS_NUM_PPN and add slots +### Getting Data to Work With -3) Run something like 'path_to_casa_bin/mpicasa -hostfile $PBS_NODEFILE path_to_casa_bin/casa ' - Probably want to do --nogui and --log2term - Puts us into a CASA environment +For real reduction work, download a dataset from the NRAO Science Data Archive at [data.nrao.edu](https://data.nrao.edu) — pick something small (a few GB) to start with. This QuickByte only demonstrates the pipeline (Slurm → CASA → your script); it does not cover calibration/imaging workflows, which are extensively documented in [NRAO's own CASA docs](https://casaguides.nrao.edu/). -4) Run python script that does all the fun stuff - Could run mpicasa call with '-c myscript.py' - Can also do 'exec(open('./filename').read())' - Shortcut as execfile 'filename.py' +*This quickbyte was tested against `casa/6.5.2` (run from its direct install path, since the module is currently broken — see above) on Easley.*