Multi-GPU simulation hangs

[lgloege]

I am trying to start running simulations on multiple GPUs on Yale's Bouchet cluster. I was able to successfully run the one-degree global sea-ice simulation on a single Nvidia h200 GPU and augmented my code to run on two h200 GPUs. When I start the simulation it just hangs. Any thoughts on why this may happen?

Here are the modules and exports I use:

module load Julia/1.12.4-linux-x86_64
module load OpenMPI/5.0.3-GCC-13.3.0-CUDA-12.6.0

export JULIA_MPI_HAS_CUDA=true
export UCX_WARN_UNUSED_ENV_VARS=n
export UCX_MEMTYPE_CACHE=n
export UCX_TLS=sm,cuda_copy,cuda_ipc,rc

export UCX_RNDV_SCHEME=put_zcopy

export OMPI_MCA_pml=ob1
export OMPI_MCA_btl=self,vader,tcp
export UCX_ERROR_SIGNALS="SIGILL,SIGBUS,SIGFPE"

and here is how I setup the architecture:

using MPI
MPI.Init()

using NumericalEarth
using Oceananigans
using Oceananigans.Units
using Oceananigans.DistributedComputations
using Dates
using Printf
using Statistics
using CUDA

comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)

#arch = Distributed(GPU(), partition=Partition(x=2))
arch = Distributed(GPU(); partition = Partition(y = DistributedComputations.Equal()), synchronized_communication=false)

One of the issues I was having is packages couldn't find MPI. This was solved from the help of this discussion. I had to explicitly set the paths in the LocalPreferences.toml. This feels like a clunky solution ... but it works.

[CUDA_Runtime_jll]
__clear__ = ["local"]
version = "12.6"

[MPIPreferences]
__clear__ = ["preloads_env_switch"]
_format = "1.0"
abi = "OpenMPI"
binary = "system"
cclibs = []
libmpi = "libmpi"
mpiexec = "srun"
preloads = []

[OpenMPI_jll]
libmpi_path = "/apps/software/2024a/software/OpenMPI/5.0.3-GCC-13.3.0-CUDA-12.6.0/lib/libmpi.so"
mpiexec_path = "/apps/software/2024a/software/OpenMPI/5.0.3-GCC-13.3.0-CUDA-12.6.0/bin/mpiexec"

[HDF5]
libhdf5 = "/apps/software/2024a/software/HDF5/1.14.5-gompi-2024a/lib/libhdf5.so"
libhdf5_hl = "/apps/software/2024a/software/HDF5/1.14.5-gompi-2024a/lib/lib/hdf5_hl.so"

Please let me know if you need more information, any help would be greatly appreciated. Thanks! @simone-silvestri

[simone-silvestri]

yeah, configuring MPI might be a bit of a hassle.
I am not 100% sure what is going wrong there.
You should try first with a very simple message passing test to see if MPI is configured correctly:

using MPI
MPI.Init()
# using CUDA # For CUDA-Aware MPI

rank = MPI.Comm_rank(MPI.COMM_WORLD)
size = MPI.Comm_size(MPI.COMM_WORLD)

a = zeros(Int, size)
a[rank] = rank + 1

# CUDA.device!(rank)
# a = CuArray(a) # for CUDA-Aware MPI

MPI.Allreduce!(a, +, MPI.COMM_WORLD)
@info rank, a # should show a = [1, 2, 3, 4] for all the ranks

I would test first that the CPU works, then uncomment the GPU parts to make sure GPU-GPU communication is enabled.

[lgloege]

Thanks @simone-silvestri , CPU works fine. In your code I had to change a[rank] = rank +1 to a[rank+1] = rank +1, but anyways. CPU works, but when I uncomment the GPU parts I get this error message:

[ljg48@a1122u02n01.bouchet numerical-earth]$ srun julia --project=${PROJECT} message-pass.jl
┌ Warning: CUDA runtime library `libcudart.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libcudart.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
┌ Warning: CUDA runtime library `libcublasLt.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libcublasLt.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
┌ Warning: CUDA runtime library `libnvJitLink.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libnvJitLink.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
┌ Warning: CUDA runtime library `libcusparse.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libcusparse.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
┌ Warning: CUDA runtime library `libcudart.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libcudart.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
┌ Warning: CUDA runtime library `libcublasLt.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libcublasLt.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
┌ Warning: CUDA runtime library `libnvJitLink.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libnvJitLink.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
┌ Warning: CUDA runtime library `libcusparse.so.12` was loaded from a system path, `/apps/software/system/software/CUDA/12.6.0/targets/x86_64-linux/lib/libcusparse.so.12`.
│ This may cause errors.
│
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│
│ In any other case, please file an issue.
└ @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/src/initialization.jl:198
ERROR: LoadError: CUDA error: invalid device ordinal (code 101, ERROR_INVALID_DEVICE)
Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/lib/cudadrv/libcuda.jl:30
  [2] check(f::CUDA.var"#cuDeviceGet##0#cuDeviceGet##1"{Base.RefValue{Int32}, Int64})
    @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/lib/cudadrv/libcuda.jl:37
  [3] cuDeviceGet(device::Base.RefValue{Int32}, ordinal::Int64)
    @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/GPUToolbox/x8hVh/src/ccalls.jl:33
  [4] CuDevice
    @ /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/lib/cudadrv/devices.jl:17 [inlined]
  [5] device!
    @ /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/lib/cudadrv/state.jl:324 [inlined]
  [6] device!(dev::Int64)
    @ CUDA /nfs/roberts/scratch/pi_me586/ljg48/julia_depot/packages/CUDA/Il00B/lib/cudadrv/state.jl:324
  [7] top-level scope
    @ /nfs/roberts/project/pi_me586/ljg48/numerical-earth/message-pass.jl:11
  [8] include(mod::Module, _path::String)
    @ Base ./Base.jl:306
  [9] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:317
 [10] _start()
    @ Base ./client.jl:550
in expression starting at /nfs/roberts/project/pi_me586/ljg48/numerical-earth/message-pass.jl:11
srun: error: a1122u02n01: task 1: Exited with exit code 1
slurmstepd: error:  mpi/pmix_v5: _errhandler: a1122u02n01 [0]: pmixp_client_v2.c:212: Error handler invoked: status = -61, source = [slurm.pmix.9327659.5:1]
slurmstepd: error: *** STEP 9327659.5 ON a1122u02n01 CANCELLED AT 2026-04-23T17:06:42 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: a1122u02n01: task 0: Killed

It didn't like the CUDA.device!(rank) command. However, it works when I explicity tell it there is 1 task, like this srun --ntasks=1 --gpus-per-task=1 julia --project=${PROJECT} message-pass.jl

Looping in Tom Langford @tlangfor, who been helping me with this too.

@simone-silvestri , any thoughts?

[simone-silvestri]

It looks like you only have one GPU per node, you can confirm it by looking at CUDA.devices() when you log in into a node. If this is the case, you can use only one rank per node since we have a 1 to 1 mapping between ranks in a node (ordered by their inter-node rank) and GPUs in a node.

[lgloege]

This PR in Oceananigans.jl solved this issue.

CliMA/Oceananigans.jl#5552