operator_mpi_example.py results in DISTRIBUTED_FAILURE (27)

[SzbKrisztian]

Hi,

I'm trying to run the operator_mpi_example.py example. At the point where the communicator of the WorkStrem is set fails, with DISTRIBUTED_FAILURE (27) error.

`Rank 0: ===== device info ======
Rank 0: GPU-local-id: 0
Rank 0: GPU-name: NVIDIA A100-SXM4-40GB
Rank 0: GPU-clock: 1410000
Rank 0: GPU-memoryClock: 1215000
Rank 0: GPU-nSM: 108
Rank 0: GPU-major: 8
Rank 0: GPU-minor: 0
Rank 0: ========================
Rank 1: ===== device info ======
Rank 1: GPU-local-id: 1
Rank 1: GPU-name: NVIDIA A100-SXM4-40GB
Rank 1: GPU-clock: 1410000
Rank 1: GPU-memoryClock: 1215000
Rank 1: GPU-nSM: 108
Rank 1: GPU-major: 8
Rank 1: GPU-minor: 0
Rank 1: ========================
Rank 0: Created WorkStream (execution context) on current device.
Rank 1: Created WorkStream (execution context) on current device.
Traceback (most recent call last):
Traceback (most recent call last):
File "/project/home/pr_1sk/operator_mpi_example.py", line 50, in
ctx.set_communicator(comm=MPI.COMM_WORLD.Dup(), provider="MPI")
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pr_1sk/.local/lib/python3.13/site-packages/cuquantum/densitymat/work_stream.py", line 195, in set_communicator
self._handle.set_communicator(comm, provider)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/pr_1sk/.local/lib/python3.13/site-packages/cuquantum/densitymat/_internal/library_handle.py", line 108, in set_communicator
cudm.reset_distributed_configuration(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self._validated_ptr, _comm_provider_map[provider], _comm_ptr, _size
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "cuquantum/bindings/cudensitymat.pyx", line 180, in cuquantum.bindings.cudensitymat.reset_distributed_configuration
File "cuquantum/bindings/cudensitymat.pyx", line 193, in cuquantum.bindings.cudensitymat.reset_distributed_configuration
check_status(status)
File "cuquantum/bindings/cudensitymat.pyx", line 135, in cuquantum.bindings.cudensitymat.check_status
raise cuDensityMatError(status)
cuquantum.bindings.cudensitymat.cuDensityMatError: DISTRIBUTED_FAILURE (27):
File "/project/home/pr_1sk/operator_mpi_example.py", line 50, in
ctx.set_communicator(comm=MPI.COMM_WORLD.Dup(), provider="MPI")
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pr_1sk/.local/lib/python3.13/site-packages/cuquantum/densitymat/work_stream.py", line 195, in set_communicator
self._handle.set_communicator(comm, provider)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/pr_1sk/.local/lib/python3.13/site-packages/cuquantum/densitymat/_internal/library_handle.py", line 108, in set_communicator
cudm.reset_distributed_configuration(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self._validated_ptr, _comm_provider_map[provider], _comm_ptr, _size
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "cuquantum/bindings/cudensitymat.pyx", line 180, in cuquantum.bindings.cudensitymat.reset_distributed_configuration
File "cuquantum/bindings/cudensitymat.pyx", line 193, in cuquantum.bindings.cudensitymat.reset_distributed_configuration
check_status(status)
File "cuquantum/bindings/cudensitymat.pyx", line 135, in cuquantum.bindings.cudensitymat.check_status
raise cuDensityMatError(status)
cuquantum.bindings.cudensitymat.cuDensityMatError: DISTRIBUTED_FAILURE (27):

`

I've installed the cuquantum package with conda and openmpi. I tried to look up the meaning of DISTRIBUTED_FAILURE (27), but found basically nothing.

Any suggestions where to start?

[b-kloss]

Could you please share the following:

• the exact command you ran to execute the example
• output from echo $CUDENSITYMAT_COMM_LIB
• the sequence of your conda install instructions
• output from conda list | grep cudensitymat
• output from running the sample like this: CUDENSITYMAT_LOG_LEVEL=5 mpirun -n 2 python operator_mpi_example.py

I verified locally that the sample executes without issue in a conda environment with:

conda list | grep cudensitymat
cudensitymat                     0.4.0            mpi_openmpi_h8ceb1f2_0    conda-forge

[SzbKrisztian]

Sure,

• the exact command you ran to execute the example
  mpirun -n 2 python operator_mpi_example.py

• output from echo $CUDENSITYMAT_COMM_LIB
/project/pr_1/micromamba/envs/cuqu_mpi/lib/libcudensitymat_distributed_interface_mpi.so

• the sequence of your conda install instructions
  I've created a conda (micromamba) env with python 3.13 and used micromamba install -c conda-forge cuquantum openmpi. Also installed mpi4py with openmpi according to mpi4py documentation.

• output from conda list | grep cudensitymat
cudensitymat 0.4.0 mpi_openmpi_ha1a7aa1_0 conda-forge

• output from running the sample like this: CUDENSITYMAT_LOG_LEVEL=5 mpirun -n 2 python operator_mpi_example.py
[2026-02-10 09:43:59][cuDensityMat][2429823][Api][cudensitymatCreate] handle=0X7FFD19FEB908 [2026-02-10 09:43:59][cuDensityMat][2429822][Api][cudensitymatCreate] handle=0X7FFD9BEE9258 [2026-02-10 09:44:00][cuDensityMat][2429822][Api][cudensitymatCreateWorkspace] handle=0X5602CEED66C0 workspaceDescr=0X7FFD9BEE9508 [2026-02-10 09:44:00][cuDensityMat][2429823][Api][cudensitymatCreateWorkspace] handle=0X55C2CE47A840 workspaceDescr=0X7FFD19FEBBB8 Rank 0: Created WorkStream (execution context) on current device. Rank 1: Created WorkStream (execution context) on current device. [2026-02-10 09:44:00][cuDensityMat][2429823][Api][cudensitymatResetDistributedConfiguration] handle=0X55C2CE47A840, provider=1, commPtr=0X7FF8D20CF0A0, commSize=8 [2026-02-10 09:44:00][cuDensityMat][2429822][Api][cudensitymatResetDistributedConfiguration] handle=0X5602CEED66C0, provider=1, commPtr=0X7F7501E491E0, commSize=8 [2026-02-10 09:44:00][cuDensityMat][2429823][Info][cudensitymatResetDistributedConfiguration] Initializing distributed communication service interface [2026-02-10 09:44:00][cuDensityMat][2429822][Info][cudensitymatResetDistributedConfiguration] Initializing distributed communication service interface [2026-02-10 09:44:00][cuDensityMat][2429823][Info][cudensitymatResetDistributedConfiguration] Opening distributed communication service interface library /project/pr_1/micromamba/envs/cuqu_mpi/lib/libcudensitymat_distributed_interface_mpi.so [2026-02-10 09:44:00][cuDensityMat][2429822][Info][cudensitymatResetDistributedConfiguration] Opening distributed communication service interface library /project/pr_1/micromamba/envs/cuqu_mpi/lib/libcudensitymat_distributed_interface_mpi.so [2026-02-10 09:44:00][cuDensityMat][2429823][Info][cudensitymatResetDistributedConfiguration] Loading symbols from the distributed communication service interface library [2026-02-10 09:44:00][cuDensityMat][2429823][Error][cudensitymatResetDistributedConfiguration] The dynamically loaded distributed service interface library has a wrong version: 260110 VS 241110 [2026-02-10 09:44:00][cuDensityMat][2429822][Info][cudensitymatResetDistributedConfiguration] Loading symbols from the distributed communication service interface library [2026-02-10 09:44:00][cuDensityMat][2429822][Error][cudensitymatResetDistributedConfiguration] The dynamically loaded distributed service interface library has a wrong version: 260110 VS 241110

I see a version mismatch there...

I forgot to mention, that I'm using a HPC cluster, that's why I'm using micromamba instead of conda. Also, micromamba can't handle "mpich=*=external_*" during install so I went with openmpi. (I made sure to load openmpi modules according to the HPC docs.)

[b-kloss]

Are you sure you are working in a clean environment? I wasn't able to replicate the failure with the exact same conda package (x86, cuda version 12) in a clean environment.

Your output indicates that you are using an older version of cudensitymat, which expects the lower version of the distributed interface library (241110). cudensitymat v0.4.0 expects version 260110 of the distributed interface, which is also what seems to be present in your environment. Is it possible that your LD_LIBRARY_PATH is pointing to a folder with an older cudensitymat binary?

[b-kloss]

As a side note, conda installs its own openmpi and it won't pick up the system openmpi. While the conda's openmpi package is in principle able to support external installs, the cudensitymat conda-forge package dropped that support with the 26.01 release. If this is a problem for your workflow, please open an issue here.

[SzbKrisztian]

Sorry for the late reply, I also consulted with the cluster administrator. They suggested an environment setup, which uses the host's cray-mpich implementation. With this, I managed to run a test script which uses some basic cupy operations on 2 nodes and 4 gpus/node. On this basis, I extended the environment with cuquantum according to the documentation.

The env setup:

module unload craype
module load cuda/12.9
module load craype-accel-nvidia80
module load miniconda

conda create --prefix /project/pr_1/miniconda/cuqumpich python=3.13

conda activate /project/pr_1/miniconda/cuqumpich

pip install --no-binary=mpi4py mpi4py

conda install cupy

conda install -c conda-forge cuquantum "mpich=*=external_*"

conda install -c conda-forge cuquantum-python

With this setup, the output is

[2026-02-15 18:35:07][cuDensityMat][391204][Api][cudensitymatCreate] handle=0X7FFFD08746A8
[2026-02-15 18:35:07][cuDensityMat][391205][Api][cudensitymatCreate] handle=0X7FFD98217178
[2026-02-15 18:35:07][cuDensityMat][391206][Api][cudensitymatCreate] handle=0X7FFE8C8A6388
[2026-02-15 18:35:07][cuDensityMat][391207][Api][cudensitymatCreate] handle=0X7FFC8A867618
[2026-02-15 18:35:08][cuDensityMat][391207][Api][cudensitymatCreateWorkspace] handle=0X55A8F5670A60 workspaceDescr=0X7FFC8A8678B8
[2026-02-15 18:35:08][cuDensityMat][391205][Api][cudensitymatCreateWorkspace] handle=0X55C2F15F0A10 workspaceDescr=0X7FFD98217418
[2026-02-15 18:35:08][cuDensityMat][391204][Api][cudensitymatCreateWorkspace] handle=0X55886FC60920 workspaceDescr=0X7FFFD0874948
[2026-02-15 18:35:08][cuDensityMat][391206][Api][cudensitymatCreateWorkspace] handle=0X55E0AF61AA70 workspaceDescr=0X7FFE8C8A6628
[2026-02-15 18:35:08][cuDensityMat][391204][Api][cudensitymatResetDistributedConfiguration] handle=0X55886FC60920, provider=1, commPtr=0X7F24871F9360, commSize=4
[2026-02-15 18:35:08][cuDensityMat][391204][Info][cudensitymatResetDistributedConfiguration] Initializing distributed communication service interface
[2026-02-15 18:35:08][cuDensityMat][391204][Info][cudensitymatResetDistributedConfiguration] Opening distributed communication service interface library /project/pr_1/miniconda/cuqumpich/lib/libcudensitymat_distributed_interface_mpi.so
[2026-02-15 18:35:08][cuDensityMat][391205][Api][cudensitymatResetDistributedConfiguration] handle=0X55C2F15F0A10, provider=1, commPtr=0X7F217EECD360, commSize=4
[2026-02-15 18:35:08][cuDensityMat][391205][Info][cudensitymatResetDistributedConfiguration] Initializing distributed communication service interface
[2026-02-15 18:35:08][cuDensityMat][391205][Info][cudensitymatResetDistributedConfiguration] Opening distributed communication service interface library /project/pr_1/miniconda/cuqumpich/lib/libcudensitymat_distributed_interface_mpi.so
[2026-02-15 18:35:08][cuDensityMat][391206][Api][cudensitymatResetDistributedConfiguration] handle=0X55E0AF61AA70, provider=1, commPtr=0X7FD6F8271360, commSize=4
[2026-02-15 18:35:08][cuDensityMat][391206][Info][cudensitymatResetDistributedConfiguration] Initializing distributed communication service interface
[2026-02-15 18:35:08][cuDensityMat][391206][Info][cudensitymatResetDistributedConfiguration] Opening distributed communication service interface library /project/pr_1/miniconda/cuqumpich/lib/libcudensitymat_distributed_interface_mpi.so
[2026-02-15 18:35:08][cuDensityMat][391207][Api][cudensitymatResetDistributedConfiguration] handle=0X55A8F5670A60, provider=1, commPtr=0X7F24A92DD360, commSize=4
[2026-02-15 18:35:08][cuDensityMat][391207][Info][cudensitymatResetDistributedConfiguration] Initializing distributed communication service interface
[2026-02-15 18:35:08][cuDensityMat][391207][Info][cudensitymatResetDistributedConfiguration] Opening distributed communication service interface library /project/pr_1/miniconda/cuqumpich/lib/libcudensitymat_distributed_interface_mpi.so
[2026-02-15 18:35:08][cuDensityMat][391204][Info][cudensitymatResetDistributedConfiguration] WARNING: Unable to open distributed communication service interface library: No distributed communication service!
[2026-02-15 18:35:08][cuDensityMat][391205][Info][cudensitymatResetDistributedConfiguration] WARNING: Unable to open distributed communication service interface library: No distributed communication service!
#WARNING(cuDensityMat): Unable to open distributed communication service interface library: No distributed communication service!
[2026-02-15 18:35:08][cuDensityMat][391206][Info][cudensitymatResetDistributedConfiguration] WARNING: Unable to open distributed communication service interface library: No distributed communication service!
#WARNING(cuDensityMat): Unable to open distributed communication service interface library: No distributed communication service!
[2026-02-15 18:35:08][cuDensityMat][391206][Info][cudensitymatResetDistributedConfiguration] Resetting distributed communicator inside a library context
[2026-02-15 18:35:08][cuDensityMat][391206][Info][cudensitymatResetDistributedConfiguration] Synchronizing distributed communicator via barrier
[2026-02-15 18:35:08][cuDensityMat][391206][Error][cudensitymatResetDistributedConfiguration] Unable to accept a distributed communicator without active distributed service!
[2026-02-15 18:35:08][cuDensityMat][391206][Hint][cudensitymatResetDistributedConfiguration] Make sure $CUDENSITYMAT_COMM_LIB points to the cuDensityMat-MPI interface library.
[2026-02-15 18:35:08][cuDensityMat][391205][Info][cudensitymatResetDistributedConfiguration] Resetting distributed communicator inside a library context
[2026-02-15 18:35:08][cuDensityMat][391205][Info][cudensitymatResetDistributedConfiguration] Synchronizing distributed communicator via barrier
[2026-02-15 18:35:08][cuDensityMat][391205][Error][cudensitymatResetDistributedConfiguration] Unable to accept a distributed communicator without active distributed service!
[2026-02-15 18:35:08][cuDensityMat][391205][Hint][cudensitymatResetDistributedConfiguration] Make sure $CUDENSITYMAT_COMM_LIB points to the cuDensityMat-MPI interface library.

Despite both env variable $CUDENSITYMAT_COMM_LIB and $CUTENSORNET_COMM_LIB points to the correct .so, it complains about them.

Since with this new environment, I have a working script that uses both the gpus and MPI, I really think there's something with the cuquantum lib. It's like the cuqu lib does not know about the host MPI implementation.

[DmitryLyakh]

Are you sure your $CUDENSITYMAT_COMM_LIB path (which in your case is /project/pr_1/miniconda/cuqumpich/lib/libcudensitymat_distributed_interface_mpi.so) is correct? Typically, the communication service library libcudensitymat_distributed_interface_mpi.so resides in the lib/distributed_interfaces folder. In any case, you could just insert ls -l $CUDENSITYMAT_COMM_LIB in your submission script to see during run time whether the communication library is there or not.

[b-kloss]

I am surprised you were able to
conda install -c conda-forge cuquantum "mpich=*=external_*"
since the external builds for mpich have been labeled as broken. I was only able to
conda install -c conda-forge/label/broken cuquantum "mpich=*=external_*"
which forces you into releases <=25.11.1 of the cuQuantum SDK.

Regardless, your logs indicate that cudensitymat distributed interface binary that ships with the conda-forge mpich variants is broken. We will look into this.

As a workaround, you can install cuquantum-python-cuXY (where XY is your cuda major version) via pip and build the distributed interface yourself against your MPI library (following these instructions). After setting MPI_PATH and CUDA_PATH to point to the respective installations, the interface can be built and the CUDENSITYMAT_COMM_LIB environment variable exported via:

source $ENV_PREFIX/lib/python3.11/site-packages/cuquantum/distributed_interfaces/activate_mpi_cudm.sh

where ENV_PREFIX=$CONDA_PREFIX in a conda environment, and otherwise it's the path to your pip environment.

[DmitryLyakh]

Apart from checking the location of the communication service plugin library, since the goal is to use a proprietary cray-mpich library, it will likely require building the distributed interface manually, as recommended in the last paragraph above.

[SzbKrisztian]

Ok, now I got an entirely new kind of error. I'll retrace the steps I made, to be extra sure I'm doing everything fine:

I've created a new environment, with the same commands up to mpi4py install, then
pip install cuquantum-python-cu12
in /project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/distributed_interfaces/ found the activate scripts.
To find CUDA_PATH (sorry for the obvious steps, I just want to make sure I'm not messing up smthing):

$ which nvcc
/opt/software/packages/cuda/12.9/bin/nvcc
$ export CUDA_PATH=/opt/software/packages/cuda/12.9

Then to find MPI_PATH:

$ python -m mpi4py --mpi-library
/opt/cray/pe/lib64/libmpi_cray.so.12
$ ls -l /opt/cray/pe/lib64/libmpi_cray.so.12
lrwxrwxrwx 1 root root 61 Oct 16  2024 /opt/cray/pe/lib64/libmpi_cray.so.12 -> /opt/cray/pe/mpich/8.1.31/ofi/cray/18.0/lib/libmpi_cray.so.12
$ export MPI_PATH=/opt/cray/pe/mpich/8.1.31/ofi/cray/18.0

Then

source activate_mpi_cudm.sh
source activate_mpi_cutn.sh
$ echo $CUDENSITYMAT_COMM_LIB
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/distributed_interfaces/libcudensitymat_distributed_interface_mpi.so
$ echo $CUTENSORNET_COMM_LIB
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/distributed_interfaces/libcutensornet_distributed_interface_mpi.so

I also export _COMM_LIB in the submission file:

#!/bin/bash
#SBATCH --job-name=test
#SBATCH --nodes=1                # use 2 nodes
#SBATCH --ntasks-per-node=4      # 4 MPI ranks / node
#SBATCH --cpus-per-task=1
#SBATCH --gres=gpu:4             # 1 GPU / rank -> 4 GPU / node
#SBATCH --time=00:10:00
#SBATCH --partition=test

module unload craype
module load cuda/12.9
module load craype-accel-nvidia80
module load miniconda


module list

export CONDA_PREFIX=/project/${project}/miniconda
eval "$(conda shell.bash hook)"
conda activate /project/pr_1/miniconda/cuqumpich

export CUTENSORNET_COMM_LIB=/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/distributed_interfaces/libcutensornet_distributed_interface_mpi.so
export CUDENSITYMAT_COMM_LIB=/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/distributed_interfaces/libcudensitymat_distributed_interface_mpi.so

ls -l $CUDENSITYMAT_COMM_LIB
ls -l $CUTENSORNET_COMM_LIB
#srun -n $((SLURM_NNODES * SLURM_NTASKS_PER_NODE)) python test_mpi_cuda.py
CUDENSITYMAT_LOG_LEVEL=5 srun -n $((SLURM_NNODES * SLURM_NTASKS_PER_NODE)) python operator_mpi_example.py

I attached the output, as it's quite lengthy, but the main thing is this part:

process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
/opt/cray/pe/lib64/libmpi_cray.so.12(MPL_backtrace_show+0x26) [0x7f5c7c42911b]
/opt/cray/pe/lib64/libmpi_cray.so.12(+0x1c7a1e4) [0x7f5c7bcb51e4]
/opt/cray/pe/lib64/libmpi_cray.so.12(+0x221fc0a) [0x7f5c7c25ac0a]
/opt/cray/pe/lib64/libmpi_cray.so.12(+0x223c3bf) [0x7f5c7c2773bf]
/opt/cray/pe/lib64/libmpi_cray.so.12(+0x2218327) [0x7f5c7c253327]
/opt/cray/pe/lib64/libmpi_cray.so.12(+0xd58169) [0x7f5c7ad93169]
/opt/cray/pe/lib64/libmpi_cray.so.12(+0xd7ea7e) [0x7f5c7adb9a7e]
/opt/cray/pe/lib64/libmpi_cray.so.12(MPI_Wait+0x526) [0x7f5c7adba496]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/distributed_interfaces/libcudensitymat_distributed_interface_mpi.so(cudensitymatMpiWaitRequest+0x1f) [0x7f5c0400a548]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(+0xb91a1) [0x7f5bbcd701a1]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(+0xd6911) [0x7f5bbcd8d911]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(+0xbb69f) [0x7f5bbcd7269f]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(+0xbc17e) [0x7f5bbcd7317e]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(+0xd6911) [0x7f5bbcd8d911]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(+0xa424b) [0x7f5bbcd5b24b]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(+0x95323) [0x7f5bbcd4c323]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/_internal/../../lib/libcudensitymat.so.0(cudensitymatOperatorComputeAction+0x497) [0x7f5bbcd39dd7]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/cycudensitymat.cpython-313-x86_64-linux-gnu.so(+0x4efa) [0x7f5c7735eefa]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/cudensitymat.cpython-313-x86_64-linux-gnu.so(+0x1eacf) [0x7f5c77383acf]
/project/pr_1/miniconda/cuqumpich/lib/python3.13/site-packages/cuquantum/bindings/cudensitymat.cpython-313-x86_64-linux-gnu.so(+0x1f221) [0x7f5c77384221]
/project/pr_1/miniconda/cuqumpich/bin/python(PyObject_Vectorcall+0x2e) [0x56172dc344ae]
/project/pr_1/miniconda/cuqumpich/bin/python(_PyEval_EvalFrameDefault+0x3532) [0x56172dc45dd2]
/project/pr_1/miniconda/cuqumpich/bin/python(PyEval_EvalCode+0x9f) [0x56172dd0938f]
/project/pr_1/miniconda/cuqumpich/bin/python(+0x2a79f3) [0x56172dd479f3]
/project/pr_1/miniconda/cuqumpich/bin/python(+0x2a4b4c) [0x56172dd44b4c]
/project/pr_1/miniconda/cuqumpich/bin/python(+0x2a1926) [0x56172dd41926]
/project/pr_1/miniconda/cuqumpich/bin/python(+0x2a15a3) [0x56172dd415a3]
/project/pr_1/miniconda/cuqumpich/bin/python(+0x2a13bc) [0x56172dd413bc]
/project/pr_1/miniconda/cuqumpich/bin/python(Py_RunMain+0x3b4) [0x56172dd3fd74]
/project/pr_1/miniconda/cuqumpich/bin/python(Py_BytesMain+0x37) [0x56172dcf48c7]
/lib64/libc.so.6(__libc_start_main+0xe5) [0x7f5ce7a667e5]
/project/pr_1/miniconda/cuqumpich/bin/python(_start+0x2e) [0x56172dcf3c8e]
MPICH ERROR [Rank 0] [job id 13366833.0] [Mon Feb 16 15:34:52 2026] [x1001c4s3b0n1] - Abort(1): Internal error

slurm-13366833.txt

Am I using wrong path? Or something is incompatible?

[DmitryLyakh]

Per what I see, the error occurs inside the MPI_Wait call inside the cray-mpich library. We would need more details to investigate this issue. Do you have any support from the HPC center to clarify what exactly is failing inside their MPI library? Just to be sure, I assume this cray-mpich library build supports GPU pointers (it should).

[DmitryLyakh]

I would also try to run one of the C examples if you have a chance.

[SzbKrisztian]

Just quickly checked mpichversion, and found that it was built with cuda 12.0
MPICH Custom Information: GTL-built-with,ROCM-6.0,CUDA-12.0
Could this be the source of the error?