The current installation guide (https://hugobaudchon.github.io/CanopyRS/getting-started/installation/) does not verify that detrex's multi_scale_deform_attn CUDA extension is actually compiled with GPU support. On HPC systems this fails silently: the install completes, torch.cuda.is_available() returns True, but inference crashes only once the model runs:
RuntimeError: Not compiled with GPU support
File ".../detrex/layers/multi_scale_deform_attn.py", line 56, in forward
output = _C.ms_deform_attn_forward(...)
Root cause: on many HPC setups (e.g. ETH Euler) the conda env ships torch with cu126 but no nvcc / CUDA_HOME, so detrex's setup.py falls back to a CPU-only build without erroring. After ~2h of tile preprocessing the pipeline dies at the first forward pass.
Suggested doc additions:
- State explicitly that
nvcc matching torch.version.cuda must be available before installing detrex, and that build must happen on a GPU node.
- Add a pre-install check block:
nvcc --version # must match torch.version.cuda
nvidia-smi # GPU visible
echo $CUDA_HOME # set
With a fix for the common conda case:
conda install -c nvidia/label/cuda-12.6.0 cuda-nvcc cuda-cudart-dev cuda-libraries-dev
export CUDA_HOME=$CONDA_PREFIX FORCE_CUDA=1 TORCH_CUDA_ARCH_LIST="8.9"
- Add a post-install verification step users can run in seconds:
python -c "
import torch
from detrex.layers.multi_scale_deform_attn import MultiScaleDeformableAttnFunction
from detrex.layers import _C
assert hasattr(_C, 'ms_deform_attn_forward')
print('detrex CUDA op OK')
"
Environment where this occurred:
- ETH Euler, RTX 4090 node
- conda env, torch 2.7.1+cu126, driver 580 / CUDA 13.0
- nvcc missing until installed manually into the env
The current installation guide (https://hugobaudchon.github.io/CanopyRS/getting-started/installation/) does not verify that detrex's
multi_scale_deform_attnCUDA extension is actually compiled with GPU support. On HPC systems this fails silently: the install completes,torch.cuda.is_available()returns True, but inference crashes only once the model runs:Root cause: on many HPC setups (e.g. ETH Euler) the conda env ships torch with cu126 but no
nvcc/CUDA_HOME, so detrex's setup.py falls back to a CPU-only build without erroring. After ~2h of tile preprocessing the pipeline dies at the first forward pass.Suggested doc additions:
nvccmatchingtorch.version.cudamust be available before installing detrex, and that build must happen on a GPU node.Environment where this occurred: