Skip to content

Installation docs don't verify MultiScaleDeformableAttention CUDA op is built — silent CPU-only fallback on HPC #36

Description

@ffhammer

The current installation guide (https://hugobaudchon.github.io/CanopyRS/getting-started/installation/) does not verify that detrex's multi_scale_deform_attn CUDA extension is actually compiled with GPU support. On HPC systems this fails silently: the install completes, torch.cuda.is_available() returns True, but inference crashes only once the model runs:

RuntimeError: Not compiled with GPU support
  File ".../detrex/layers/multi_scale_deform_attn.py", line 56, in forward
    output = _C.ms_deform_attn_forward(...)

Root cause: on many HPC setups (e.g. ETH Euler) the conda env ships torch with cu126 but no nvcc / CUDA_HOME, so detrex's setup.py falls back to a CPU-only build without erroring. After ~2h of tile preprocessing the pipeline dies at the first forward pass.

Suggested doc additions:

  1. State explicitly that nvcc matching torch.version.cuda must be available before installing detrex, and that build must happen on a GPU node.
  2. Add a pre-install check block:
    nvcc --version                 # must match torch.version.cuda
    nvidia-smi                     # GPU visible
    echo $CUDA_HOME                # set
    
    With a fix for the common conda case:
    conda install -c nvidia/label/cuda-12.6.0 cuda-nvcc cuda-cudart-dev cuda-libraries-dev
    export CUDA_HOME=$CONDA_PREFIX FORCE_CUDA=1 TORCH_CUDA_ARCH_LIST="8.9"
    
  3. Add a post-install verification step users can run in seconds:
    python -c "
    import torch
    from detrex.layers.multi_scale_deform_attn import MultiScaleDeformableAttnFunction
    from detrex.layers import _C
    assert hasattr(_C, 'ms_deform_attn_forward')
    print('detrex CUDA op OK')
    "
    

Environment where this occurred:

  • ETH Euler, RTX 4090 node
  • conda env, torch 2.7.1+cu126, driver 580 / CUDA 13.0
  • nvcc missing until installed manually into the env

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions