Skip to content

torch.compile causes undefined symbol: cuModuleGetFunction with distrobox #369

@inikishev

Description

@inikishev

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

I created a fresh new environment:

conda create -n testcompile -c conda-forge python=3.12.* pytorch-gpu torchvision cuda-compiler

I deleted cache folders - /tmp/torchinductor_jj and /home/jj/distrobox/fedora/.triton/

So I have pytorch and CUDA works in it. But when I run any torch.compile-d function, on cpu or cuda, it causes the following exception:

BackendCompilerFailed: backend='inductor' raised:
ImportError: /var/home/jj/distrobox/fedora/.triton/cache/q4oIpkjOtdHHfi8xBkm4jC4JWIk5AjKtN8WRkZb8MD8/cuda_utils.so: undefined symbol: cuModuleGetFunction

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

for example I run this and it causes the exception:

import site
import torch

print(f'{site.getsitepackages() = }') # make sure corrent env is selected
print(f'{torch.cuda.is_available() = }') # check if CUDA is enabled

@torch.compile
def square(x):
  return x ** 2

x = torch.randn(1, device='cpu')
square(x)

I also tried

conda create -n testcompile2 -c conda-forge python=3.12.* pytorch-gpu torchvision cuda-compiler=12.6

but I get the same "cuModuleGetFunction" error.

torch.compile does work on pip version of torch (but I have issues with some other unrelated packages on it). I also tried doing this in ubuntu distrobox instead of fedora, and got the same issue.

Installed packages

# packages in environment at /var/home/jj/distrobox/fedora/miniconda3/envs/testcompile:
#
# Name                    Version                   Build  Channel
_openmp_mutex             4.5                  3_kmp_llvm    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
binutils                  2.43                 h4852527_4    conda-forge
binutils_impl_linux-64    2.43                 h4bf12b8_4    conda-forge
binutils_linux-64         2.43                 h4852527_4    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-compiler                1.9.0                h2b85faf_0    conda-forge
ca-certificates           2025.1.31            hbcca054_0    conda-forge
cpython                   3.12.9          py312hd8ed1ab_1    conda-forge
cuda-cccl_linux-64        12.8.90              ha770c72_1    conda-forge
cuda-compiler             12.8.1               hbad6d8a_0    conda-forge
cuda-crt-dev_linux-64     12.8.93              ha770c72_0    conda-forge
cuda-crt-tools            12.8.93              ha770c72_0    conda-forge
cuda-cudart               12.8.90              h5888daf_1    conda-forge
cuda-cudart-dev           12.8.90              h5888daf_1    conda-forge
cuda-cudart-dev_linux-64  12.8.90              h3f2d84a_1    conda-forge
cuda-cudart-static        12.8.90              h5888daf_1    conda-forge
cuda-cudart-static_linux-64 12.8.90              h3f2d84a_1    conda-forge
cuda-cudart_linux-64      12.8.90              h3f2d84a_1    conda-forge
cuda-cuobjdump            12.8.90              hbd13f7d_0    conda-forge
cuda-cupti                12.8.90              hbd13f7d_0    conda-forge
cuda-cuxxfilt             12.8.90              hbd13f7d_0    conda-forge
cuda-driver-dev_linux-64  12.8.90              h3f2d84a_1    conda-forge
cuda-nvcc                 12.8.93              hcdd1206_0    conda-forge
cuda-nvcc-dev_linux-64    12.8.93              he91c749_0    conda-forge
cuda-nvcc-impl            12.8.93              h85509e4_0    conda-forge
cuda-nvcc-tools           12.8.93              he02047a_0    conda-forge
cuda-nvcc_linux-64        12.8.93              h04802cd_0    conda-forge
cuda-nvdisasm             12.8.90              hbd13f7d_0    conda-forge
cuda-nvprune              12.8.90              hbd13f7d_0    conda-forge
cuda-nvrtc                12.8.93              hbd13f7d_0    conda-forge
cuda-nvtx                 12.8.90              hbd13f7d_0    conda-forge
cuda-nvvm-dev_linux-64    12.8.93              ha770c72_0    conda-forge
cuda-nvvm-impl            12.8.93              he02047a_0    conda-forge
cuda-nvvm-tools           12.8.93              he02047a_0    conda-forge
cuda-version              12.8                 h5d125a7_3    conda-forge
cudnn                     9.8.0.87             h81d5506_0    conda-forge
cusparselt                0.7.0.0              hcd2ec93_0    conda-forge
cxx-compiler              1.9.0                h1a2810e_0    conda-forge
filelock                  3.17.0             pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fsspec                    2025.3.0           pyhd8ed1ab_0    conda-forge
gcc                       13.3.0               h9576a4e_2    conda-forge
gcc_impl_linux-64         13.3.0               h1e990d8_2    conda-forge
gcc_linux-64              13.3.0               hc28eda2_8    conda-forge
giflib                    5.2.2                hd590300_0    conda-forge
gmp                       6.3.0                hac33072_2    conda-forge
gmpy2                     2.1.5           py312h7201bc8_3    conda-forge
gxx                       13.3.0               h9576a4e_2    conda-forge
gxx_impl_linux-64         13.3.0               hae580e1_2    conda-forge
gxx_linux-64              13.3.0               h6834431_8    conda-forge
icu                       75.1                 he02047a_0    conda-forge
jinja2                    3.1.6              pyhd8ed1ab_0    conda-forge
kernel-headers_linux-64   3.10.0              he073ed8_18    conda-forge
lcms2                     2.17                 h717163a_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_4    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20240722.0      cxx17_hbbce691_4    conda-forge
libblas                   3.9.0           31_hfdb39a5_mkl    conda-forge
libcap                    2.71                 h39aace5_0    conda-forge
libcblas                  3.9.0           31_h372d94f_mkl    conda-forge
libcublas                 12.8.4.1             h9ab20c4_0    conda-forge
libcudss0                 0.4.0.2              he55f5cd_2    conda-forge
libcufft                  11.3.3.83            hbd13f7d_0    conda-forge
libcufile                 1.13.1.3             h12f29b5_0    conda-forge
libcurand                 10.3.9.90            hbd13f7d_0    conda-forge
libcusolver               11.7.3.90            h9ab20c4_0    conda-forge
libcusparse               12.5.8.93            hbd13f7d_0    conda-forge
libdeflate                1.23                 h4ddbbb0_0    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.6                h2dba641_0    conda-forge
libgcc                    14.2.0               h767d61c_2    conda-forge
libgcc-devel_linux-64     13.3.0             hc03c837_102    conda-forge
libgcc-ng                 14.2.0               h69a702a_2    conda-forge
libgcrypt-lib             1.11.0               hb9d3cd8_2    conda-forge
libgomp                   14.2.0               h767d61c_2    conda-forge
libgpg-error              1.51                 hbd13f7d_1    conda-forge
libhwloc                  2.11.2          default_h0d58e46_1001    conda-forge
libiconv                  1.18                 h4ce23a2_1    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           31_hc41d3b0_mkl    conda-forge
libllvm20                 20.1.0.rc1           hb8a341e_0    conda-forge
liblzma                   5.6.4                hb9d3cd8_0    conda-forge
libmagma                  2.8.0                h566cb83_2    conda-forge
libnl                     3.11.0               hb9d3cd8_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjitlink              12.8.93              hbd13f7d_0    conda-forge
libnvjpeg                 12.3.5.92            h97fd463_0    conda-forge
libpng                    1.6.47               h943b412_0    conda-forge
libprotobuf               5.28.3               h6128344_1    conda-forge
libsanitizer              13.3.0               he8ea267_2    conda-forge
libsqlite                 3.49.1               hee588c1_1    conda-forge
libstdcxx                 14.2.0               h8f9b012_2    conda-forge
libstdcxx-devel_linux-64  13.3.0             hc03c837_102    conda-forge
libstdcxx-ng              14.2.0               h4852527_2    conda-forge
libsystemd0               257.3                h3dc2cb9_0    conda-forge
libtiff                   4.7.0                hd9ff511_3    conda-forge
libtorch                  2.6.0           cuda126_mkl_h8247c52_300    conda-forge
libudev1                  257.3                h9a4d06a_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libuv                     1.50.0               hb9d3cd8_0    conda-forge
libwebp-base              1.5.0                h851e524_0    conda-forge
libxcb                    1.17.0               h8a09558_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.13.6               h8d12d68_0    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
llvm-openmp               19.1.7               h024ca30_0    conda-forge
lz4-c                     1.10.0               h5888daf_1    conda-forge
markupsafe                3.0.2           py312h178313f_1    conda-forge
mkl                       2024.2.2            ha957f24_16    conda-forge
mpc                       1.3.1                h24ddda3_1    conda-forge
mpfr                      4.2.1                h90cbb55_3    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_1    conda-forge
nccl                      2.25.1.1             ha44e49d_0    conda-forge
ncurses                   6.5                  h2d0b736_3    conda-forge
networkx                  3.4.2              pyh267e887_2    conda-forge
numpy                     2.2.3           py312h72c5963_0    conda-forge
openjpeg                  2.5.3                h5fbd93e_0    conda-forge
openssl                   3.4.1                h7b32b05_0    conda-forge
optree                    0.14.1          py312h68727a3_0    conda-forge
pillow                    11.1.0          py312h80c1187_0    conda-forge
pip                       25.0.1             pyh8b19718_0    conda-forge
pthread-stubs             0.4               hb9d3cd8_1002    conda-forge
pybind11                  2.13.6             pyh1ec8472_2    conda-forge
pybind11-global           2.13.6             pyh415d2e4_2    conda-forge
python                    3.12.9          h9e4cc4f_1_cpython    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytorch                   2.6.0           cuda126_mkl_py312_h8bb5dc9_300    conda-forge
pytorch-gpu               2.6.0           cuda126_mkl_ha999a5f_300    conda-forge
rdma-core                 56.0                 h5888daf_0    conda-forge
readline                  8.2                  h8c095d6_2    conda-forge
setuptools                75.8.2             pyhff2d567_0    conda-forge
sleef                     3.8                  h1b44611_0    conda-forge
sympy                     1.13.3           pyh2585a3b_105    conda-forge
sysroot_linux-64          2.17                h0157908_18    conda-forge
tbb                       2021.13.0            hceb3a55_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
torchvision               0.21.0          cuda126_py312_h3e02b30_0    conda-forge
triton                    3.2.0           cuda126py312h5a3d8a8_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_1    conda-forge
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tzdata                    2025a                h78e105d_0    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
xorg-libxau               1.0.12               hb9d3cd8_0    conda-forge
xorg-libxdmcp             1.1.5                hb9d3cd8_0    conda-forge
zstd                      1.5.7                hb8e6e7a_1    conda-forge

Environment info

active environment : testcompile
    active env location : /var/home/jj/distrobox/fedora/miniconda3/envs/testcompile
            shell level : 2
       user config file : /var/home/jj/distrobox/fedora/.condarc
 populated config files : /var/home/jj/distrobox/fedora/miniconda3/.condarc
                          /var/home/jj/distrobox/fedora/.condarc
          conda version : 25.1.1
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=25.1.1=0
                          __cuda=12.8=0
                          __glibc=2.40=0
                          __linux=6.13.5=0
                          __unix=0=0
       base environment : /var/home/jj/distrobox/fedora/miniconda3  (writable)
      conda av data dir : /var/home/jj/distrobox/fedora/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /var/home/jj/distrobox/fedora/miniconda3/pkgs
                          /var/home/jj/distrobox/fedora/.conda/pkgs
       envs directories : /var/home/jj/distrobox/fedora/miniconda3/envs
                          /var/home/jj/distrobox/fedora/.conda/envs
               platform : linux-64
             user-agent : conda/25.1.1 requests/2.32.3 CPython/3.12.8 Linux/6.13.5-102.bazzite.fc41.x86_64 fedora/41 glibc/2.40 solver/libmamba conda-libmamba-solver/25.1.1 libmambapy/2.0.5 aau/0.6.1 c/. s/. e/.
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions