CUDA free failed: cudaErrorInvalidAdressSpace

[Bjonze]

Dear Enya,
I'm a masters student from Technical University of Denmark who is trying to use your framework for a project of mine.
I don't have much experience with TensorFlow so I'm using your Pytorch implementation.
I'm running the PicassoNet-II on a GPU-cluster operating on on a Ubuntu 18.04.6 LTS GPU cluster and i encounter the following error:

terminate called after throwing an instance of 'thrust::system::system_error'
what(): CUDA free failed: cudaErrorInvalidAddressSpace: operation not supported on global/shared address space

I believe the error comes from the decimate_gpu.cu however I'm not entirely sure why or how.
Can you help with this one?

Best,
Bjonze

[EnyaHermite]

Hi Bjonze.

may I know the type of your GPU, and other configurations such as drive version, cuda+cudnn versions? also which pytorch version you are using?

The code is currently tested on single GPU usage scenario (2080/3090 GPU). To help, can I share the data and code you are using (e.g. via email) to repeat the error?

best
Huan

[Bjonze]

Hello EnyaHermite,
The GPU is a NVIDIA GeForce GTX TITAN X
Cuda version 11.3
Torch version 1.11.0.
The NVIDIA is using driver version 470.103.01
Can you let me know which drivers in particular you want me to check?

I have only made slight modifications to the code, in particular the training script and dataloader.
I'm currently running the pytorch implementation on the BU3DFE dataset which means that the meshes are manifold however they are not watertight. I don't know if this is what's causing issue regarding neighbourhoods in the QEM decimation.

I'm also encountering the error seen below which is very similar to the one i opened the isssue about.

I suspect the error stems from somewhere between line 706 and below in the decimate_gpu.cu

I have added .to(self.device) where self.device = "cuda" at multiple lines throughout the code.

I will send you an email with a data example (which also generates the error) and the code I momentarily.

Thank you very much for your help.

Best,
Bjonze