Linking with egadsliteCuda

[Angelyr]

I have pumi-pic working with the official egads library, but I am running into an issue while linking with egadsliteCuda that I have not been able to fix.

When I attempt to link pumi-pic to egadsliteCuda I am receiving this error multiple times:

[ 92%] Linking CXX executable particle_adapt
nvlink error   : Undefined reference to 'kokkos_impl_cuda_constant_memory_buffer' in 'CMakeFiles/particle_adapt.dir/particle_adapt.cpp.o'
gmake[2]: *** [test/CMakeFiles/particle_adapt.dir/build.make:119: test/particle_adapt] Error 255
gmake[1]: *** [CMakeFiles/Makefile2:1224: test/CMakeFiles/particle_adapt.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

I believe the error is related to using this flag in omega_h, CUDA_SEPARABLE_COMPILATION. Which I have enabled in egadsliteCuda, kokkos, and omega_h. I believe this is required for usage of egadsliteCuda

Things I have tried:

1. Adding CUDA_SEPARABLE_COMPILATION and CUDA_RESOLVE_DEVICE_SYMBOLS to pumi-pic (this didn't seem to have any effect)
2. Adding CUDA_RESOLVE_DEVICE_SYMBOLS to Omega_h (this made the error occur more often)
3. Building with CUDA_SEPARABLE_COMPILATION in Kokkos, but egadsliteCuda disabled in Omega_h (this compiled successfully)

Here is the verbose output:

[ 92%] Linking CXX executable particle_adapt
cd /space/castia5/build-dev-pumi-pic/pumi-pic/test && /opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/cmake-3.26.3-2duxfcdiykykfynkadpp2m2y6zrmuvz4/bin/cmake -E cmake_link_script CMakeFiles/particle_adapt.dir/link.txt --verbose=1
/opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/mpich-4.1.1-xpoyz4tqgfxtrm6m7qq67q4ccp5pnlre/bin/mpicxx -g -DKOKKOS_DEPENDENCE --relocatable-device-code=true -arch=sm_86 CMakeFiles/particle_adapt.dir/particle_adapt.cpp.o -o particle_adapt  -Wl,-rpath,/opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/zlib-1.2.13-mjocrm2cwyth6kvpmj3yrg52ulwd64ow/lib /space/castia5/build-dev-pumi-pic/omega_h-egads-cuda/install/lib64/libomega_h.a ../src/libpumipic-core.a /space/castia5/build-dev-pumi-pic/omega_h-egads-cuda/install/lib64/libomega_h.a /space/castia5/build-dev-pumi-pic/libMeshb/install/lib/libMeshb.7.a /space/castia5/build-dev-pumi-pic/omega_h-egads-cuda/install/lib64/libohEgadslite.a /space/castia5/build-dev-pumi-pic/egadsliteCuda/install/lib64/libegadslite.a -lcuda /opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/zlib-1.2.13-mjocrm2cwyth6kvpmj3yrg52ulwd64ow/lib/libz.so ../particle_structs/src/libparticleStructs.a ../support/libsupport.a /space/castia5/build-dev-pumi-pic/kokkos/install/lib64/libkokkoscontainers.a /space/castia5/build-dev-pumi-pic/kokkos/install/lib64/libkokkoscore.a /usr/lib64/libcuda.so /opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/cuda-12.1.1-zxa4mskqvbkiefzkvnuatlq7skxjnzt6/lib64/libcudart.so -ldl /space/castia5/build-dev-pumi-pic/kokkos/install/lib64/libkokkossimd.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libengpar.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libdiffusive.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libmultilevel.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libengpar_metrics.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libcoloring.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libagi.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libengpar_support.a /space/castia5/build-dev-pumi-pic/engpar/install/lib/libpcu.a 
nvlink error   : Undefined reference to 'kokkos_impl_cuda_constant_memory_buffer' in 'CMakeFiles/particle_adapt.dir/particle_adapt.cpp.o'
gmake[2]: *** [test/CMakeFiles/particle_adapt.dir/build.make:119: test/particle_adapt] Error 255
gmake[2]: Leaving directory '/space/castia5/build-dev-pumi-pic/pumi-pic'
gmake[1]: *** [CMakeFiles/Makefile2:1224: test/CMakeFiles/particle_adapt.dir/all] Error 2
gmake[1]: Leaving directory '/space/castia5/build-dev-pumi-pic/pumi-pic'
gmake: *** [Makefile:146: all] Error 2

[Angelyr]

Fixed by removing unnecessary usages of cuda

[cwsmith]

Nice! What calls were the problem?

[Angelyr]

The issue was this line:
set_source_files_properties(Omega_h_egads_lite.cpp PROPERTIES LANGUAGE CUDA)

I believe since the file was written in c++ and didn't contain any cuda code.