Issues in julia 1.12 with older system?

[evelyne-ringoot]

I am getting some weird issues on a HPC cluster (stampede3), after removing .julia and installing julia 1.12, unset $LD_LIBRARY_PATH, starting a new project and doing ]add oneMKL, I get: (this is with gcc/13.2.0+ onemkl/24.0 loaded)

julia> using oneAPI
terminate called after throwing an instance of 'std::out_of_range'
  what():  bitset::set: __position (which is 4294967288) >= _Nb (which is 9)

[3522716] signal 6 (-6): Aborted
in expression starting at REPL[2]:1
__pthread_kill_implementation at /lib64/libc.so.6 (unknown line)
raise at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__verbose_terminate_handler at /workspace/srcdir/gcc-14.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
__terminate at /workspace/srcdir/gcc-14.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
terminate at /workspace/srcdir/gcc-14.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
__cxa_throw at /workspace/srcdir/gcc-14.2.0/libstdc++-v3/libsupc++/eh_throw.cc:98
__throw_out_of_range_fmt at /workspace/srcdir/gcc-14.2.0/libstdc++-v3/src/c++11/functexcept.cc:101
_ZN3NEO10EngineInfo9mapEngineEPKNS_11IoctlHelperERKNS_18EngineCapabilitiesERKNS_21RootDeviceEnvironmentERPKN10aub_stream10EngineTypeERNS0_14EngineCountersEj at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO15IoctlHelperI91516createEngineInfoEb at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO3Drm15queryEngineInfoEb at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO3Drm6createEOSt10unique_ptrINS_13HwDeviceIdDrmESt14default_deleteIS2_EERNS_21RootDeviceEnvironmentE at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO18initDrmOsInterfaceEOSt10unique_ptrINS_10HwDeviceIdESt14default_deleteIS1_EEjPNS_21RootDeviceEnvironmentE at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEOL23initHwDeviceIdResourcesERNS_20ExecutionEnvironmentEOSt10unique_ptrINS_10HwDeviceIdESt14default_deleteIS3_EEj.lto_priv.0 at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO25prepareDeviceEnvironmentsERNS_20ExecutionEnvironmentERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEj at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN2L09DriverImp10initializeEP12_ze_result_t at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZZNSt9once_flag18_Prepare_executionC4IZSt9call_onceIZN2L09DriverImp10driverInitEvEUlvE_JEEvRS_OT_DpOT0_EUlvE_EERS7_ENUlvE_4_FUNEv at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
__pthread_once_slow at /lib64/libc.so.6 (unknown line)
_ZN2L010initDriverEv at /home1/11134/eringoot/.julia/artifacts/722b403773ba4cd6fe4104cc541d2301b426c8d0/lib/libze_intel_gpu.so.1 (unknown line)
_ZN6loader9context_t11init_driverERNS_8driver_tEjP27_ze_init_driver_type_desc_tP21_ze_global_dditable_tP22_zes_global_dditable_tb at /home1/11134/eringoot/.julia/artifacts/1a96eae820fa2371f1d3e4c8260ee44953d35e35/lib/libze_loader.so (unknown line)
_ZN6loader9context_t13check_driversEjP27_ze_init_driver_type_desc_tP21_ze_global_dditable_tP22_zes_global_dditable_tPbb at /home1/11134/eringoot/.julia/artifacts/1a96eae820fa2371f1d3e4c8260ee44953d35e35/lib/libze_loader.so (unknown line)
zelLoaderDriverCheck at /home1/11134/eringoot/.julia/artifacts/1a96eae820fa2371f1d3e4c8260ee44953d35e35/lib/libze_loader.so (unknown line)
_ZN6ze_lib9context_t4InitEjbP27_ze_init_driver_type_desc_t at /home1/11134/eringoot/.julia/artifacts/1a96eae820fa2371f1d3e4c8260ee44953d35e35/lib/libze_loader.so (unknown line)
_ZZSt9call_onceIZ6zeInitEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv at /home1/11134/eringoot/.julia/artifacts/1a96eae820fa2371f1d3e4c8260ee44953d35e35/lib/libze_loader.so (unknown line)
__pthread_once_slow at /lib64/libc.so.6 (unknown line)
zeInit at /home1/11134/eringoot/.julia/artifacts/1a96eae820fa2371f1d3e4c8260ee44953d35e35/lib/libze_loader.so (unknown line)
macro expansion at /home1/11134/eringoot/.julia/packages/oneAPI/CHy01/lib/level-zero/libze.jl:2150 [inlined]
#zeInit##0 at /home1/11134/eringoot/.julia/packages/GPUToolbox/JLBB1/src/ccalls.jl:34 [inlined]
#check##0 at /home1/11134/eringoot/.julia/packages/oneAPI/CHy01/lib/level-zero/libze.jl:15 [inlined]
retry_reclaim at /home1/11134/eringoot/.julia/packages/oneAPI/CHy01/lib/level-zero/utils.jl:4
check at /home1/11134/eringoot/.julia/packages/oneAPI/CHy01/lib/level-zero/libze.jl:13 [inlined]
zeInit at /home1/11134/eringoot/.julia/packages/GPUToolbox/JLBB1/src/ccalls.jl:33 [inlined]
__init__ at /home1/11134/eringoot/.julia/packages/oneAPI/CHy01/lib/level-zero/oneL0.jl:164
jfptr___init___2713 at /home1/11134/eringoot/.julia/compiled/v1.12/oneAPI/X4ZbC_ovvmr.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
jl_module_run_initializer at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/toplevel.c:68
run_module_init at ./loading.jl:1440
register_restored_modules at ./loading.jl:1428
#_include_from_serialized#867 at ./loading.jl:1316
_include_from_serialized at ./loading.jl:1271 [inlined]
#_require_search_from_serialized#883 at ./loading.jl:2112
_require_search_from_serialized at ./loading.jl:2006 [inlined]
__require_prelocked at ./loading.jl:2624
_require_prelocked at ./loading.jl:2490
macro expansion at ./loading.jl:2418 [inlined]
macro expansion at ./lock.jl:376 [inlined]
__require at ./loading.jl:2383
jfptr___require_62821.1 at /home1/11134/eringoot/julia-1.12.2/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
jl_f_invoke_in_world at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/builtins.c:900
require at ./loading.jl:2359
jfptr_require_66738.1 at /home1/11134/eringoot/julia-1.12.2/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
call_require at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/toplevel.c:494 [inlined]
eval_import_path at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/toplevel.c:532
jl_toplevel_eval_flex at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/toplevel.c:830
ijl_toplevel_eval at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/toplevel.c:1047
eval_body at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/interpreter.c:655
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/interpreter.c:898
jl_toplevel_eval_flex at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/toplevel.c:1035
__repl_entry_eval_expanded_with_loc at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:301
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
jl_f_invokelatest at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/builtins.c:881
toplevel_eval_with_hooks at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:308
toplevel_eval_with_hooks at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:312
toplevel_eval_with_hooks at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:305 [inlined]
eval_user_input at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:330
repl_backend_loop at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:452
#start_repl_backend#41 at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:427
start_repl_backend at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:424 [inlined]
#run_repl#50 at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:653
run_repl at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:639
jfptr_run_repl_19709.1 at /home1/11134/eringoot/julia-1.12.2/share/julia/compiled/v1.12/REPL/u0gqU_DlLf6.so (unknown line)
run_std_repl at ./client.jl:478
jfptr_run_std_repl_36841.1 at /home1/11134/eringoot/julia-1.12.2/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
jl_f_invokelatest at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/builtins.c:881
run_main_repl at ./client.jl:499
repl_main at ./client.jl:586 [inlined]
_start at ./client.jl:561
jfptr__start_47004.1 at /home1/11134/eringoot/julia-1.12.2/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/julia.h:2391 [inlined]
true_main at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/jlapi.c:971
jl_repl_entrypoint at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/src/jlapi.c:1139
main at /cache/build/builder-amdci5-2/julialang/julia-release-1-dot-12/cli/loader_exe.c:58
__libc_start_call_main at /lib64/libc.so.6 (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S
Allocations: 20133225 (Pool: 20130998; Big: 2227); GC: 66
Aborted (core dumped)

Maybe it's some compatibility error with the older intel version on the system?

I tried to load intel modules (24.0) instead of gcc/mkl and got the below:

  ✗ oneAPI
  ✗ Atomix → AtomixoneAPIExt
Precompiling packages finished.
  1 dependency successfully precompiled in 30 seconds. 85 already precompiled.
  2 dependencies errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

julia> err
PkgPrecompileError: The following 1 direct dependency failed to precompile:

oneAPI

Failed to precompile oneAPI [8f75cd03-7ff8-4ecb-9b8f-daf728133b1b] to "/home1/11134/eringoot/.julia/compiled/v1.12/oneAPI/jl_9N7wJX".
ERROR: LoadError: InitError: could not load library "/home1/11134/eringoot/.julia/artifacts/a5555ca7240cdc4c7fc2353452ac51b417d954c8/lib/liboneapi_support.so"
/home1/11134/eringoot/.julia/artifacts/a5555ca7240cdc4c7fc2353452ac51b417d954c8/lib/libmkl_sycl_vm.so.5: undefined symbol: vmsYnI_64
Stacktrace:
  [1] #dlopen#3
    @ ./libdl.jl:120 [inlined]
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:119
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/m2Pjh/src/products/library_generators.jl:63 [inlined]
  [4] __init__()
    @ oneAPI_Support_jll ~/.julia/packages/oneAPI_Support_jll/szLg7/src/wrappers/x86_64-linux-gnu-cxx11.jl:11
  [5] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1440
  [6] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1428
  [7] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}; register::Bool)
    @ Base ./loading.jl:1316
  [8] _include_from_serialized
    @ ./loading.jl:1271 [inlined]
  [9] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:2112
 [10] _require_search_from_serialized
    @ ./loading.jl:2006 [inlined]
 [11] __require_prelocked(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2624
 [12] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2490
 [13] macro expansion
    @ ./loading.jl:2418 [inlined]
 [14] macro expansion
    @ ./lock.jl:376 [inlined]
 [15] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2383
 [16] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2359
 [17] top-level scope
    @ ~/.julia/packages/oneAPI/CHy01/src/oneAPI.jl:19
 [18] include(mod::Module, _path::String)
    @ Base ./Base.jl:306
 [19] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:3021
 [20] top-level scope
    @ stdin:5
 [21] eval(m::Module, e::Any)
    @ Core ./boot.jl:489
 [22] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:2867
 [23] include_string
    @ ./loading.jl:2877 [inlined]
 [24] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:315
 [25] _start()
    @ Base ./client.jl:550
during initialization of module oneAPI_Support_jll
in expression starting at /home1/11134/eringoot/.julia/packages/oneAPI/CHy01/src/oneAPI.jl:1
in expression starting at stdin:

Am I missing something obvious or should one of the above work?

[michel2323]

Deploying oneAPI.jl on a cluster is indeed not straightforward. oneAPI.jl comes prebuilt with the Intel libraries and binaries by default through Julia's artifact and BinaryBuilder.jl system. However, at least on Aurora, the Intel libraries are custom-built for that machine. There is a script to use the local libraries in res/local.jl. Its usage is described in README.md. For Aurora, I use these files for deployment: https://github.com/anlsys/julia_alcf/tree/main/aurora. In the LocalPreferences.toml, you see that I manually set the paths to the local Intel libraries. The res/local.jl script would not discover all.

Anyway, this is all WIP, feel free to reach out!

You'll have the best out-of-the-box experience with a recent Linux distro on a consumer discrete GPU.

[evelyne-ringoot]

Hi @michel2323 sorry for the late reply, but I finally got this to (partially) work on Stampede after some more experimenting, the files you linked were superhelpful, thank you so much!!

[michel2323]

@evelyne-ringoot Glad to hear! Feel free to get in touch with me. There are still issues that I'm in the process of fixing. So it's still WIP, but I hope to reach something user-friendly without critical bugs in the coming months.