- How to attach the debugger to running nekrs job - How to print stacktraces when the job hangs - How to correctly identify if a kernel failed (CUDA_LAUNCH_BLOCKING=1)? - Useful tools: valgrind, compute-sanitizer - Potential workarounds: UCX_MEMTYPE_CACHE=n, OMPI_MCA_io=romio341 etc.