Debugging produced PTX?

[dbeckwith]

I'm trying to figure out how to produce a debuggable binary from the PTX file that I can debug with the NVIDIA tools. It's looking like I'm going to need a non-release-mode version of the PTX file. Could you could expose whether to pass the --release option to xargo or not? Is this even the right thing to do for trying to debug a CUDA kernel produced with this tool?

[dbeckwith]

I've found that I can get the kernel to pause on an exception and then I can attach cuda-gdb to it, but I still need a PTX file with debug symbols to be able to debug.

[dbeckwith]

Actually, this is pretty easy. I can make a PR.

[denzp]

Could you please tell more about your CUDA debugging experience? I'm curious because I never tried this before even with C++ CUDA.

[dbeckwith]

This is my first go at it as well. Reading the NVIDIA docs it seems pretty straightforward. The best feature I've seen so far is that if you set the environment variable CUDA_DEVICE_WAITS_ON_EXCEPTION, the CUDA kernel will pause for you when it encounters an error and then you can attach a debugging session with cuda-gdb.

However, I'm still having a problem with my application and haven't actually gotten to the point of being able to use the debugger yet. Building with the latest ptx-builder, ptx-linker, and llvm 6.0 in debug mode I still get a linker error (truncated because there was a lot of debug output):

[PTX]            [INFO] Linking without optimisations
[PTX]           LLVM ERROR: Cannot select: 0x7f64bb3f2ea0: f64 = fpow 0x7f64bb3e43a8, 0x7f64bb3e4138
[PTX]             0x7f64bb3e43a8: f64,ch = CopyFromReg 0x7f64bc0c5840, Register:f64 %11
[PTX]               0x7f64bb31d2d8: f64 = Register %11
[PTX]             0x7f64bb3e4138: f64 = fdiv ConstantFP:f64<1.000000e+00>, 0x7f64baa42068
[PTX]               0x7f64bb3e4bc8: f64 = ConstantFP<1.000000e+00>
[PTX]               0x7f64baa42068: f64 = uint_to_fp 0x7f64bb3f2958
[PTX]                 0x7f64bb3f2958: i16 = and 0x7f64bb3f2d00, Constant:i16<255>
[PTX]                   0x7f64bb3f2d00: i16,ch = CopyFromReg 0x7f64bc0c5840, Register:i16 %13
[PTX]                     0x7f64bb17d548: i16 = Register %13
[PTX]                   0x7f64baa42c30: i16 = Constant<255>
[PTX]           In function: _ZN6common4util17sample_hemisphere17h943cfe33e8d056d2E

I believe the line of code it's trying to compile is this:

let z = unsafe { core::intrinsics::powf64(r1, 1. / (n + 1) as f64) };

I don't get this error when I compile in release mode however which is strange. Any ideas?

[dbeckwith]

From playing round a bit, in both release and debug mode, I get linker errors saying it cannot select things like fpow and fcos, which wasn't happening with llvm 5.0. How do I use these math functions in PTX code?

[dbeckwith]

Also, with a very simple kernel that does compile and link in debug mode, when I try to load it the driver gives me CUDA_ERROR_INVALID_PTX. It still works in release mode though. So, to summarize where I'm at right now:

• Kernel with math functions, debug mode: ❌ linker errors
• Kernel with math functions, release mode: ❌ linker errors
• Kernel without math functions, debug mode: ❌ CUDA_ERROR_INVALID_PTX
• Kernel without math functions, release mode: ✔️

[denzp]

I think it worth trying external libm implementation for math functions. fpow issue is kind of expected because there is no pow PTX instruction. A little more details can be found in the tutorial.

About CUDA_ERROR_INVALID_PTX issue, it likely can be solved with ptx-linker. We already make "corrections" to produced LLVM IR, like for example replacing . with _ in globals (because variable named const.3 is not valid for PTX). We need to figure out, what parts of debug assembly cause the issue.

[denzp]

My plan is to finish rust-crate-compile-test first and then we will be able to test ptx-linker more reliably. The linker is the core of Rust's CUDA stack and unfortunately, it currently lacks proper testing.

I'm afraid I need to make debug mode optional for now :(

[dbeckwith]

Thanks, I'll look into using a library to provide the math functions.
Would it help if I provided the full PTX causing CUDA_ERROR_INVALID_PTX?

[denzp]

I think it would be nice if that's possible.

I suspect that even PTX of very small crates can also cause CUDA_ERROR_INVALID_PTX. I've seen this even when compiled a tutorial crate in debug mode.

[dbeckwith]

https://www.dropbox.com/s/kghoah6wobqcx5a/proxy.ptx?dl=1
I've noticed that the PTX file has names with . in them, which you said before was invalid, so maybe that's part of it?