Platform: Jetson Orin Nano 8GB (Seeed reComputer J3011), JetPack 7.2 / L4T
R39.2.0, custom Yocto-built minimal OS (not stock JetPack), CUDA 13.2. Native
aarch64 cmake build.
Build command:
cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr \
-DEMBEDDED_TARGET=jetson-orin -DCUDA_CTK_VERSION=13.2 -DENABLE_CUTE_DSL=ALL
make
Symptom
The build fails during XQA cubin generation:
ptxas fatal : Output file '../../cpp/kernels/decodeAttentionKernels/cubin/xqa_kernel_....cubin' could not be opened
Note the path: it is the relative module-level default, not the absolute
--output_dir CMake passes in.
What I think is happening
Line numbers are against main as of 2026-08-04 (kernelSrcs/xqa/gen_cubins.py).
cubin_dir is defined at module scope with a relative default:
# line 81
cubin_dir = "../../cpp/kernels/decodeAttentionKernels/cubin/"
It is rebound to the resolved output directory, but only inside the __main__
block:
# line 705
if __name__ == "__main__":
...
# line 709
cubin_dir = os.path.abspath(args.output_dir)
The directory that gets created is that resolved path:
# lines 829-831
if os.path.exists(cubin_dir):
shutil.rmtree(cubin_dir)
os.makedirs(cubin_dir)
But the value the workers actually use is read at module scope, inside
build_commands() (line 394, for the nvcc -o argument) and
save_cubin_cpp_file() (line 414), both reached from run_cubin_gen(), which is
dispatched to a pool:
# lines 835-836
with multiprocessing.Pool(processes=thread_count) as pool:
name_size_list = pool.map(run_cubin_gen, arch_macro_lists)
Under the fork start method this is fine — children inherit the parent's memory
after line 709 has run, so they see the resolved path. Under a start method that
re-imports the module in the child (spawn, forkserver), __name__ is not
"__main__" there, line 709 never executes, and the workers fall back to the
line 81 relative default — which is not the directory os.makedirs created, and
generally will not exist relative to the worker's cwd. That matches the failure
exactly: ptxas reporting the relative path as unopenable.
What I could not confirm
I have not reproduced this under fork, and I don't believe it can occur
there. Checking after the fact, the two Python installs I still have access to
(3.12.3 on the device, 3.13.13 on my host) both default to fork, where I would
expect the current code to work. I no longer have the exact build container from
the failing run and did not record its Python version, so I can't state which
start method was in play — I'm inferring it from the relative path in the error.
Since Python 3.14 changes the default start method on Linux to forkserver, this
would begin affecting fork-based setups on newer interpreters even where it
works today, which is the main reason I'm reporting it despite the incomplete
reproduction.
Questions
- Is the module-level
cubin_dir at line 81 intended as a real fallback, or is
it vestigial and only ever meant to be overridden by --output_dir?
- If the latter, would you accept a change that makes the resolved path explicit
to the workers — passing it as part of the pool.map payload, or setting it
via a Pool(initializer=...) — so the behaviour no longer depends on the
start method?
Happy to test a patch on this hardware, or to provide the full cmake/make logs.
Platform: Jetson Orin Nano 8GB (Seeed reComputer J3011), JetPack 7.2 / L4T
R39.2.0, custom Yocto-built minimal OS (not stock JetPack), CUDA 13.2. Native
aarch64 cmake build.
Build command:
Symptom
The build fails during XQA cubin generation:
Note the path: it is the relative module-level default, not the absolute
--output_dirCMake passes in.What I think is happening
Line numbers are against
mainas of 2026-08-04 (kernelSrcs/xqa/gen_cubins.py).cubin_diris defined at module scope with a relative default:It is rebound to the resolved output directory, but only inside the
__main__block:
The directory that gets created is that resolved path:
But the value the workers actually use is read at module scope, inside
build_commands()(line 394, for thenvcc -oargument) andsave_cubin_cpp_file()(line 414), both reached fromrun_cubin_gen(), which isdispatched to a pool:
Under the
forkstart method this is fine — children inherit the parent's memoryafter line 709 has run, so they see the resolved path. Under a start method that
re-imports the module in the child (
spawn,forkserver),__name__is not"__main__"there, line 709 never executes, and the workers fall back to theline 81 relative default — which is not the directory
os.makedirscreated, andgenerally will not exist relative to the worker's cwd. That matches the failure
exactly:
ptxasreporting the relative path as unopenable.What I could not confirm
I have not reproduced this under
fork, and I don't believe it can occurthere. Checking after the fact, the two Python installs I still have access to
(3.12.3 on the device, 3.13.13 on my host) both default to
fork, where I wouldexpect the current code to work. I no longer have the exact build container from
the failing run and did not record its Python version, so I can't state which
start method was in play — I'm inferring it from the relative path in the error.
Since Python 3.14 changes the default start method on Linux to
forkserver, thiswould begin affecting
fork-based setups on newer interpreters even where itworks today, which is the main reason I'm reporting it despite the incomplete
reproduction.
Questions
cubin_dirat line 81 intended as a real fallback, or isit vestigial and only ever meant to be overridden by
--output_dir?to the workers — passing it as part of the
pool.mappayload, or setting itvia a
Pool(initializer=...)— so the behaviour no longer depends on thestart method?
Happy to test a patch on this hardware, or to provide the full cmake/make logs.