Skip to content

[Bug] gen_cubins.py: XQA cubin generation writes to the module-level default path instead of --output_dir when workers do not inherit __main__ state #161

Description

@Hi5808

Platform: Jetson Orin Nano 8GB (Seeed reComputer J3011), JetPack 7.2 / L4T
R39.2.0, custom Yocto-built minimal OS (not stock JetPack), CUDA 13.2. Native
aarch64 cmake build.

Build command:

cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr \
         -DEMBEDDED_TARGET=jetson-orin -DCUDA_CTK_VERSION=13.2 -DENABLE_CUTE_DSL=ALL
make

Symptom

The build fails during XQA cubin generation:

ptxas fatal   : Output file '../../cpp/kernels/decodeAttentionKernels/cubin/xqa_kernel_....cubin' could not be opened

Note the path: it is the relative module-level default, not the absolute
--output_dir CMake passes in.

What I think is happening

Line numbers are against main as of 2026-08-04 (kernelSrcs/xqa/gen_cubins.py).

cubin_dir is defined at module scope with a relative default:

# line 81
cubin_dir = "../../cpp/kernels/decodeAttentionKernels/cubin/"

It is rebound to the resolved output directory, but only inside the __main__
block:

# line 705
if __name__ == "__main__":
    ...
    # line 709
    cubin_dir = os.path.abspath(args.output_dir)

The directory that gets created is that resolved path:

# lines 829-831
if os.path.exists(cubin_dir):
    shutil.rmtree(cubin_dir)
os.makedirs(cubin_dir)

But the value the workers actually use is read at module scope, inside
build_commands() (line 394, for the nvcc -o argument) and
save_cubin_cpp_file() (line 414), both reached from run_cubin_gen(), which is
dispatched to a pool:

# lines 835-836
with multiprocessing.Pool(processes=thread_count) as pool:
    name_size_list = pool.map(run_cubin_gen, arch_macro_lists)

Under the fork start method this is fine — children inherit the parent's memory
after line 709 has run, so they see the resolved path. Under a start method that
re-imports the module in the child (spawn, forkserver), __name__ is not
"__main__" there, line 709 never executes, and the workers fall back to the
line 81 relative default — which is not the directory os.makedirs created, and
generally will not exist relative to the worker's cwd. That matches the failure
exactly: ptxas reporting the relative path as unopenable.

What I could not confirm

I have not reproduced this under fork, and I don't believe it can occur
there.
Checking after the fact, the two Python installs I still have access to
(3.12.3 on the device, 3.13.13 on my host) both default to fork, where I would
expect the current code to work. I no longer have the exact build container from
the failing run and did not record its Python version, so I can't state which
start method was in play — I'm inferring it from the relative path in the error.

Since Python 3.14 changes the default start method on Linux to forkserver, this
would begin affecting fork-based setups on newer interpreters even where it
works today, which is the main reason I'm reporting it despite the incomplete
reproduction.

Questions

  1. Is the module-level cubin_dir at line 81 intended as a real fallback, or is
    it vestigial and only ever meant to be overridden by --output_dir?
  2. If the latter, would you accept a change that makes the resolved path explicit
    to the workers — passing it as part of the pool.map payload, or setting it
    via a Pool(initializer=...) — so the behaviour no longer depends on the
    start method?

Happy to test a patch on this hardware, or to provide the full cmake/make logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions