feat: add nvidia-tegra-nvgpu package for Jetson Orin (GA10B)

[schwankner]

Summary

Adds the OE4T-patched GPU driver stack for NVIDIA Jetson Orin NX (Tegra234 / GA10B) as a Talos kernel package.

Hardware: Jetson Orin NX 16GB (Tegra234 / GA10B)
Tested with: Talos v1.13, kernel 6.18, CUDA 12.6

Modules

Module	Source	Purpose
host1x.ko	OE4T linux-nv-oot (wip-r36.5-take-2)	Syncpoint API with GA10B support + syncpt[0] reservation fix
host1x-fence.ko	OE4T linux-nv-oot	DMA fence bridge for syncpoints
nvmap.ko	OE4T linux-nv-oot	GPU memory allocator
mc-utils.ko	OE4T linux-nv-oot	EMC frequency management
nvhost-ctrl-shim.ko	embedded .c (this pkg)	/dev/nvhost-ctrl for JetPack 6 CUDA runtime
nvgpu.ko	OE4T linux-nvgpu	Main GA10B GPU driver

nvhost-ctrl-shim

The JetPack 6 CUDA runtime (libnvrm_host1x.so) requires /dev/nvhost-ctrl for cudaStreamSynchronize. Without it, CUDA falls back to CPU semaphore polling (~7 tok/s vs ~60 tok/s for LLM inference on Orin NX 16GB).

The shim bridges 8 ioctls (GET_VERSION, SYNCPT_READ, SYNCPT_READ_MAX, SYNCPT_WAITMEX, SYNC_FENCE_CREATE, GET_CHARACTERISTICS, POLL_FD_CREATE, SYNC_FILE_EXTRACT) to the OE4T host1x syncpoint API using dma_fence for interrupt-driven waits.

The source (nvhost_ctrl_shim.c) is embedded directly in the package directory and made available at /pkg/ by bldr at build time.

host1x shadow path

OE4T host1x.ko is installed at kernel/drivers/gpu/host1x/ to shadow the in-tree module. The squashfs overlay replaces the built-in host1x so nvgpu's CONFIG_TEGRA_GK20A_NVHOST=y path gets the OE4T version with HOST1X_SYNCPT_GPU support and the GA10B ERRATA_SYNCPT_INVALID_ID_0 fix.

Build notes

• Built with Clang (LLVM=1), matching the Talos toolchain
• OE4T linux-nv-oot branch wip-r36.5-take-2 adds kernel 6.18 compatibility fixes (__assign_str, f_ref, __alloc_pages_bulk 5-arg)
• Module signing uses the kernel-build stage's auto-generated certs/signing_key.pem
• CONFIG_GCC_PLUGIN_LATENT_ENTROPY stripped from auto.conf / autoconf.h before OOT builds (Clang compat)
• CI-validated with ARM64 build (~90 min cold / ~20 min warm kernel cache), all 6 .ko files confirmed present

Result

Full CUDA inference in Kubernetes pods on Talos Linux without privileged: true, using CDI device injection. Verified ~60 tok/s on qwen2.5:0.5b (Ollama) on Jetson Orin NX 16GB.

References

• Continues: (WIP)feat: add l4t package #1166 (WIP l4t pkg — established that modules need to be a pkg first)
• Discussion: feat: add support for Jetson Orin Nano and AGX Orin SBCs sbc-jetson#23 (frezbo/smira confirmed shim belongs in pkgs)
• Extension PR (follow-up, once this merges): (WIP)feat: add extension for jetson orin SBC extensions#624

[frezbo]

@claude review

[Copilot]

Pull request overview

Adds a new Talos kernel package (nvidia-tegra-nvgpu) to build and ship the OE4T-patched NVIDIA Tegra (Jetson Orin / GA10B) GPU driver stack, including a nvhost-ctrl shim module needed for JetPack 6 CUDA runtime behavior.

Changes:

• Introduces the nvidia-tegra-nvgpu package build/install pipeline for multiple OE4T OOT modules + nvgpu.ko, including patching steps for kernel 6.18 compatibility.
• Adds an in-tree nvhost_ctrl_shim.c module implementing /dev/nvhost-ctrl and bridging NVHOST ioctls to host1x syncpoints/fences.
• Adds OE4T source pins (commit + hashes) to Pkgfile to fetch/build the required upstream driver sources reproducibly.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
nvidia-tegra-nvgpu/pkg.yaml	New package recipe: fetch OE4T sources, apply fixups/patches, build modules with clang wrapper, install + sign modules, add modprobe softdeps.
nvidia-tegra-nvgpu/nvhost_ctrl_shim.c	New kernel module implementing /dev/nvhost-ctrl and required ioctls for CUDA runtime syncpoint waits and fence interop.
nvidia-tegra-nvgpu/scripts/fixup.sh	Build-time patch script to adjust OE4T OOT module Makefiles and force kernel-compat conftest paths for 6.18.
nvidia-tegra-nvgpu/scripts/patch_nvhost_host1x.py	Build-time source patch to add a retry loop in nvhost_host1x.c for GA10B syncpt allocation failures.
Pkgfile	Adds pinned OE4T source revisions and checksums used by the new package.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Adds a new Talos kernel module package providing the OE4T-patched NVIDIA Tegra GA10B GPU driver stack (Jetson Orin NX / Tegra234), including a /dev/nvhost-ctrl shim required by the JetPack 6 CUDA runtime.

Changes:

• Introduces a new nvidia-tegra-nvgpu package that builds/installs OE4T host1x, host1x-fence, nvmap, mc-utils, devfreq governor, nvgpu, plus an in-package nvhost-ctrl ioctl shim module.
• Adds build-time fixup/patch scripts to force kernel-6.18 conftest paths and apply GA10B syncpoint allocation workarounds.
• Adds OE4T source pin variables (commit + checksums) to Pkgfile.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
nvidia-tegra-nvgpu/pkg.yaml	New package build/install/test pipeline for OE4T modules + nvhost-ctrl shim; applies multiple source patches.
nvidia-tegra-nvgpu/nvhost_ctrl_shim.c	New kernel module implementing /dev/nvhost-ctrl NVHOST ioctls backed by host1x syncpoints/dma_fence.
nvidia-tegra-nvgpu/scripts/fixup.sh	Build-time Makefile/source patching for OE4T OOT modules to compile against kernel 6.18 with Clang.
nvidia-tegra-nvgpu/scripts/patch_nvhost_host1x.py	Build-time Python patcher adding a retry loop to nvhost_host1x.c syncpt allocation.
Pkgfile	Adds pinned OE4T commits and checksums for linux-nvgpu, linux-nv-oot, and linux-hwpm.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[schwankner]

Finally managed to post a question to the NVIDIA Developer Forum to get clarity on the /dev/nvhost-ctrl situation: /dev/nvhost-ctrl missing with JetPack 6 Out-of-Tree modules on mainline kernel

[frezbo]

Finally managed to post a question to the NVIDIA Developer Forum to get clarity on the /dev/nvhost-ctrl situation: /dev/nvhost-ctrl missing with JetPack 6 Out-of-Tree modules on mainline kernel

They seemed to ask more questions to try with stock one

[github-actions]

This PR is stale because it has been open 45 days with no activity.

[mmalyska]

Hi @schwankner Any updates on this?

[schwankner]

Hi @mmalyska, @frezbo. Sorry for the long delay, I've been very busy privately and hadn't been able to get back to testing until now. It's tested now.

Short version of what the actual problem was: the blocker was never /dev/nvhost-ctrl. The stock mainline tegra-drm.ko is built against the upstream host1x ABI and refuses to load on top of the OE4T-patched host1x.ko (unresolved host1x_job_alloc, err -22), so /dev/dri/renderD128 never appeared and cuInit returned 801. NVIDIA confirmed in the forum thread that stock JetPack 6 also has no generic /dev/nvhost-ctrl (only nvhost-ctrl-gpu).

The fix is to build tegra-drm, host1x-nvhost and nvhwpm from the same OE4T tree as host1x so the ABI matches. With the DRM render node present, CUDA works without the shim at all.

I've verified this on a fresh Jetson Orin NX 16 GB (Talos v1.13.0, kernel 6.18.24), confirmed real GPU usage (models fully in VRAM, library=CUDA), ~64 tok/s (qwen2.5:0.5b) / ~16 tok/s (qwen3:4b), on par with native JetPack 6.2.

Given this, I will update this PR accordingly, dropping nvhost_ctrl_shim and switching to the OE4T DRM stack, since that's the approach that actually works. I'll refresh the branch accordingly. Reference implementation: https://github.com/schwankner/talos-jetson-orin

[schwankner]

Pushed the update. The shim is gone and the package now builds the OE4T DRM stack instead:

• host1x-nvhost, nvhwpm (OE4T linux-hwpm) and tegra-drm are built from the same OE4T tree as host1x, so the ABI matches
• tegra-drm is installed at kernel/drivers/gpu/drm/tegra/ so depmod/modprobe resolve this module rather than the in-tree one, which is what caused the host1x_job_alloc version mismatch
• tegra-drm is built headless (CONFIG_DRM_TEGRA_HAVE_DISPLAY=n); only the render node is needed
• load order is now host1x -> host1x-fence -> host1x-nvhost -> tegra-drm -> nvmap -> nvgpu
• nvhost_ctrl_shim.c removed (-510 lines)

Two build fixes came out of validating this, both pre-existing:

1. scripts/fixup.sh and scripts/clang-oot were committed without the executable bit, so /pkg/scripts/fixup.sh aborted with exit code 126. Mode is set now and the script is invoked through bash so it no longer depends on the mode.
2. CONFIG_GCC_PLUGIN_LATENT_ENTROPY is now stripped before building. It declares a global via a GCC plugin that Clang does not run, which breaks linux/random.h for every OOT module. The sed is a no-op when the option is not set.

Since a fork cannot use the pkgs runner group, I validated the build separately on an arm64 runner: all modules compile (host1x, host1x-fence, host1x-nvhost, nvhwpm, tegra-drm, nvmap, mc-utils, nvgpu), no shim is produced, and both host1x and tegra-drm land at their shadow paths. Runtime verification was done on a Jetson Orin NX 16 GB as described above.

Happy to split the two build fixes into a separate commit or rebase if you prefer, and to add other Orin variants to the description.

[schwankner]

Rebased onto current main to resolve a merge conflict (siderolabs/pkgs main had moved 122 commits ahead, including another PR adding gds-nvidia-fs vars at the same insertion point in Pkgfile as this one's oe4t_* vars; both blocks are independent and now sit side by side).

Re-validated against the latest main (kernel 6.18.45): all modules build (host1x, host1x-fence, host1x-nvhost, nvhwpm, tegra-drm, nvmap, mc-utils, nvgpu), no shim, both host1x and tegra-drm at their shadow paths. No changes were needed in nvidia-tegra-nvgpu/ itself, only the Pkgfile conflict.

[frezbo]

maybe env vars can be used to override?

[frezbo]

I meant CFLAGS etc

[frezbo]

is there anyting pending that can be simplified in here, this is still huge, maybe a patch file is easier to maintain

[schwankner]

Thanks for the review, and sorry for the slow turnaround. Both suggestions were right, and
digging into them turned up the actual reason the recipe had grown so large. I pushed the
result: pkg.yaml is down from 547 to 137 lines, scripts/fixup.sh and scripts/clang-oot
are gone.

The CC wrapper is dead code. I instrumented it to log every flag it touched. Across more
than 1500 compiler invocations it did exactly two things: it dropped
-fpatchable-function-entry=4,2, which Clang supports and ftrace needs, and it rewrote
-Wimplicit-fallthrough=3 to -Wimplicit-fallthrough, which comes from nvgpu's own
cc-option probe that already drops the flag by itself under Clang. No GCC-only flags reach
the compiler any more now that the pkgs kernel is Clang-built, so the wrapper was at best
useless. The four -Wno-* flags it appended turned out to be unnecessary as well once the right code
paths are taken; the only global flags left are the include paths, passed as KCFLAGS in the
package env:, which is what you suggested.

The macro rewrites had a root cause. scripts/conftest/Makefile uses CC ?= cc, and cc
in the build image is GCC. Every probe was compiled with GCC against the headers of a
Clang-built kernel, so the probes failed wholesale and every NV_* macro stayed undefined.
The sources then took the wrong branch, for example host1x cdma.c calling iommu_map()
with five arguments. That is what the roughly 30 LINUX_VERSION_CODE rewrites were papering
over. Running conftest with CC=clang makes all probes behave, and the rewrites are gone.
The conftest Makefile also hardcodes -Werror in NV_CONFTEST_CFLAGS, so probes that merely
warn under Clang fail as well; I drop it on the working copy so a probe only fails on a real
compile error. The recipe now asserts that one macro every 6.3+ kernel must have is defined,
so a broken conftest fails the build instead of silently selecting old code paths.

What is left. Three patch files under nvidia-tegra-nvgpu/patches/, grouped per source tree:
the headless tegra-drm build (the kernel config has DRM_FBDEV_EMULATION=y, which the module
inherits through autoconf.h), the GA10B syncpoint retry and id-0 workaround, and a netlist
flexible-array change that silences a UBSAN false positive. Three further fixes I carried
(class_create() owner argument, the devnode signature, get_user_pages() without vmas)
turned out to be guarded by NV_* macros in the OE4T sources already, so they went away with
the working conftest. -Werror in the OE4T module Makefiles is filtered with
ccflags-remove-y=-Werror on the make command line instead of editing nine Makefiles, only the
make variables that differ from the OE4T defaults are passed, and one shell function covers all
nine module directories in both the build and install steps.

I validated this the same way as before, plus on hardware this time. The build links all ten
modules with no errors, and on a Jetson Orin NX the extension gives /dev/dri/renderD128,
library=CUDA compute=8.7 in Ollama, and 66.0 tok/s on qwen2.5:0.5b against 65 to 66 tok/s
for the previous recipe, so the simplification is behaviour-neutral.

Two open points from my side: the GA10B firmware still has to come from somewhere, either
linux-firmware in pkgs or bundled in the extension, and there is no Jetson GPU extension in
siderolabs/extensions yet. Happy to follow whichever route you prefer.

[dsseng]

Two open points from my side: the GA10B firmware still has to come from somewhere, either
linux-firmware in pkgs or bundled in the extension, and there is no Jetson GPU extension in
siderolabs/extensions yet. Happy to follow whichever route you prefer.

linux-firmware in pkgs contains all the firmwares officially shipped by linux-firmware. If the needed blobs are available - please create the extension consisting of this driver and the necessary firmware files copied from there. If not, please find another clearly licensed and reliable source, and use that for the extension

[schwankner]

Thanks, that settles the direction. I checked both routes:

linux-firmware does not carry the GA10B blobs. The upstream tree has nvidia/gk20a, gm20b, gp10b, gv100, the tegra* bootloader/XUSB files and the desktop GSP firmware (ga100, ga102, ...), but no nvidia/ga10b. NVIDIA has never submitted the Orin GPU firmware there, so the pkgs linux-firmware package cannot supply it.

The clearly licensed source is NVIDIA's own Jetson Linux package. The 17 files nvgpu loads on Orin ship in nvidia-l4t-firmware from the official Jetson apt repository (repo.download.nvidia.com/jetson/t234, currently 36.5.2-20260716114719), under lib/firmware/nvidia/ga10b/. Its copyright file places them under the NVIDIA Driver License Agreement, and section 1.1(d) of that agreement explicitly permits distribution "for use with operating system kernels distributed under the terms of an OSI-approved open source license", provided the binaries stay unmodified and the agreement is shipped alongside. That is the same license text siderolabs/extensions already carries as NVIDIA-driver-LICENSE for the nonfree kmod-nvidia and Kata GPU extensions, so the packaging would follow the existing pattern rather than introduce a new one. OE4T's meta-tegra takes the firmware from the same deb for its Yocto BSP.

Proposal. I would add a nonfree extension to siderolabs/extensions next to nvidia-gpu/nonfree/kmod-nvidia, depending on the nvidia-tegra-nvgpu-pkg image from this PR and fetching the pinned nvidia-l4t-firmware deb (URL plus sha256 in vars) to copy exactly the ga10b directory into /usr/lib/firmware/ga10b, together with the license file and the modprobe.d softdep ordering. Firmware and modules come from the same JetPack release, so they stay in lockstep. Once this PR is merged I will open that extension PR; if you would rather see the extension first, I can open it against my fork's pkgs image so both can be reviewed together.