-
Notifications
You must be signed in to change notification settings - Fork 206
feat: add nvidia-tegra-nvgpu package for Jetson Orin (GA10B) #1518
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
schwankner
wants to merge
23
commits into
siderolabs:main
Choose a base branch
from
schwankner:feat/nvidia-tegra-jetson-orin-main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
ab7c963
feat: add nvidia-tegra-nvgpu package for Jetson Orin NX (GA10B)
schwankner 8805349
refactor: move OOT Makefile patching to scripts/fixup.sh
schwankner e60c74f
refactor: extract nvhost_host1x patch to script, fail on governor bui…
schwankner 6da63a1
chore: add renovate tags for OE4T kernel module dependencies
schwankner 0ca0ba0
refactor: use modules_install with INSTALL_MOD_STRIP=1 CONFIG_MODULE_…
schwankner fd4bb90
fix: release device ref, guard devnode mode, drop duplicate clang-oot
schwankner 335ee46
fix: add post-check to awk nvhost_host1x syncpt id=0 patch
schwankner 9e04f9c
chore: remove duplicate latent_entropy sed from nvgpu build block
schwankner 6ea1e15
chore: remove unused CONFTEST_OUT variable from fixup.sh
schwankner 311f187
fix: add -pkg suffix to name, register in .kres.yaml targets
schwankner 08a6cfd
fix: copy modules.order and modules.builtin into rootfs
schwankner 3f18915
chore: switch oe4t_nv_oot to patches-r36.5 branch
schwankner 6368e07
chore: extract clang-oot wrapper to scripts/clang-oot
schwankner 3624f1f
chore: remove modprobe.d softdep and debug find from install step
schwankner 76d0086
Merge branch 'main' into feat/nvidia-tegra-jetson-orin-main
schwankner 559098f
refactor: replace patch scripts with standard .patch files
schwankner 2c4167e
fix: correct oe4t_nv_oot_commit to patches-r36.5 HEAD (ea32e7f97dd0)
schwankner 754aee5
refactor: remove dead code from fixup.sh (kref_init + GCC_PLUGIN_LATE…
schwankner eb7b15e
feat(nvidia-tegra-nvgpu): replace #if 1 guards with LINUX_VERSION_CODE
schwankner aad6adb
feat(nvidia-tegra-nvgpu): replace nvhost-ctrl-shim with OE4T DRM stack
schwankner 7c17807
fix(nvidia-tegra-nvgpu): make build scripts executable, drop GCC entr…
schwankner 81f8979
Merge branch 'main' into feat/nvidia-tegra-jetson-orin-main
schwankner 6bcd94e
feat: reduce the nvidia-tegra-nvgpu build recipe to the minimum
schwankner File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
77 changes: 77 additions & 0 deletions
77
nvidia-tegra-nvgpu/patches/nvgpu/0001-nvhost-syncpt-retry-and-skip-id0.patch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| nvgpu: make host1x syncpoint allocation robust on GA10B | ||
|
|
||
| Two fixes for CUDA error 999 on the first cudaStreamSynchronize(): | ||
| 1. Retry host1x_syncpt_alloc() for up to 5 ms; it returns NULL during | ||
| the short async GR init window after the first kernel submit. | ||
| 2. Never hand out syncpt id 0. Without a "gpu" syncpt pool in the DT, | ||
| host1x allocates id 0 first, which GA10B rejects | ||
| (NVGPU_ERRATA_SYNCPT_INVALID_ID_0). | ||
|
|
||
| diff --git a/drivers/gpu/nvgpu/os/linux/nvhost_host1x.c b/drivers/gpu/nvgpu/os/linux/nvhost_host1x.c | ||
| index 945fff7..bf88a52 100644 | ||
| --- a/drivers/gpu/nvgpu/os/linux/nvhost_host1x.c | ||
| +++ b/drivers/gpu/nvgpu/os/linux/nvhost_host1x.c | ||
| @@ -1,3 +1,4 @@ | ||
| +#include <linux/delay.h> | ||
| /* | ||
| * Copyright (c) 2020-2022, NVIDIA CORPORATION. All rights reserved. | ||
| * | ||
| @@ -237,17 +238,50 @@ void nvgpu_nvhost_syncpt_put_ref_ext(struct nvgpu_nvhost_dev *nvhost_dev, | ||
| u32 nvgpu_nvhost_get_syncpt_client_managed(struct nvgpu_nvhost_dev *nvhost_dev, | ||
| const char *syncpt_name) | ||
| { | ||
| - struct host1x_syncpt *sp; | ||
| + struct host1x_syncpt *sp = NULL; | ||
| struct host1x *host1x; | ||
| + int retry; | ||
|
|
||
| - host1x = platform_get_drvdata(nvhost_dev->host1x_pdev); | ||
| - if (!host1x) | ||
| - return 0; | ||
| - | ||
| - sp = host1x_syncpt_alloc(host1x, HOST1X_SYNCPT_CLIENT_MANAGED | HOST1X_SYNCPT_GPU, | ||
| - syncpt_name); | ||
| - if (!sp) | ||
| + /* | ||
| + * During the ~1-2 ms async GR init window after the first kernel submit, | ||
| + * host1x_syncpt_alloc() can return NULL. Returning 0 here makes channel | ||
| + * sync creation fail (GA10B rejects syncpt id 0) and CUDA reports error 999 | ||
| + * on the first cudaStreamSynchronize(). Retry briefly to cover the window. | ||
| + */ | ||
| + for (retry = 0; retry < 5; retry++) { | ||
| + host1x = platform_get_drvdata(nvhost_dev->host1x_pdev); | ||
| + if (!host1x) { | ||
| + pr_warn_ratelimited("nvgpu: host1x not ready, syncpt retry %d/5\n", retry + 1); | ||
| + msleep(1); | ||
| + continue; | ||
| + } | ||
| + sp = host1x_syncpt_alloc(host1x, | ||
| + HOST1X_SYNCPT_CLIENT_MANAGED | HOST1X_SYNCPT_GPU, | ||
| + syncpt_name); | ||
| + if (sp) | ||
| + break; | ||
| + pr_warn_ratelimited("nvgpu: syncpt_alloc NULL, retry %d/5\n", retry + 1); | ||
| + msleep(1); | ||
| + } | ||
| + if (!sp) { | ||
| + pr_err_ratelimited("nvgpu: get_syncpt_client_managed: failed after retries\n"); | ||
| return 0; | ||
| + } | ||
| + /* | ||
| + * GA10B (NVGPU_ERRATA_SYNCPT_INVALID_ID_0) rejects syncpt id 0. Without a | ||
| + * "gpu" syncpt pool in the DT, host1x hands out id 0 first. Hold it and | ||
| + * allocate again so the GPU gets id >= 1; the Tegra234 syncpt shim covers | ||
| + * all 1024 ids, so any non-zero id is GPU-signable. | ||
| + */ | ||
| + if (host1x_syncpt_id(sp) == 0U) { | ||
| + struct host1x_syncpt *sp_skip = sp; | ||
| + sp = host1x_syncpt_alloc(host1x, | ||
| + HOST1X_SYNCPT_CLIENT_MANAGED | HOST1X_SYNCPT_GPU, | ||
| + syncpt_name); | ||
| + host1x_syncpt_put(sp_skip); | ||
| + if (!sp) | ||
| + return 0; | ||
| + } | ||
|
|
||
| return host1x_syncpt_id(sp); | ||
| } |
19 changes: 19 additions & 0 deletions
19
nvidia-tegra-nvgpu/patches/nvgpu/0002-netlist-flexible-array.patch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| nvgpu: netlist regions as flexible array member | ||
|
|
||
| regions[1] is indexed by the region count from the firmware header, so | ||
| UBSAN flags every access with index >= 1. A flexible array member has the | ||
| same layout and silences the false positive. | ||
|
|
||
| diff --git a/drivers/gpu/nvgpu/common/netlist/netlist_priv.h b/drivers/gpu/nvgpu/common/netlist/netlist_priv.h | ||
| index 0ad50e2..b224ed9 100644 | ||
| --- a/drivers/gpu/nvgpu/common/netlist/netlist_priv.h | ||
| +++ b/drivers/gpu/nvgpu/common/netlist/netlist_priv.h | ||
| @@ -113,7 +113,7 @@ struct netlist_image_header { | ||
|
|
||
| struct netlist_image { | ||
| struct netlist_image_header header; | ||
| - struct netlist_region regions[1]; | ||
| + struct netlist_region regions[]; | ||
| }; | ||
|
|
||
| struct netlist_gr_ucode { |
29 changes: 29 additions & 0 deletions
29
nvidia-tegra-nvgpu/patches/nvidia-oot/0001-tegra-drm-headless-no-fbdev.patch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| tegra-drm: build headless, without fbdev emulation | ||
|
|
||
| The kernel config has CONFIG_DRM_FBDEV_EMULATION=y, which the module | ||
| inherits through autoconf.h and cannot undefine via ccflags. fbdev.c | ||
| relies on drm_fb_helper internals that changed heavily between 6.0 and | ||
| 6.18 and is not needed for a render node, so always take the stub path. | ||
|
|
||
| diff --git a/drivers/gpu/drm/tegra/drm.h b/drivers/gpu/drm/tegra/drm.h | ||
| index 147ce75..1224e96 100644 | ||
| --- a/drivers/gpu/drm/tegra/drm.h | ||
| +++ b/drivers/gpu/drm/tegra/drm.h | ||
| @@ -228,7 +228,7 @@ struct drm_framebuffer *tegra_fb_create(struct drm_device *drm, | ||
| const struct drm_mode_fb_cmd2 *cmd); | ||
|
|
||
| #if defined(NV_DRM_DRIVER_HAS_FBDEV_PROBE) /* Linux v6.13 */ | ||
| -#ifdef CONFIG_DRM_FBDEV_EMULATION | ||
| +#if 0 /* fbdev not needed for headless CUDA */ | ||
| int tegra_fbdev_driver_fbdev_probe(struct drm_fb_helper *helper, | ||
| struct drm_fb_helper_surface_size *sizes); | ||
| #define TEGRA_FBDEV_DRIVER_OPS \ | ||
| @@ -238,7 +238,7 @@ int tegra_fbdev_driver_fbdev_probe(struct drm_fb_helper *helper, | ||
| .fbdev_probe = NULL | ||
| #endif | ||
| #else | ||
| -#ifdef CONFIG_DRM_FBDEV_EMULATION | ||
| +#if 0 /* fbdev not needed for headless CUDA */ | ||
| void tegra_fbdev_setup(struct drm_device *drm); | ||
| #else | ||
| static inline void tegra_fbdev_setup(struct drm_device *drm) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| name: nvidia-tegra-nvgpu-pkg | ||
| variant: scratch | ||
| shell: /bin/bash | ||
| dependencies: | ||
| - stage: base | ||
| - stage: kernel-build | ||
| - image: "{{ .LLVM_IMAGE }}:{{ .TOOLS_REV }}" | ||
| steps: | ||
| - sources: | ||
| # OE4T patched nvgpu - supports kernel 6.x (fixes platform_driver.remove, hrtimer, struct fd) | ||
| - url: https://github.com/OE4T/linux-nvgpu/archive/{{ .oe4t_nvgpu_commit }}.tar.gz | ||
| destination: nvgpu.tar.gz | ||
| sha256: "{{ .oe4t_nvgpu_sha256 }}" | ||
| sha512: "{{ .oe4t_nvgpu_sha512 }}" | ||
| # OE4T patched nvidia-oot - patches-r36.5 branch (kernel 6.18 compat: __assign_str, | ||
| # f_count->f_ref, __alloc_pages_bulk 5-arg, and all earlier 6.x fixes) | ||
| - url: https://github.com/OE4T/linux-nv-oot/archive/{{ .oe4t_nv_oot_commit }}.tar.gz | ||
| destination: nvidia-oot.tar.gz | ||
| sha256: "{{ .oe4t_nv_oot_sha256 }}" | ||
| sha512: "{{ .oe4t_nv_oot_sha512 }}" | ||
| # OE4T patched hwpm - supports kernel 6.x (fixes platform_driver.remove, MODULE_IMPORT_NS) | ||
| - url: https://github.com/OE4T/linux-hwpm/archive/{{ .oe4t_hwpm_commit }}.tar.gz | ||
| destination: hwpm.tar.gz | ||
| sha256: "{{ .oe4t_hwpm_sha256 }}" | ||
| sha512: "{{ .oe4t_hwpm_sha512 }}" | ||
| env: | ||
| ARCH: arm64 | ||
| LLVM: "1" | ||
| LLVM_IAS: "1" | ||
| # conftest compiles its probes with $CC (Makefile default: cc = GCC). Probing | ||
| # Clang-built kernel headers with GCC fails wholesale, which leaves every NV_* | ||
| # macro undefined and selects the wrong code paths. The kernel build itself | ||
| # ignores CC because LLVM=1 selects clang. | ||
| CC: clang | ||
| # The OE4T top-level Makefiles add these include paths for every module; the | ||
| # module directories are built individually here, so pass them globally. | ||
| KCFLAGS: >- | ||
| -I/oot-src/out/nvidia-conftest | ||
| -I/oot-src/nvidia-oot/include | ||
| -I/oot-src/nvidia-oot/drivers/gpu/host1x/include | ||
| -I/oot-src/nvidia-oot/drivers/video/tegra/nvmap/include | ||
| # Common make arguments for every out-of-tree module directory. -Werror is | ||
| # dropped from the OE4T Makefiles because Clang warns where GCC does not. | ||
| OOT_MAKE: >- | ||
| CONFIG_TEGRA_OOT_MODULE=m ccflags-remove-y=-Werror LLVM=1 | ||
| srctree.nvidia-oot=/oot-src/nvidia-oot srctree.nvidia=/oot-src/nvidia-oot | ||
| srctree.hwpm=/oot-src/hwpm srctree.nvconftest=/oot-src/out/nvidia-conftest | ||
| KBUILD_EXTRA_SYMBOLS=/oot-src/out/Module.symvers.nvidia | ||
| prepare: | ||
| - | | ||
| for t in nvgpu nvidia-oot hwpm; do | ||
| mkdir -p /oot-src/$t | ||
| tar xzf $t.tar.gz -C /oot-src/$t --strip-components=1 | ||
| done | ||
| # patches/<tree>/*.patch applies to /oot-src/<tree> | ||
| for p in /pkg/patches/*/*.patch; do t=${p%/*}; patch -p1 -d /oot-src/${t##*/} < $p; done | ||
| build: | ||
| - | | ||
| # conftest probes the kernel API and writes conftest/*.h, which selects the | ||
| # code paths in every module (see CC above). Its Makefile hardcodes -Werror, | ||
| # which turns Clang-only warnings into failed probes: 26 of 129 macros go | ||
| # missing (measured), e.g. NV___ASSIGN_STR_HAS_NO_SRC_ARG, and one inverted | ||
| # "function present" probe reports the wrong answer. | ||
| CONFTEST=/oot-src/out/nvidia-conftest/nvidia | ||
| mkdir -p $CONFTEST | ||
| cp -a /oot-src/nvidia-oot/scripts/conftest/. $CONFTEST/ | ||
| sed -i 's/ -Werror / /' $CONFTEST/Makefile | ||
| make -j$(nproc) -f $CONFTEST/Makefile src=$CONFTEST obj=$CONFTEST \ | ||
| NV_KERNEL_SOURCES=/src NV_KERNEL_OUTPUT=/src | ||
| # a probe every 6.3+ kernel must pass: fails the build if conftest is broken | ||
| grep -rq '#define NV_IOMMU_MAP_HAS_GFP_ARG' $CONFTEST/conftest/ | ||
| - | | ||
| OOT=/oot-src/nvidia-oot | ||
| : > /oot-src/out/Module.symvers.nvidia | ||
|
|
||
| # Build one module directory against the kernel and add its exported symbols | ||
| # for the directories that follow, so the order below matters. | ||
| oot() { | ||
| make -j$(nproc) -C /src M=$1 "${@:2}" $OOT_MAKE modules | ||
| cat $1/Module.symvers >> /oot-src/out/Module.symvers.nvidia | ||
| } | ||
|
|
||
| oot $OOT/drivers/gpu/host1x | ||
| oot $OOT/drivers/platform/tegra/mc-utils | ||
| oot $OOT/drivers/gpu/host1x-fence | ||
| oot $OOT/drivers/gpu/host1x-nvhost | ||
| oot /oot-src/hwpm/drivers/tegra/hwpm | ||
| # headless: no fbdev (the kernel config has it enabled, so it must be forced off) | ||
| oot $OOT/drivers/gpu/drm/tegra CONFIG_DRM_FBDEV_EMULATION=n | ||
| # the OE4T defaults for 5.10+ are handle-as-id and NvSciIpc support, neither is wanted here | ||
| oot $OOT/drivers/video/tegra/nvmap NVMAP_CONFIG_HANDLE_AS_ID=n NVMAP_CONFIG_SCIIPC=n | ||
| oot $OOT/drivers/devfreq | ||
| # syncpoints through the OE4T host1x (the kernel has neither GRHOST nor HOST1X_NEXT) | ||
| oot /oot-src/nvgpu/drivers/gpu/nvgpu CONFIG_TEGRA_GK20A_NVHOST=y CONFIG_TEGRA_GK20A_NVHOST_HOST1X=y | ||
| install: | ||
| - | | ||
| OOT=/oot-src/nvidia-oot | ||
| MOD=/rootfs/usr/lib/modules/$(cat /src/include/config/kernel.release) | ||
| mkdir -p $MOD | ||
| cp /src/modules.order /src/modules.builtin /src/modules.builtin.modinfo $MOD/ | ||
|
|
||
| # Modules are stripped after signing, see | ||
| # https://www.kernel.org/doc/html/v4.15/admin-guide/module-signing.html#signed-modules-and-stripping | ||
| oot_install() { | ||
| make -j$(nproc) -C /src M=$1 "${@:3}" $OOT_MAKE \ | ||
| INSTALL_MOD_PATH=/rootfs/usr INSTALL_MOD_DIR=$2 \ | ||
| INSTALL_MOD_STRIP=1 CONFIG_MODULE_SIG_ALL=y modules_install | ||
| } | ||
|
|
||
| # host1x and tegra-drm are installed at the in-tree paths they shadow. depmod | ||
| # resolves kernel/drivers/gpu/drm/tegra/tegra-drm.ko against | ||
| # kernel/drivers/gpu/host1x/host1x.ko, so installing only under extra/ leaves | ||
| # modprobe pointing at the vanilla tegra-drm.ko, which then fails with | ||
| # "tegra_drm: disagrees about version of symbol host1x_job_alloc". | ||
| oot_install $OOT/drivers/gpu/host1x kernel/drivers/gpu/host1x | ||
| oot_install $OOT/drivers/gpu/drm/tegra kernel/drivers/gpu/drm/tegra CONFIG_DRM_FBDEV_EMULATION=n | ||
| oot_install $OOT/drivers/gpu/host1x-fence extra/nvidia-tegra | ||
| oot_install $OOT/drivers/gpu/host1x-nvhost extra/nvidia-tegra | ||
| oot_install /oot-src/hwpm/drivers/tegra/hwpm extra/nvidia-tegra | ||
| oot_install $OOT/drivers/video/tegra/nvmap extra/nvidia-tegra NVMAP_CONFIG_HANDLE_AS_ID=n NVMAP_CONFIG_SCIIPC=n | ||
| oot_install $OOT/drivers/platform/tegra/mc-utils extra/nvidia-tegra | ||
| oot_install $OOT/drivers/devfreq extra/nvidia-tegra | ||
| oot_install /oot-src/nvgpu/drivers/gpu/nvgpu extra/nvidia-tegra \ | ||
| CONFIG_TEGRA_GK20A_NVHOST=y CONFIG_TEGRA_GK20A_NVHOST_HOST1X=y | ||
|
|
||
| # Load order: host1x provides syncpoint alloc, host1x-fence the dma_fence | ||
| # bridge, host1x-nvhost the nvhost device layer. tegra-drm needs all three and | ||
| # registers /dev/dri/renderD128, which libcuda requires to enumerate the iGPU. | ||
| # nvmap provides GPU memory, mc-utils emc_freq_to_bw, nvgpu comes last. | ||
| mkdir -p /rootfs/usr/lib/modprobe.d | ||
| printf 'softdep tegra-drm pre: host1x host1x-fence host1x-nvhost\n' \ | ||
| > /rootfs/usr/lib/modprobe.d/nvidia-tegra.conf | ||
| printf 'softdep nvgpu pre: host1x nvmap host1x-fence host1x-nvhost tegra-drm mc-utils\n' \ | ||
| >> /rootfs/usr/lib/modprobe.d/nvidia-tegra.conf | ||
| finalize: | ||
| - from: /rootfs | ||
| to: / |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where do these revisions come from? They do not seem to be tags in the repos, so are they perhaps referenced by meta-tegra or whatever?
Please let us know how to recognize proper, released versions of these drivers and not WIP snapshots which may be unstable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid concern. The three commits come from:
oe4t_nvgpu_commit(d530a48): OE4Tlinux-nvgpubranchpatches-rel-36— the project's default stable branch for the r36.x seriesoe4t_nv_oot_commit(ccf7646): OE4Tlinux-nv-ootbranchwip-r36.5-take-2— admittedly a WIP branch. A more stable alternative ispatches-r36.5(last updated 2026-03-05 with kernel 6.18 build fixes). I can switch to that.oe4t_hwpm_commit(4d8a699): OE4Tlinux-hwpmdefault branchNone have release tags — that is the current state of OE4T for GA10B on kernel 6.x.