Jetson Devices

[sam-cts]

I am curious that if this nvidia device plugin can be deployed in any jetson devices.
if not will there be any support for Jetson Devices in near future.

[asierzd]

I have the same issue, the k8s-device-plugin does not work for me with containerd, the gpu-operator tool neither, and in the documentation says that there is not support currently for Jetson devices, only for discrete GPUs.

I hope we can get some updates on this issue, thanks!

[mmalyska]

I'm also interested in this. Will there by any comment from NVIDIA staff?

[DeltaRule]

If you want to run Kubernetes on Nvidia-Jetson you can use this tutorial. https://medium.com/@jpraveenkanna/step-by-step-guide-installing-kubernetes-on-a-jetson-nano-67da8fa183f8 I know it skips the nvidia runtime manager. But if you need something that works, this might help you

[NoeSamaille]

+1 the GPU device plugin does not label my Jetson Orin Nano Super Dev Kit 8GB Node :/

[lesomnus]

Im not sure this is what you guys want but I have deployed device plugin and check that nvidia.com/gpu is available for resourcce and GFD is working.

Follow:

At the first tiem I deployed device plugin,
(note that name of my jetson node is ajax)

kubectl descirbe node ajax

In the command above, the nvidia.com/gpu resource was not visible.
First, the NVIDIA Device Plugin daemon was not deployed on the Jetson node.
I believe this was because NFD (Node Feature Discovery) failed to properly detect Jetson features, since the NVIDIA Device Plugin daemonset has a node selector set to feature.node.kubernetes.io/nvidia-gpu.present=true.

I am not sure whether this can be solved via NFD configuration, but I manually added the label, and the NVIDIA Device Plugin daemon was successfully deployed.

$ kubectl label node ajax feature.node.kubernetes.io/nvidia-gpu.present=true

# wait for the nvidia device plugin daemonset to be deployed

$ kubectl describe node ajax | grep nvidia/gpu
> ...
> nvidia.com/gpu: 1
> ...

After that, the GFD (GPU Feature Discovery) daemon also started deploying on the ajax node, but the pod failed.
I expected the NVIDIA driver libraries to be mounted by setting the RuntimeClass to nvidia, but when I launched a pod manually, it looked like the NVIDIA libraries were not mounted.

I resolved the issue by manually mounting the library path and setting the LD_LIBRARY_PATH environment variable instead of using RuntimeClass.

$ export LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/tegra:$LD_LIBRARY_PATH
$ nvidia-smi
# working
$ gpu-feature-discovery
# working

Ideally, I think, gpu-feature-discovery should intelligently discover the correct library path, but for now I solved the problem by mounting the path and setting LD_LIBRARY_PATH in the Pod.

  - target:
      kind: DaemonSet
      name: infra-jetson-nvidia-device-plugin-gpu-feature-discovery
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value:
          name: LD_LIBRARY_PATH
          value: /usr/lib/aarch64-linux-gnu/tegra:/lib/distroless/
      - op: add
        path: /spec/template/spec/volumes/-
        value:
          name: tegra-libs
          hostPath:
            path: /usr/lib/aarch64-linux-gnu/tegra
            type: Directory
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts/-
        value:
          name: tegra-libs
          mountPath: /usr/lib/aarch64-linux-gnu/tegra
          readOnly: true

Note that /lib/distroless/ is a default path configured in the container.

Now I have:

$ kubectl describe node ajax
...
                    nvidia.com/cuda.runtime-version.full=12.6
                    nvidia.com/cuda.runtime-version.major=12
                    nvidia.com/cuda.runtime-version.minor=6
                    nvidia.com/cuda.runtime.major=12
                    nvidia.com/cuda.runtime.minor=6
                    nvidia.com/gfd.timestamp=1770657836
                    nvidia.com/gpu.compute.major=8
                    nvidia.com/gpu.compute.minor=7
                    nvidia.com/gpu.count=1
                    nvidia.com/gpu.family=ampere
                    nvidia.com/gpu.machine=NVIDIA-Jetson-AGX-Orin-Developer-Kit
...
Capacity:
  nvidia.com/gpu:     1
...
Allocatable:
  nvidia.com/gpu:     1
...

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[mmalyska]

If you use talos linux for k8s there is work on extension to enable cuda workloads on Jetson devices.
Feel free to contribute if you want to push it faster :)
siderolabs/extensions#1062

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[github-actions]

This issue was automatically closed due to inactivity.