CDI behavior with NVIDIA GPU devices in Kubernetes

[shilf1]

Hello, I'm currently testing the NVIDIA CDI feature.

I've enabled CDI on a host with an NVIDIA GPU, and set runtimeClassName: nvidia-cdi in my Kubernetes pod spec.
Here's what I observed:

apiVersion: v1
kind: Pod
metadata:
  name: cdi-gpu-test
spec:
  nodeName: h100
  runtimeClassName: nvidia-cdi <- I added this
  containers:
    - name: cuda-container
      image: ubuntu22.04
      command: ["sh", "-c", "sleep 3600"]
      resources:
        limits:
          nvidia.com/gpu: 1
        requests:
          nvidia.com/gpu: 1
  restartPolicy: Never

# kubectl exec -it cdi-gpu-test -- bash

# after creating pod, checkint /dev/
root@cdi-gpu-test:/dev# ls -al /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Jul 22 02:38 /dev/nvidia-modeset
crw-rw-rw- 1 root root 498,   0 Jul 22 02:38 /dev/nvidia-uvm
crw-rw-rw- 1 root root 498,   1 Jul 22 02:38 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   6 Jul 22 02:38 /dev/nvidia6
crw-rw-rw- 1 root root 195, 255 Jul 22 02:38 /dev/nvidiactl


# try nvidia-smi command

root@cdi-gpu-test:/dev# nvidia-smi
Tue Jul 22 02:39:38 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08              Driver Version: 575.57.08      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:D2:00.0 Off |                    0 |
| N/A   27C    P0             69W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+


# after nvidia-smi, I can see all devices

root@cdi-gpu-test:/dev# ls -al /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Jul 22 02:38 /dev/nvidia-modeset
crw-rw-rw- 1 root root 498,   0 Jul 22 02:38 /dev/nvidia-uvm
crw-rw-rw- 1 root root 498,   1 Jul 22 02:38 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Jul 22 02:39 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Jul 22 02:39 /dev/nvidia1
crw-rw-rw- 1 root root 195,   2 Jul 22 02:39 /dev/nvidia2
crw-rw-rw- 1 root root 195,   3 Jul 22 02:39 /dev/nvidia3
crw-rw-rw- 1 root root 195,   4 Jul 22 02:39 /dev/nvidia4
crw-rw-rw- 1 root root 195,   5 Jul 22 02:39 /dev/nvidia5
crw-rw-rw- 1 root root 195,   6 Jul 22 02:38 /dev/nvidia6
crw-rw-rw- 1 root root 195,   7 Jul 22 02:39 /dev/nvidia7
crw-rw-rw- 1 root root 195, 255 Jul 22 02:38 /dev/nvidiactl

When the pod starts, checking /dev/ inside the pod shows only one device: nvidia6.
However, after running nvidia-smi inside the pod, all 8 NVIDIA device nodes (e.g., nvidia0 through nvidia7) appear under /dev/.
This leads me to two questions regarding CDI behavior:

1. Is it the intended behavior of CDI that only one device node (e.g., nvidia6) appears at first?
   Personally, I expected CDI (being a device virtualization/mapping layer) to expose only one device (e.g., nvidia0) and internally manage the mapping.
2. Why do all device nodes appear under /dev/ after running nvidia-smi?
   It seems like nvidia-smi somehow triggers the creation of all device nodes.
   This makes me wonder: do I need to run nvidia-smi before my actual container command to ensure all necessary devices are available? Is there a better way to ensure this without manually invoking nvidia-smi?

I'd appreciate your comments.

[elezar]

The issue is the following: By default running nvidia-smi or (depending on the driver version) invoking certain other libnvidia-ml.so the ModifyDeviceFiles parameter from /proc/driver/nvidia/params is read to determine whether the device nodes for the devices in the system should be created. In the "legacy" mode, the /proc/driver/nvidia/params file is masked in the container and modified so that it contains ModifyDeviceFiles: 0 to disable device node modification. The equivalent fucntionality has only just been added to CDI mode in #927 and will be included in the v1.18.0 release.

Note that in most cases the presence of these device nodes is purely cosmetic. The cgroups in the container should be restrictive enough to prevent a user from actually accessing these device nodes. This is why the nvidia-smi output shows only a single device even though multiple device nodes are present.

[ZeroExistence]

Hello @elezar , good day. I just want to check related to this issue, is it safe to upgrade the container-toolkit ahead of other components with gpu-operator, or is it better to wait for the next release of gpu-operator? I am currently working with gpu-operator v25.3.1.

And is it safe to switch the container-toolkit image from Ubuntu20.04 based container-toolkit into the Golang distroless release nvcr.io/nvidia/k8s/container-toolkit:v1.18.0-rc.1?

And last thing, is this fixed behavior with generated devices maps in container level is all fixed within #927 ? I assume I only need to upgrade the container-toolkit without touching the device-plugin or other component.

Thank you very much!

[elezar]

@ZeroExistence we do attempt to keep the API between the GPU Operator and the NVIDIA Container Toolkit container stable across releases, so upgrading only the NVIDIA Container Toolkit e.g to nvcr.io/nvidia/k8s/container-toolkit:v1.18.0-rc.2 should work as expected. Especially for such a recent GPU Operator version. (If you have any problems with this please reach out).

And last thing, is this fixed behavior with generated devices maps in container level is all fixed within #927 ? I assume I only need to upgrade the container-toolkit without touching the device-plugin or other component.

Yes, the changes from #947 will disable the creation of non-injected device nodes in a container when the runtime is running in CDI mode. However, since the device plugin is generating the CDI specifications in your case, we would also need to provide an updated k8s-device-plugin image that includes these changes. I will work on preparing a release candidate for the device plugin that contains these changes.

(I have created NVIDIA/k8s-device-plugin#1339 to update this).

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[elezar]

@ZeroExistence this should be addressed in the latest releases of the NVIDIA Container Toolkit and Device Plugin (v1.18.0 and v0.18.0, respectively). Please reopen or create a new issue if you're still having problems.