Multi GPU, NVIDIA_VISIBLE_DEVICES and FFMPEG 7 NVDEC errors

[baudneo]

So, this is a weird and pretty specific problem. I am at a loss at what I can do next because I am unsure if this is an issue with nvidia or ffmpeg (nvdec specifically). Issue has been observed in the frigate tensorrt images and ubuntu cuda images with ffmpeg 7.

This is in relation to ffmpeg crashing (due to not finding a CUDA device) when using multiple nvidia gpus while trying to use any other index than '0'. I have tried to only expose devices using NVIDIA_VISIBLE_DEVICES env var and assigning using index or GPU-UUID. The weird part is that I can load ONNX models onto GPU 1, which kind of leads towards this may be something specific to ffmpeg.

I am opening this issue to seek advice and to see if there are any other users having this issue.

Error from ffmpeg output:

  2025-07-29 16:57:53.773105842  [2025-07-29 16:57:53] ffmpeg.alley.detect            ERROR   : [AVHWDeviceContext @ 0x5dfde7cb58c0] cu->cuDeviceGet(&hwctx->internal->cuda_device, device_idx) failed -> CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
  2025-07-29 16:57:53.773621224  [2025-07-29 16:57:53] ffmpeg.alley.detect            ERROR   : Device creation failed: -542398533.
  2025-07-29 16:57:53.774103329  [2025-07-29 16:57:53] ffmpeg.alley.detect            ERROR   : [vist#0:0/h264 @ 0x5dfde7c75f40] [dec:h264 @ 0x5dfde7d32440] No device available for decoder: device type cuda needed for codec h264.
  2025-07-29 16:57:53.774607022  [2025-07-29 16:57:53] ffmpeg.alley.detect            ERROR   : [vist#0:0/h264 @ 0x5dfde7c75f40] [dec:h264 @ 0x5dfde7d32440] Hardware device setup failed for decoder: Generic error in an external library
  2025-07-29 16:57:53.775085378  [2025-07-29 16:57:53] ffmpeg.alley.detect            ERROR   : [vost#0:0/rawvideo @ 0x5dfde7c80000] Error initializing a simple filtergraph
  2025-07-29 16:57:53.775575817  [2025-07-29 16:57:53] ffmpeg.alley.detect            ERROR   : Error opening output file pipe:.
  2025-07-29 16:57:53.776048525  [2025-07-29 16:57:53] ffmpeg.alley.detect            ERROR   : Error opening output files: Generic error in an external library

Edit: Other issues where this error occurs:

• Select which GPU for frigate.embeddings_manager blakeblackshear/frigate#18018
• [Support]: ffmpeg unable to use nvidia GPU when run not in privileged mode blakeblackshear/frigate#18722

I have also opened a ticket on the ffmpeg bug tracker: https://trac.ffmpeg.org/ticket/11694

[elezar]

From the linked posts, it seems as if the default behaviour of some part of the code is to use the first (0th) device even if another one is selected. This is why things sometimes work when a container is started as privileged since the entire /dev filesystem from the host is then accessible to the container.

Note that when a container is started with a specific GPU, the index of that device in the container would be expected to be 0 (even if it is 1 on the host). Ideally, the environment in the container would behave as if it were a host where only that device was installed, but there are some subtle differences here. Most noteable, the device nodes from the host are NOT remapped. This means that the 0th device in the container may have a device node other than /dev/nvidia0 associated with it AND the /dev/drm symlinks will also be the ones associated with the actual host device.

If an application such as ffmpeg makes assumptions about mapping of indices, then these may not hold in this case and the application will fail unless updated. What I can recommend:

• Try to put together a minimal reproducer -- ideally not dependant on camera inputs etc.
• Run this minimal reproducer through something like strace in the container to check whether there is some unexpected device node being accessed.
• Map this back to points in the application code where this access may be triggered.

Note that it could well be that the error comes from the expectations of CUDA itself -- in which case we would have to improve our containerized test coverage internally to be able to reproduce this, and -- if possible -- address it.

[baudneo]

We fixed things by allowing all GPUs and targeting specififc gpu ids for each component that uses anything CUDA related. There does seem to be some mismatch when only allowing specific GPUs and the system then treating that gpu as index 0 rather than whatever index it actually is. I'll try to put together some data to track down what the issue may be.

Thanks for your reply!

[ehfd]

https://forums.developer.nvidia.com/t/nvenc-and-nvdec-work-on-only-one-gpu-with-multi-gpu-setups-with-nvidia-container-toolkit-in-565/347361
Brought this up with the driver team.

More things to add are that GStreamer nvcodec plugin is also affected by this issue, and that the only GPU working in my case is /dev/nvidia2, the third GPU out of five (0-4) GPUs.

More issues:
#1249
NVIDIA/k8s-device-plugin#1282

[baudneo]

What ended up working fort my use case is passing all devices through to the target container and then using the different app configs to target specific gpu index ids for each accelerated component.

Another thing I am noticing, is that I can restrict certain devices and they arent visible to nvidia-smi in the container, but I can still use the restricted, invisible device to load ONNX models and run inference (albeit at massively reduced performance: 200+ ms vs 18-20ms) in the container. IDK if this has any relation to the current issue, but it is something I thought was worthy to note.

[Zblocker64]

this is that I did for it to work for me

NVIDIA/k8s-device-plugin#1282 (comment)

[wenzt]

https://forums.developer.nvidia.com/t/nvenc-and-nvdec-work-on-only-one-gpu-with-multi-gpu-setups-with-nvidia-container-toolkit-in-565/347361 Brought this up with the driver team.

More things to add are that GStreamer nvcodec plugin is also affected by this issue, and that the only GPU working in my case is /dev/nvidia2, the third GPU out of five (0-4) GPUs.

More issues: #1249 NVIDIA/k8s-device-plugin#1282

@ehfd Same to you, only /dev/nvidia2 working.
Any update ?

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[flexgrip]

Commenting to keep this open.

[flexgrip]

I am at a loss at what I can do next because I am unsure if this is an issue with nvidia or ffmpeg (nvdec specifically). Issue has been observed in the frigate tensorrt images and ubuntu cuda images with ffmpeg 7.

Pretty certain it is an issue in the driver. When NVENC or NVDEC initializes inside a container, it asks the kernel for a list of all GPUs on the host — not just the one assigned to your container. When it sees multiple GPUs, it tries to set up peer connections to all of them, fails because those other GPU device nodes don't exist in your container, and bails out with "unsupported device." FFmpeg is just the messenger reporting the failure. The bug lives in NVIDIA's user-space libraries (libnvidia-encode.so and libnvcuvid.so)

I ended up making a shared library that can fix this from inside the container. But it really needs to be fixed in the driver.