Automatically provisioning X11 and Wayland devices of GPU inside container?

[ehfd]

Redirected from NVIDIA/k8s-device-plugin#206 to a more suitable repository.

1. Issue or feature description

In docker and kubernetes, people have had to have manual host setup to provision the X server using host path directive /tmp/.X11-unix. This is quite tedious for sysadmins and at the same time a security threat as people can spoof the host.

To mitigate this, there have been attempts (https://github.com/ehfd/docker-nvidia-glx-desktop which is based on https://github.com/ryought/glx-docker-headless-gpu) to execute an X server and use GLX inside the container after getting provisioned the GPU using libnvidia-container.

An alternative was created by the developers at VirtualGL (used widely in HPC to enable GPU-based rendering in VNC virtual display environments) have been able to develop a feature that uses the EGL API to enable 3D GL rendering such as Blender, Matlab, and Unity, previously only possible with GLX and thus an X server. As you guys know well, nvidia-docker does not support GLX but has introduced the EGL API just below two years ago.
See EGL config section of VirtualGL/virtualgl#113 (comment)

EGL is also required to start a Wayland compositor inside a container with the EGLStreams specification in NVIDIA GPUs, which is the way forward after X11 development has stopped.

These use cases require access to the devices /dev/dri/cardX corresponding to each GPU provisioned using libnvidia-container. However, it does not seem like libnvidia-container provisions this automatically. I would like to ask you whether this is possible, and how this can be configured.

2. Steps to reproduce the issue

Provision one GPU inside container nvidia/cudagl:11.0-devel-ubuntu20.04 or nvidia/opengl:1.2-glvnd-devel-ubuntu20.04 in Docker CE 19.03 (or using one nvidia.com/gpu: 1 with k8s-device-plugin v0.7.0 with default configurations in Kubernetes v1.18.6).

Do: ls /dev

Result: Inside the container you see /dev/nvidiaX, /dev/nvidia-modeset, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, HOWEVER directory /dev/dri does not exist.
Wayland compositors are unlikely to start inside a container without DRM devices. VirtualGL does not work through any devices other than /dev/dri/cardX as well.

3. Information to attach (optional if deemed irrelevant)

Other issues and repositories:
Example of VirtualGL EGL configuration that requires /dev/dri/cardX: https://github.com/ehfd/docker-nvidia-egl-desktop

Implementation of an unprivileged remote desktop bundling an X server with many hacks: https://github.com/ehfd/docker-nvidia-glx-desktop

[klueska]

I have added this feature request to our backlog. At present we have a big backlog, so it's unclear exactly when we will be able to look at this in detail.

That said, it feels like it could be added as a new NVIDIA_DRIVER_CAPABILITY that tries to look for these devices if they exist and inject them. You would set this capability either in the container image or the command line via an environment variable (which would work in the k8s context as well).

[ehfd]

As you see the thumbs up, this feature is in quite a big demand, so it would be great to be implemented quickly. Thank you.

[xkszltl]

If you get a chance to do that, maybe add /dev/gdrdrv for nvidia gdrcopy as well.

[ehfd]

https://github.com/mviereck/x11docker/wiki/Hardware-acceleration#share-nvidia-device-files-with-container

To use a custom base image, share all files matching /dev/nvidia*, /dev/nvhost* and /dev/nvmap with docker run option --device. Share /dev/dri and /dev/vga_arbiter, too. Add container user to groups video and render with --group-add video --group-add render.

In addition to the initial feature request, these are all the devices required to be provisioned automatically for NVIDIA to officially support Display (e.g. X11, Wayland) in Docker. If these devices are able to be provisioned using the container toolkit automatically, the nvidia/opengl container (nvidia-docker) can properly support the NVIDIA version of XWayland (currently undergoing support into the Linux kernel by NVIDIA devs) and thus support Displays.

There are a lot of people waiting for Display support in Docker and Kubernetes, especially because NVIDIA is to support XWayland in the near future. Please implement this feature to streamline this.

[ehfd]

Any updates? @klueska
I was able to start up an unprivileged X server inside an OCI Docker container with nvidia-docker in https://github.com/ehfd/docker-nvidia-glx-desktop, but thinking ahead to Wayland support (since the 470 driver is out), we likely this.

[ehfd]

Please use https://gitlab.com/arm-research/smarter/smarter-device-manager for /dev/dri/card* and /dev/dri/render* if you stumble upon this issue.

[ehfd]

EGL does not require /dev/dri for NVIDIA devices. VirtualGL has merged support for GLX over EGL without such devices.

[ehfd]

Still likely needed for Wayland with GBM.

[elezar]

Thanks @ehfd. We are working on improving the injection of these devices in an upcoming release. Note that the current plan is to do so using the nvidia-container-runtime at an OCI runtime specification level instead of relying on the NVIDIA Container CLI.

Do you have samples containers / test cases that you would be able to provide to ensure that we meet the requirements?

[ehfd]

@elezar
https://github.com/ehfd/docker-nvidia-glx-desktop/blob/main/entrypoint.sh
https://github.com/ehfd/docker-nvidia-egl-desktop/blob/main/entrypoint.sh

These two repositories involve a series of hacks to make NVIDIA GPUs work reliably inside a container unprivileged with a properly accelerated GUI.

docker-nvidia-glx-desktop must install the userspace driver components at startup mostly following your examples but after reading from /proc/driver/nvidia/version because libraries aren't injected to the container.

In the current state, the same userspace driver installation must be done for Wayland by reading /proc/driver/nvidia/version as well. This is undesirable.

Also, in docker-nvidia-egl-desktop, where the userspace drivers aren't installed at startup, an annoying situation arises, where Vulkan requires the display capability of NVIDIA_DRIVER_CAPABILITIES must be included because nvidia_icd.json requires libGLX_nvidia.so.0 and probably more other libraries even when not using Xorg with the NVIDIA driver.

Vulkan should be possible only with the graphics capability as intended, but it requires display as well. #140 Thank god it does work without major modifications to libnvidia-container.

And the display feature currently does not enable starting an Xorg server with the NVIDIA driver in its current state, because of the lack of the libraries being injected to the container. Hence the hacks applied by these two containers are required.

Please also consider injecting the necessary libraries for NVFBC with the video capability as well, even if the SDK must be installed inside the container.

We really hope that NVIDIA_DRIVER_CAPABILITIES starts working properly and that the hacks that my containers applied won't be needed anymore. These can all likely be done at OCI runtime specification level.

Note that we currently use https://gitlab.com/arm-research/smarter/smarter-device-manager for provisioning /dev/dri devices, but there is no methodology to push just the devices for the GPU allocated to the container.

Thanks a lot!

[elezar]

Thanks for all the information. I will comb through it while working on the feature. Hopefully we can improve things significantly!

[Zubnix]

Hi @elezar @ehfd,

I'm writing a remote Wayland compositor and am currently busy integrating it with k8s and can independently confirm everything @ehfd has stated so far, as I've hit all of them in the last couple of weeks. Being able to access /dev/dri/renderDevice12x and /dev/dri/cardx while limiting, preferably eliminating startup actions and driver dependencies of a container is an absolute must.

@elezar I'm happy to assist and answer any questions you might have to help move this forward!