Nvidia GPGPU Pass-Through with KVM

Introduction

This is a quick example of how to configure Nvidia GPGPU pass through to a Linux KVM VM.

This guid is for Intel hardware, but a similar approach can be take for AMD.

If successful, as a simple example, you should be able to do something like this from with the KVM VM:

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.138                Driver Version: 390.138                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M2090         Off  | 00000000:04:00.0 Off |                    0 |
| N/A   N/A    P0    74W /  N/A |      0MiB /  5301MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla M2090         Off  | 00000000:42:00.0 Off |                    0 |
| N/A   N/A    P0    77W /  N/A |      0MiB /  5301MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

License

CC BY-SA: https://creativecommons.org/licenses/by-sa/4.0/

Fund me here: https://ko-fi.com/richardatlateralblast

Resources

Here are some links that while not providing me with the solution, gave me some information to point me in the right direction.

Configure GPU Passthrough for Virtual Machines:

https://www.server-world.info/en/note?os=CentOS_7&p=kvm&f=10

KVM: Testing cloud-init locally using KVM for an Ubuntu cloud image:

https://fabianlee.org/2020/02/23/kvm-testing-cloud-init-locally-using-kvm-for-an-ubuntu-cloud-image/

PCI passthrough:

https://pve.proxmox.com/wiki/Pci_passthrough

Virtual machines with PCI passthrough on Ubuntu 20.04, straightforward guide for gaming on a virtual machine:

https://mathiashueber.com/pci-passthrough-ubuntu-2004-virtual-machine/

Cloud Config Examples:

https://cloudinit.readthedocs.io/en/latest/topics/examples.html

Using Cloud Images With KVM:

https://serverascode.com/2018/06/26/using-cloud-images.html

Requirements

Required hardware:

• Nvidia Cuda cable GPU
• Intal / AMD IOMMU Enablement in BIOS
• Intel / AMD IOMMU Enablement in grub

Required software:

• KVM
• Nvidia Drivers
• Nvidia Toolkit and Samples (optional)

A reboot is required to install Nvidia drivers

Overview

This example does the following:

• Determine Nvidia PCI Device IDS
• Enable IOMMU and pass management of devices to vfio
• Disable Nvidia drivers on host
• Install KVM
• Determine Nvidia PCI IDs
• Create KVM VM
• Install Nvidia drivers in KVM VM

Example

This requires you enable IOMMU ot VT-d in BIOS.

Determine Nvidia PCI IDs:

sudo lspci  |grep -i nvidia

04:00.0 3D controller: NVIDIA Corporation GF110GL [Tesla M2090] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1)
42:00.0 3D controller: NVIDIA Corporation GF110GL [Tesla M2090] (rev a1)
42:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1)

Determine Nvidia PCI Device IDs

sudo lspci -ns 04:00.0 |awk '{print $3}'

10de:1091

sudo lspci -ns 04:00.1 |awk '{print $3}'

10de:0e09

Enable IOMMU and pass management of devices to vfio in grub:

cat /etc/default/grub |grep iommu

GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt intremap=no_x2apic_optout vfio-pci.ids=10de:1091,10de:0e09"

The key parts in this are:

intel_iommu=on iommu=pt vfio-pci.ids=10de:1091,10de:0e09 

You may need additional handling to deal with X2APIC alerts/messages:

intremap=no_x2apic_optout

Update GRUB:

sudo update-grub

In general you can work out which modules to block with lspci, e.g.:

lspci -ks 05:00.0

Disable nvidia drivers, e.g.:

sudo sh -c 'echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf'
sudo sh -c 'echo "blacklist nvidia_uvm" >> /etc/modprobe.d/blacklist.conf'
sudo sh -c 'echo "blacklist nvidia_drm" >> /etc/modprobe.d/blacklist.conf'
sudo sh -c 'echo "blacklist nvidia_modeset" >> /etc/modprobe.d/blacklist.conf'
sudo sh -c 'echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf'

Make sure vfio and kvm modules get loaded:

cat /etc/modules-load.d/vfio-pci.conf

vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

Update vfio config:

sudo sh -c 'echo "options vfio-pci ids=10de:1091,10de:0e09 disable_vga=1" > /etc/modprobe.d/vfio.conf'

Reboot:

sudo reboot

After reboot you should see vfio mention the PCI ID of the Nvidia GPU in dmesg or /proc/interrupts:

sudo dmesg |grep vfio |grep add

[    9.601364] vfio_pci: add [10de:1091[ffffffff:ffffffff]] class 0x000000/00000000
[    9.653456] vfio_pci: add [10de:0e09[ffffffff:ffffffff]] class 0x000000/00000000

If devices are not handled by vfio you may need to allow unsafe config/interrupts:

sudo echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

Now we need to install KVM and associated packages:

sudo apt install -y qemu qemu-kvm libvirt-daemon libvirt-clients bridge-utils virt-manager

Add user to appropriate groups (e.g. kvm, libvirt, libvirt-qemu)

Grab a cloud image:

cd /var/lib/libvirt/images
sudo wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img

Copy Image and resize it:

sudo qemu-img convert -f qcow2 -O qcow2 focal-server-cloudimg-amd64.img nvubuntu2004vm01.img
sudo qemu-img resize nvubuntu2004vm01.img 50G

Create a cloud init config:

sudo cat nvubuntu2004vm01.cfg

#cloud-config
hostname: nvubuntu2004vm01
groups:
  - nvadmin: nvadmin
users:
  - default
  - name: nvadmin
    gecos: NvAdmin
    primary_group: nvadmin
    groups: users
    shell: /bin/bash
    passwd: PASSWORDHASH
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
packages:
  - qemu-guest-agent
  - net-tools
  - software-properties-common
  - nvidia-driver-390
  - freeglut3
  - freeglut3-dev
  - libxi-dev
  - libxmu-dev
  - gcc-7
  - g++-7
growpart:
  mode: auto
  devices: ['/']
power_state:
  mode: reboot

Create static IP config:

cat nvubuntu2004vm01_network.cfg

version: 2
ethernets:
  enp1s0:
     dhcp4: false
     # default libvirt network
     addresses: [ 192.168.122.91/24 ]
     gateway4: 192.168.122.1
     nameservers:
       addresses: [ 8.8.8.8,8.8.4.4 ]

Create config image/ISO for install:

sudo cloud-localds --network-config nvubuntu2004vm01_network.cfg nvubuntu2004vm01_cloud.img nvubuntu2004vm01.cfg

Build VM:

sudo virt-install --name nvubuntu2004vm01 --cpu host-passthrough --os-type linux \
--os-variant ubuntu20.04 --host-device 04:00.0 --features kvm_hidden=on --machine q35 \
--disk ./nvubuntu2004vm01.img,device=disk,bus=virtio \
--disk ./nvubuntu2004vm01_cloud.img,device=cdrom --graphics none --virt-type kvm \
--network network=default,model=virtio --import --memory 4096

The key things here are to pass the host device to the VM, and hide KVM (Nvidia drivers will fail to load if it determines it is a virtualised environment):

--host-device 04:00.0 --features kvm_hidden=on