VEP #254: Guest GPU Metrics via VSOCK

[machadovilaca]

VEP Metadata

Tracking issue: #254
SIG label: /sig observability /sig compute

What this PR does

GPU workloads running inside KubeVirt virtual machines currently lack observability. Cluster administrators and users have no way to monitor GPU utilization, memory usage, temperature, power consumption, or error counts for GPUs passed through to VMs.

This VEP introduces a mechanism for collecting GPU metrics from inside the guest and exposing them as Prometheus metrics on the host.

Special notes for your reviewer

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vladikr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alaypatel07]

/cc @rthallisey

[dominikholler]

Windows support is upcoming, and I would be surprised if Linux guests which are too old for vsock would utilize GPUs

[machadovilaca]

@enp0s3 said the exactly same on kv vep discussion meeting, if the general opinion is that "Windows support is upcoming" is fine, then I would be okay with it

I actually tried VSOCK first, and was able to get a poc running, ditched it because of the windows support
and since we already used virtio-serial, seemed like a good approach

[kostyanf14]

Windows already supported

[machadovilaca]

updated to use DCGM with VSOCK

[dominikholler]

@kostyanf14 (and maybe @jcanocan ) Do you agree?

[kostyanf14]

@YanVugenfirer

[jcanocan]

Virtio-serial is being used already by downward metrics and it works fine. However, I don't have any experience in Windows guests.

[kostyanf14]

Reject for Virtio-serial because we see some problems with huge data transfer virtio-win/kvm-guest-drivers-windows#1462

[machadovilaca]

here the data sent should be small, i shared the structure in the VEP document

[machadovilaca]

@dominikholler @kostyanf14 added vsock as an alternative in the design section

[machadovilaca]

/cc @michalskrivanek

added guest-file-read as an alternative in the design section

[rthallisey]

Should kubevirt be responsible for maintaining a gpu metric system? This seems generally useful for virtualization.

[machadovilaca]

KubeVirt wouldn't really be maintaining a GPU metric system. The heavy work would be done entirely by DCGM inside the guest, and KubeVirt would handle the communication and exposition of the data.

With that said, NVIDIA DCGM exporter is the de facto way to expose GPU metrics to Prometheus, so it would be ideal and simpler for users if it stayed that way. What I think that would mean:

1- KubeVirt creates a VSOCK for the VMI if any GPU device is configured
2- DCGM listening of VSOCK is running on the guest
3- NVIDIA GPU Operator deploys DCGM exporter on nodes configured for vGPU and GPU passthrough
4- DCGM exporter connects to VMI's via VSOCK to query DCGM data and expose metrics to Prometheus

I would then add some Prometheus recording rules to correlate DCGM exporter metrics with KubeVirt VMI's.

[enp0s3]

@machadovilaca I think that adding the VOSCK automatically will be a problematic approach, the correct way to configure VSOCK for a VM is via the KubeVirt API.

[rthallisey]

Using Vsock can bypass the launcher and go right to the handler or another node-loacl exporter. I can see some advantages to that.

[dominikholler]

I do not understand the full meaning of this comment. I just want to highlight that it might be beneficial to avoid bypassing KubeVirt APIs, e.g. because of #223

[rthallisey]

The data path in the vsock approach is simpler, e.g. guest -> exporter.

[rthallisey]

This is changing. The DCGM team is working on adding vsock support so we can gather metrics from inside guest and share it over vsock. Once implemented, it would make sense to always have DCGM exporter on the host.

[machadovilaca]

do you have any pointers for that work? I tried proposing the support for vsock on dcgm exporter and they suggested pursuing other approaches

NVIDIA/dcgm-exporter#649 (comment)

[rthallisey]

https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html#features

Host engine

        Added support for listening on the VSOCK protocol.

        Added support for the following fields:

                DCGM_FI_DEV_GET_GPU_RECOVERY_ACTION

                DCGM_FI_DEV_GPU_RECOVERY_ACTION

                DCGM_FI_DEV_MEMORY_UNREPAIRABLE_FLAG

                DCGM_FI_DEV_NVLINK_ECC_DATA_ERROR_COUNT_TOTAL

                DCGM_FI_DEV_NVLINK_PPCNT_IBPC_PORT_XMIT_WAIT

[machadovilaca]

updated to use DCGM with VSOCK

[enp0s3]

@machadovilaca Hi. Actually I don't see any added value for the vGPU metrics to be maintained by the KubeVirt tree. It will couple DCGM metrics development with KV release cycle. I think that DCGM can manage with its own metrics exporter.

[enp0s3]

@machadovilaca @rthallisey To be more specific, I don't understand the drawbacks of the following approach:

1. Aggregate VMI to vGPU metrics on a higher level using Prometheus PromQL
2. Using the regular VM pod network, attach kubernetes service the VM network and collect the metrics that way, instead of using VSOCK.

[enp0s3]

@rthallisey Another drawback of using VSOCK is that we are willing to graduate it to be namespace confined, which might impose obstacles for current DCGM design, leading to the need of escalating the privileges of DCGM to be able to access every network namespace on the node.

[machadovilaca]

@machadovilaca @rthallisey To be more specific, I don't understand the drawbacks of the following approach:

1. Aggregate VMI to vGPU metrics on a higher level using Prometheus PromQL
2. Using the regular VM pod network, attach kubernetes service the VM network and collect the metrics that way, instead of using VSOCK.

@enp0s3

1. whether or not we decide to support this VSOCK DCGM approach, if the metrics Prometheus ingests are coming from DCGM directly, we would always need a way to correlate them to VMIs. And even if we end up not doing any action to simplify this metric collection, we can still provide recording rules for the correlations

2. it is just a question of simplifying the work for the end user, the approach you are suggesting should even work right now. But for each VMI the user now needs to: 1. configure network for the vmi, 2. install dcgm in the vmi, 3. create the k8s service, 4. create a service monitor, 5. correlate gpu and vmi metrics

imo we should look for ways to simplify this

[rthallisey]

Another drawback of using VSOCK is that we are willing to graduate it to be namespace confined

Fair point. Perhaps though there's a way to solve global vsock with some security hardening. I'd like to at least explore that option.

The two use cases I see are:

1. Advertise kubevirt specific gpu metrics that tie vmi to gpu
2. Advertise all gpu metrics from a passthrough GPU to the dcgm-exporter

Use case 1 is meant for kubevirt admins. Use case 2 can tie into the existing GPU tooling ecosystem.

[enp0s3]

@machadovilaca @rthallisey To be more specific, I don't understand the drawbacks of the following approach:

1. Aggregate VMI to vGPU metrics on a higher level using Prometheus PromQL
2. Using the regular VM pod network, attach kubernetes service the VM network and collect the metrics that way, instead of using VSOCK.

@enp0s3

1. whether or not we decide to support this VSOCK DCGM approach, if the metrics Prometheus ingests are coming from DCGM directly, we would always need a way to correlate them to VMIs. And even if we end up not doing any action to simplify this metric collection, we can still provide recording rules for the correlations

+1 for recording rules, IMO its much more simpler then the current approach.

2. it is just a question of simplifying the work for the end user, the approach you are suggesting should even work right now. But for each VMI the user now needs to: 1. configure network for the vmi, 2. install dcgm in the vmi, 3. create the k8s service, 4. create a service monitor, 5. correlate gpu and vmi metrics

imo we should look for ways to simplify this

this can be simplified using tools like Helm.

[enp0s3]

I would prefer this VEP to be eventually converged with VEP 143, if we will go with the VSOCK approach I would like the code to live in the separate monitoring stack

[enp0s3]

How are we going to support Windows? Do we have a way to test this?

[enp0s3]

We need to mention that VSOCK had to be requested in the VM spec

[machadovilaca]

my idea was that if the feature gate is enabled and GPUs are present in the spec, virt-controller would automatically attach a VSOCK device to the domain XML

[enp0s3]

@machadovilaca But its only relevant for NVIDIA GPUs, isn't it?

[machadovilaca]

initially my idea was to keep this generic enough to handle other gpus in the future
but with some of the work moving to the dgcm exporter supporting vsock, and nvidia operator deploying the exporter on all use cases, it is now more focused on nvidia, yes

[enp0s3]

IMO its suboptimal to always attach VSOCK upon GPU resource request.

[machadovilaca]

i think many times if a gpu is attached, dcgm will be installed in the guest alongside with the drivers, so the dcgm exporter would be able to collect metrics with no additional user operation over what he does today

but i can change that. what do you think is better, using the existing attach vsock field or creating a new one specific for gpu metrics, that if enabled, would create the vsock?

[enp0s3]

@machadovilaca I would leave the responsibility to attach the VSOCK for the user. In case the VSOCK isn't attached via the VM spec we won't collect the metrics.

[machadovilaca]

ok

then with the new dcgm exporter support for vsock
we should be able to just add some documentation for the users to enable vsock on the spec, install dcgm in the guest, and start the dcgm service listening on the vsock

now, nvidia needs to update nvidia gpu operator to deploy the the dcgm exporter on nodes configured for vgpu or gpu passthrough, which it currently doesn't

on kubevirt side, we would only need to add the recording rules, that correlated the dcgm exporter metrics, with the vmis

[enp0s3]

@machadovilaca Can you please add in the alternative why inline communication method using regular kubernetes network was rejected? What are the drawbacks of deploying additional kubernetes resources in order to expose the metric service that is running inside the guest?

In addition what are the drawbacks of using recording rules or any other high level tools to tie VM to GPU? Why explicit instrumentation is needed to tie these resources?

[machadovilaca]

added DCGM via regular Kubernetes Networking alternative

recording rules are not an issue, even if we expose VSOCK for DCGM exporter to collect the DCGM metrics, we would need the recording rules

[enp0s3]

@machadovilaca I would leave the responsibility to attach the VSOCK for the user. In case the VSOCK isn't attached via the VM spec we won't collect the metrics. sorry confused with another thread.

[enp0s3]

/lgtm

@machadovilaca Thank you!

[enp0s3]

@vladikr Hi, could you please have a look?

[enp0s3]

@machadovilaca Three more topics we should converge:

• Windows support, can we commit to it?
• Combined consumption of VSOCK, both by the metric collector and by KubeVirt API. We should mention the limitations.
• The trigger to collect the metrics, there can be non-NVIDIA GPU with attached VSOCK, how would we differ that case?

[vladikr]

@machadovilaca @enp0s3 To be honest, personally, I didn't have the capacity to review this proposal deep enough to understand whether it fits KubeVirt or not ...
I am not able to give my approval for this cycle.
Let's defer it to the next one, which will give us enough time to have a proper discussion.

[kubevirt-bot]

New changes are detected. LGTM label has been removed.

[enp0s3]

@vladikr Makes sense. Sorry for the noise. It looks like we need more time.

[rthallisey]

DCGM-exporter running on the host will poll for metrics over the socket. DCGM should be running in the guest and listening on the socket. Having dcgm-exporter or dcgm connect to the launcher over vosck doesn't sound right to me.