Duplicated, missing or wrong metrics if using MIG, Grafana dashboard showing wrong duplicated / false values

[frittentheke]

What is the version?

3.3.5-3.4.1

What happened?

When activating MIG we saw duplicated and plain wrong metrics in the provided Grafana dashboard (https://github.com/NVIDIA/dcgm-exporter/tree/main/grafana).

The issue seems to be two-fold, with Grafana as well as the raw metrics themselves:

1. Firstly the dashboard: Legends, ... and PromQL queries used to fetch metrics do not take MIG into account. So metrics returning MIG subdevices (GPU_I_ID) are not considered.
   GPU metrics regarding have not been up

2. Secondly the metrics:

• Even if the queries were updated via e.g some aggregations like max(), avg() or sum() to avoid duplication, there are some metrics reported back per GPU_I_ID, that do not have this granularity. See me comment Attributing GPU power among MIG instances. #257 (comment). So if the power draw is not measured per GPU_I_ID you cannot return it individually as you would be returning false values.
• Reading DCGM_FI_DEV_GPU_UTIL with MIG devices DCGM#80 (comment) it seems the GPU metrics should be replaced by DCGM_FI_PROF_* .
• But apparently there are even more open discussions around that: DCGM_FI_PROF_GR_ENGINE_ACTIVE and MIG DCGM#138, Question about DCGM fields DCGM#64 and Questions about fields identifiers DCGM#48.
• This comment by @bstollenvidia seems to sum up quite nicely how things work: Question about DCGM fields DCGM#64 (comment)

What did you expect to happen?

Provided MIG and other ways of partitioning GPUs (vGPU, time-slicing, ...) is quite common, I'd expect the exporter and the provided dashboard to take those into account.

Metrics that are available per-subdevice should be returned, if they are just duplicates of each other, they should be dropped and only returned per "main" GPU.

What is the GPU model?

H100s, using different MIG profiles and whole GPUs

What is the environment?

Kubernetes

How did you deploy the dcgm-exporter and what is the configuration?

Kubernetes with GPU-Operator

How to reproduce the issue?

Enable MIG on a GPU and look at the dashboard.

Anything else we need to know?

There are multiple issues with DCGM or the operator open:

• DCGM_FI_DEV_GPU_UTIL with MIG devices DCGM#80
• How to configure dcgm metrics for MIG? gpu-operator#798

[nvvfedorov]

@frittentheke, Thank you for reporting about the issue. Am I right that the main request is the following: Metrics that are available per-subdevice should be returned; if they are just duplicates of each other, they should be dropped and only returned per "main" GPU.

[frittentheke]

Thanks @nvvfedorov for your fast response!

@frittentheke, Thank you for reporting about the issue. Am I right that the main request is the following: Metrics that are available per-subdevice should be returned; if they are just duplicates of each other, they should be dropped and only returned per "main" GPU.

1. Yes. Please also see my PR ([dashboard] Rework dashboard (MIG support, Grafana deprecations, Hostname) #355) in which I (had to) apply aggregations like max() to work around this for the dashboard. If you'd consider removing those duplicated metrics, I gladly simplify the PromQL queries for the dashboard / my PR (again).

2. If I may add another matter to my findings (which I also hit during the dashboard rework) - the exported labels are somewhat mixed-case with about all variants possible: Hostname vs. DCGM_FI_DRIVER_VERSION vs gpu vs. modelName. Please also consider cleaning this up. Especially when trying to join time-series via a set of labels it's really painful.