How to configure dcgm metrics for MIG?

[laszlocph]

Hello,

I am kinda in a rabbit hole:

• DCGM_FI_DEV_GPU_UTIL is not supported for MIG devices
  DCGM_FI_DEV_GPU_UTIL with MIG devices DCGM#80 (comment)

• DCGM_FI_PROF_SM_OCCUPANCY could be a substitute, but it is disabled by default in
  kubectl exec -it nvidia-dcgm-exporter-rh46x -- cat /etc/dcgm-exporter/dcp-metrics-included.csv | less

• To enable DCGM_FI_PROF_* I found this issue, but the refferred piece of documentation is gone:
  GPU-operator doesn't allow to specify a volume to mount metrics file for nvidia-dcgm-exporter #275 (comment)

Anybody managed to monitor MIG devices memory utilization?
Anybody managed to configure custom metrics for dgcm-exporter?

Thank you.

[frittentheke]

@laszlocph I ran into the same issues and raised an issue with the DCGM exporter: NVIDIA/dcgm-exporter#353

[frittentheke]

@laszlocph maybe you'd like to a take a look at my dashboard PR - NVIDIA/dcgm-exporter#355
Depending on the result of NVIDIA/dcgm-exporter#353 (de-duplication of metrics due to e.g. MIG) I might be able to remove some of the aggregations again.

There now also is a dedicated issue about cleaning up the labels: NVIDIA/dcgm-exporter#356

[laszlocph]

@frittentheke Amazing. This did it for us.
Now we have utilization metrics for MIG partitions 🥳