dcgm exporter conflict with self-installed dcgm exporter

[honghan-wong]

Description

we are using helm installed gpu-operator which installed another set of dcgm exporter in the kubernetes cluster.
the dcgm exporter installed by hardware observer is conflicting with helm installed dcgm exporter with hostNetwork.

We need to have a way to scrape the helm installed dcgm exporter properly.

Charm revision

755

Juju version

3.6.21

Cloud

Kubernetes

Expected behaviour

COS is able to scraping helm installed dcgm exporter.

Reproduce / Test

When we want to scrape the metrics from dcgm exporter, we need to

1. stop the snap installed dcgm exporter from hardware observer
2. keep the relation between hardware observer with grafana-agent-k8s for the scrape to work.
3. enable hostNetwork in helm installed dcgm exporter for scraping.

Notes & References

No response

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/SOLENG-2106.

This message was autogenerated

[jneo8]

Hi @gabrielcocenza , I remember you handle this issue before?

[gabrielcocenza]

There are some possible ways to handle this:

1. Do as you said:

• stop the snap installed dcgm exporter from hardware observer
• keep the relation between hardware observer with grafana-agent-k8s for the scrape to work.
• enable hostNetwork in helm installed dcgm exporter for scraping.

This solution was discussed some time ago, but didn't got traction. See the docs

An alternative to not use hostNetwork (preferable because of security) is to use the nvidia-gpu-operator-tf using the prometheus-scrape-target-k8s

I guess in both ways, we need to have a config to not install the dcgm snap when the helm deployment because they cannot work together.

@jneo8 Maybe we can add a new possible value for dcgm-snap-channel as disable or something similar to not install or remove the dcgm snap