Scrape failed: "i/o timeout" and missing Pod metadata

[jpboy8]

I am encountering i/o timeout errors when Prometheus tries to scrape metrics from dcgm-exporter. Additionally, I am struggling to get Pod mapping working correctly (metrics are missing pod, namespace labels).

Deployment method: Helm Chart 4.7.1
Image: nvcr.io/nvidia/k8s/dcgm-exporter:4.5.1-4.8.0-distroless
Kubernetes: 1.33.6
GPU: Nvidia H100 (4 MIG)

values.yaml

kubernetes:
  enablePodLabels: true
  enablePodUID: true
  rbac:
    create: true
image:
  repository: nvcr.io/nvidia/k8s/dcgm-exporter
  pullPolicy: IfNotPresent
  tag: 4.5.1-4.8.0-distroless
extraEnv:
  - name: DCGM_EXPORTER_INTERVAL
    value: "1000"
  - name: DCGM_EXPORTER_KUBERNETES_GPU_ID_TYPE
    value: "device-name"
serviceMonitor:
  enabled: true
  interval: 60s
  honorLabels: true
  additionalLabels:
    release: kube-prometheus-stack

LOGS

time=2026-02-02T04:42:44.018Z level=ERROR msg="Failed to write response." error="write tcp 10.42.0.54:9400->10.42.0.66:47334: i/o timeout"
time=2026-02-02T04:42:44.018Z level=INFO msg="http: superfluous response.WriteHeader call from github.com/NVIDIA/dcgm-exporter/internal/pkg/server.(*MetricsServer).Metrics (server.go:257)

[xiaoshutong2]

same issue

[jaeeyoungkim]

Hi @xiaoshutong2 @jpboy8

I was experiencing similar performance and timeout issues, especially in large MIG environments.
I've submitted a PR (#626) that replaces the synchronous API calls with a SharedInformer cache. In my tests (8x A100, 56 MIG slices), this reduced CPU load from 10+ cores to just 2~3 cores and completely eliminated the hangs.

If anyone wants to test the fix while it's waiting for review, I've published a test image here:
public.ecr.aws/whatap/dcgm-exporter:4.5.1-4.8.0-pr-test-ubuntu22.04

Any feedback on the PR if this works for you would be greatly appreciated!

[nccurry]

This should be fixed in #626
@jpboy8 If you find that #626 isn't sufficient, please re-open.

Thanks!