Exporter crashes at start when loading libdcgm.so.4 on EKS 1.34 / Bottlerocket 1.54

[mindw]

What is the version?

4.5.2-4.8.1

What happened?

The dcgm container crushed at start with with very dew logs (--debug, and dcgm log at were enabled):
"the libdcgm.so.4 library was not found".
The same image start without issues on EKS 1.33.
Investigation revealed the /etc/ld.so.cache was missing a libdcgm.so.4 entry when running on 1.34.
Changed pod command argument to refresh the ldconfig cache before starting the exporter allowed it to start normally.:

      - args:
        - |
          rm /etc/ld.so.cache; ldconfig; exec /usr/local/dcgm/dcgm-exporter-entrypoint.sh -f /etc/dcgm-exporter/default-counters.csv
        command:
        - /bin/bash
        - -exc

What did you expect to happen?

The exporter start normally and serve metrics

What is the GPU model?

AWS g6.xlarge

What is the environment?

EKS 1.34
bottlerocket-nvidia 1.34

How did you deploy the dcgm-exporter and what is the configuration?

Deployed using helm-chart.

How to reproduce the issue?

Run dcgm-exporter on EKS 1.34 g6 node using bottlerocket 1.54+

Anything else we need to know?

• All three images failed (distroless, ubi9 & ubuntu22.4).
• All three images started normally in EKS 1.33 bottlerocket 1.51
• nodes were started using karpenter - the following useData was used:

              [settings.boot]
              reboot-to-reconcile = true

              [settings.boot.kernel-parameters]
              "psi" = ["1"]
              "nvidia.NVreg_EnableGpuFirmware" = ["1"]

              [settings.kubernetes.nvidia.container-runtime]
              "visible-devices-as-volume-mounts" = true
              "visible-devices-envvar-when-unprivileged" = true

              [settings.metrics]
              send-metrics = false

We have "Bottlerocket OS 1.55.0 (aws-k8s-1.34-nvidia)" on EKS 1.34,with Karpenter.
We don't have any custom settings.

DCGM Exporter works on our side

Regards

[sammoseley]

EKS 1.34
Bottlerocket OS 1.55.0 (aws-k8s-1.34-nvidia)
nvcr.io/nvidia/k8s/dcgm-exporter:4.5.2-4.8.1-distroless

All DCGM pods crashloopbackoff, very little/nothing useful in pod logs.

Previous DCGM image (nvcr.io/nvidia/k8s/dcgm-exporter:4.4.1-4.6.0-ubuntu22.04) appears fine. Only change we made was the chart version bump

[mindw]

@bdupuis-cell @sammoseley thank you for the reports!.
Can you please add the following additional information:

1. Is the nvidia-operator used?
2. please share the running pod's yaml (kubectl get pod <> -o yaml)

I suspect that the nvidia-container-runtime update-ldcache hook was not executed for some reason.

Hi,

We do not use nvidia-operator

I can't attach pod description name has a file extension that does not match the content type: .yaml != application/yaml :~x