prometheus.exporter.cadvisor leaks one containerd gRPC ClientConn per container start on Docker's containerd image store

[voron]

Component(s)

prometheus.exporter.cadvisor

What's wrong?

On a Docker host whose storage driver is the containerd snapshotter (docker info → Storage Driver: overlayfs, driver-type: io.containerd.snapshotter.v1, the Docker 29 default), every container start makes prometheus.exporter.cadvisor open a new gRPC ClientConn to /run/containerd/containerd.sock that is never closed. Each leaked connection holds 5 goroutines (3× grpcsync.(*CallbackSerializer).run, transport.(*loopyWriter).run, the HTTP/2 reader) plus its buffers, so go_goroutines and heap grow linearly with the number of container starts for the life of the process.

On hosts that run a docker run --rm job on a one-minute timer this is +4320 goroutines per day per job and roughly +10–18 MB heap per day; one host reached 65k goroutines after 7.5 days. On a 1 GB VM without swap the growth ended in a major-page-fault thrash. Hosts on overlay2 never take this code path and stay flat.

Steps to reproduce

1. Docker 29.x with the containerd image store (the default), Alloy v1.19.2 as a systemd service with the config below.
2. while true; do docker run --rm alpine true; sleep 60; done
3. Watch go_goroutines on Alloy's /metrics: +5 per container start, never released. ss -xp | grep containerd.sock shows Alloy's connection count to containerd growing by one per start.
4. Goroutine profile (/debug/pprof/goroutine?debug=2): the accumulating CallbackSerializer goroutines are all created by github.com/google/cadvisor/manager.(*manager).watchForNewContainers.func1.

Goroutine-profile diff over 15 minutes / 15 container starts on one host (everything else flat):

+45  google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run
+15  google.golang.org/grpc/internal/transport.(*loopyWriter).run   (created by NewHTTP2Client)
+15  google.golang.org/grpc/internal/transport/readyreader.(*bufReadyReader).Read

Config:

prometheus.exporter.cadvisor "containers" {
  docker_host = "unix:///var/run/docker.sock"
}

prometheus.scrape "cadvisor" {
  targets    = prometheus.exporter.cadvisor.containers.targets
  forward_to = []
}

Root cause

Alloy replaces github.com/google/cadvisor with the grafana/cadvisor fork, branch grafana-v0.54.1-noglobals (go.mod at v1.19.2 and on main: replace github.com/google/cadvisor => github.com/grafana/cadvisor v0.0.0-20260204200106-865a22723970, commented "non-singleton cadvisor fork").

In that fork:

• container/docker/handler.go newDockerContainerHandler calls opts.ContainerDClient() when storageDriver == ContainerdSnapshotterStorageDriver.
• container/docker/client.go (*Options).ContainerDClient() builds a fresh containerd.Options{} on every call and returns cOpts.Client(...).
• container/containerd/client.go (*Options).Client guards its grpc.DialContext with opts.once — a sync.Once that lives on that throwaway Options, so there is a new dial per container handler and nothing ever closes the connection.

Upstream cadvisor v0.54.1 keeps var once sync.Once; var ctrdClient ContainerdClient at package level in container/containerd/client.go and the docker handler uses that shared client, so upstream does not leak. The regression is in the fork's de-globalisation: a per-Options once is right for the containerd exporter (one Options per component instance), but the docker exporter builds a new containerd.Options per container instead of holding one per component.

Suggested fix

Give the docker Options a single lazily-initialised containerd client (a containerd.Options field with its own sync.Once), so ContainerDClient() returns the same ClientConn for the lifetime of the exporter instance, and close it on component shutdown.

Environment

• Alloy v1.19.2 (deb package, systemd), Linux amd64, Go 1.26.7 per the binary
• Docker 29.7.2 with the containerd image store, Debian 13 (trixie)
• Workarounds verified: (a) not creating containers on a timer — moving the periodic job into a long-lived container driven by docker exec made go_goroutines flat over the same window in which an untouched host kept growing at +3/min; (b) bounding the process with GOMEMLIMIT + systemd MemoryMax, so the leak ends in an in-cgroup OOM kill and a Restart=always restart (positions intact) rather than host memory pressure.

[exSnake]

Independent confirmation, with two details that may be worth having.

It is not a 1.19.x regression same behaviour on v1.18.0, Docker 29.7.1 / containerd v2.2.6 with the containerd snapshotter, Debian 13. Rate matches yours closely: docker run -d --restart=always alpine sh -c 'sleep 60; exit 1' gave +308 goroutines/hour, i.e. ~4.9 per container restart, linear over 24 minutes and 25 restarts (442 -> 565). The control with no churn on the same host was flat: -3/hour over 4.7 hours. An Alloy restart reclaimed all of it.

The log tailers are not involved. That config also runs loki.source.docker. Commenting out only prometheus.exporter.cadvisor and its prometheus.scrape, leaving the Docker log tailers enabled and the same churn running, made it flat: 312 -> 314 over 15 minutes and ~15 restarts.

The goroutine profile agrees with your root cause, the accumulating bucket is grpcsync.(*CallbackSerializer).run, and it stood at 174 against only 13 live HTTP/2 transports (transport.(*controlBuffer).get), which is what a ClientConn created and dropped without Close() leaves behind. Every other bucket tracked the container count exactly and stayed there.