azure-cni /v2/ images install a dynamically linked azure-vnet (needs GLIBC_2.32/2.34), breaking CNI on Ubuntu 20.04 nodes during 1.34→1.35 upgrades

[w3aman]

Summary

The azure-cni images published under mcr.microsoft.com/containernetworking/v2/ ship an azure-vnet binary that is dynamically linked and requires GLIBC_2.32 and GLIBC_2.34. The images under the original mcr.microsoft.com/containernetworking/ path ship a statically linked binary with no libc dependency.

Because dropgz / cni-installer copies this binary out of the container and onto the host (/opt/cni/bin/azure-vnet), the container's own base image is irrelevant at runtime — the node's glibc decides whether the binary can run. On Ubuntu 20.04 (glibc 2.31) it cannot, and all pod sandbox creation on that node fails:

plugin type="azure-vnet" failed (add): netplugin failed:
"/opt/cni/bin/azure-vnet: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found
 (required by /opt/cni/bin/azure-vnet)
 /opt/cni/bin/azure-vnet: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
 (required by /opt/cni/bin/azure-vnet)"

Why this matters on AKS

The azure-cns DaemonSet image is selected by the cluster's Kubernetes version, and a DaemonSet by definition rolls out to every node. Observed on AKS:

Cluster Kubernetes version	cni-installer init container image	linkage
1.34	containernetworking/azure-cni:v1.7.16-0	static
1.35	containernetworking/v2/azure-cni:v1.8.11	dynamic (GLIBC 2.32 / 2.34)

During a 1.34 → 1.35 upgrade the control plane is upgraded first, so the new binary is installed on nodes that are still running the old node image. Ubuntu 20.04 is still the OS for FIPS-enabled node pools, so those clusters end up with un-upgraded nodes whose CNI no longer works.

Already-running pods are unaffected (the CNI only runs at sandbox creation), so nothing appears broken. The failure only surfaces when a new pod is scheduled onto a not-yet-upgraded node — which is exactly what happens during the rolling node drain. Any pod that lands there stays Pending indefinitely, and Kubernetes does not reschedule a pod that fails sandbox creation. If that pod is a replica of a workload with a PodDisruptionBudget, the budget can then refuse eviction of its healthy sibling, and the node pool upgrade deadlocks and retries until it is abandoned.

Reproduction (no cluster required)

Both images can be pulled anonymously from MCR:

# original repo — 1 layer, FROM scratch
mcr.microsoft.com/containernetworking/azure-cni:v1.7.16-0
  label image.base.ref.name = scratch
  dropgz: ELF 64-bit LSB executable, statically linked
  embedded azure-vnet: statically linked, no libc references

# /v2/ repo — 2 layers, Azure Linux base
mcr.microsoft.com/containernetworking/v2/azure-cni:v1.8.11
  dropgz: ELF 64-bit LSB executable, dynamically linked,
          interpreter /lib64/ld-linux-x86-64.so.2
  embedded azure-vnet: dynamically linked, libc.so.6,
          required symbols: GLIBC_2.2, 2.3, 2.4, 2.32, 2.34

Steps: pull the linux/amd64 manifest for each tag, extract the largest layer, take dropgz (/dropgz in the old image, /usr/bin/dropgz in the new one) and run file on it. The CNI binaries themselves are embedded in dropgz as plain gzip streams and can be extracted by scanning for the gzip magic bytes and decompressing; running file and strings | grep GLIBC_ on the resulting ELF files shows the same difference.

Note this is not a version bump: v1.7.17 exists in both repos — statically linked under containernetworking/azure-cni:v1.7.17-0, dynamically linked under containernetworking/v2/azure-cni:v1.7.17. The change came with the build pipeline, not the code.

Likely origin

This looks like a consequence of the Microsoft Go toolchain move rather than an intentional change:

• chore: upgrade Go 1.24 → 1.26.4 #4494 (Go 1.24 → 1.26.4) notes that MS Go 1.26 enables systemcrypto by default, which requires CGO, and that GOEXPERIMENT=ms_nocgo_opensslcrypto became mandatory for CGO_ENABLED=0 Linux builds.
• fix: use distroless/base for ipv6-hp-bpf (Go 1.26 compat) [backport release/v1.7] #4673 hit the same class of problem for ipv6-hp-bpf, and describes it precisely: "Go 1.26 (Microsoft fork) with GOEXPERIMENT=ms_nocgo_opensslcrypto dynamically loads these libraries at runtime via dlopen(), even with CGO_ENABLED=0." That was fixed by changing the container base image.

The base-image fix works for binaries that execute inside the container. It does not help binaries that dropgz installs onto the host, where the node's own libraries are what matter. The repo's cni.Dockerfile still declares FROM scratch and the build still sets CGO_ENABLED=0, so the shipped /v2/ image does not match the invariant the build config appears to intend.

Impact

Any AKS cluster with FIPS-enabled node pools on Ubuntu 20.04 that upgrades to Kubernetes 1.35 will have broken pod networking on every node that has not yet been replaced. This is the migration path currently recommended for the Ubuntu 20.04 retirement (Azure/AKS#4874), so it is likely to be hit more as that deadline approaches.

Expected behaviour

Either:

1. Build the CNI binaries that are installed onto the host without a runtime dependency on host libraries (as the original images did), or
2. Do not roll a CNI image onto nodes whose OS cannot execute the binary it installs — e.g. keep the DaemonSet image aligned with the node image rather than only with the control plane version.

At minimum, a documented minimum host glibc / OS version for the /v2/ images would let operators avoid the upgrade path that triggers this.

[rbtr]

Thanks for the report Aman Gupta (@w3aman), we're working on it. If you can file an azure support case I recommend you do that also