CNS corrupts IPAM and Endpoint state when CNI calls arrive out of order

[rbtr]

Summary:

In stateless-CNI mode (ManageEndpointState=true), CNS handles CNI ADD and DEL as two independent, unsynchronized sequences that mutate two different stores in opposite order:

Operation	1st mutation	2nd mutation
ADD (requestipconfigs)	in-memory IPAM (assign IP)	endpoint statefile (write owner)
DEL (releaseipconfigs)	endpoint statefile (delete owner)	in-memory IPAM (free IP)

The CNI runtime may issue a cleanup DEL while an ADD for the same infra container is still in flight. When that happens, and when there is a delay in processing the original ADD, the two sequences may interleave and cross:

DEL: remove endpoint entry   -> no entry yet, no-op
ADD: assign IP in memory     -> IP = Assigned
ADD: write endpoint entry    -> statefile now says "containerX owns 10.x.y.z"
DEL: release IP in memory    -> IP = Available

CNS state is now corrupt: its memory IPAM pool says the IP is free, while its persisted endpoint statefile says the IP is owned by a container that no longer exists. CNS can later hands that same IP to a different pod, because it appears free in-memory, and it then writes a second endpoint entry for it to the statefile.

On the next CNS restart, endpointStateToPodInfoByIP() walks the statefile, sees the same IP owned by two container IDs, and returns ErrDuplicateIP. This is a terminal, unrecoverable error: CNS never finishes reconciliation and can never safely serve IPAM. Restarting CNS does not help, because the corruption is on disk.

Fixes: CNS should use Tombstone Deletes - processing a DEL should prevent a future ADD for the same containerid. CNS ADD/DEL should be synchronized and exclusive for the same containerid.

Mitigation / recovery for an already-broken node

Replace the node. Restarting is CNS or the Node is not sufficient.

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
# then delete/recreate the node via the node pool

⚠️ Do not simply delete the whole azure-endpoints.json. In stateless-CNI mode that file is the source of truth CNS uses to rebuild IPAM mappings on restart. Deleting it makes CNS believe every IP is free while running pods still hold them, which can cause genuine duplicate assignment and IP leaks.

[rbtr]

Customer-facing behavior

Symptoms

1. azure-cns pod on the node is Running but never becomes Ready. Readiness probe returns HTTP 500 continuously:

   kubelet: "Probe failed" probeType="Readiness" pod="kube-system/azure-cns-xxxxx"
     containerName="cns-container" probeResult="failure" output="HTTP probe failed with statuscode: 500"
   

2. New pods on the node are stuck in ContainerCreating indefinitely. kubelet/containerd report:

   RunPodSandbox from runtime service failed: rpc error: code = Unknown desc =
   failed to setup network for sandbox "<id>": plugin type="cilium-cni" failed (add):
   http request failed: Post "http://localhost:10090/network/requestipconfigs":
   dial tcp 127.0.0.1:10090: connect: connection refused; failed to request IP address from CNS
   

   (connection refused because CNS never opened its IPAM listener — it is stuck in the failed initializer.)

3. Already-running pods keep working. Their datapath is already programmed; only new pod admission is broken. This makes the failure look like a scheduling/capacity problem at first.

4. The node does not recover on its own. Deleting/restarting the azure-cns pod, or rebooting the node, reproduces the same failure, because the corrupted state is persisted at:

   • Linux: /var/run/azure-cns/azure-endpoints.json
   • Windows: /k/azurecns/azure-endpoints.json

Confirming the issue

In azure-cns container logs:

[cns-rc] initializer failed during reconcile: provider failed to provide PodInfoByIP:
  100.64.61.38: duplicate IP detected in CNS initialization

or, on older lines:

failed to reconcile initial CNS state, attempt: 8 err: provider failed to provide PodInfoByIP:
  10.244.1.124: duplicate IP detected in CNS initialization

with the accompanying reconciler error:

level=error msg="Reconciler error" controller=nodenetworkconfig controllerGroup=acn.azure.com
  controllerKind=NodeNetworkConfig
  error="initializer failed during reconcile: provider failed to provide PodInfoByIP:
         <IP>: duplicate IP detected in CNS initialization"

This message repeats forever with controller-runtime backoff.

Impact

Per-node, but permanent until manually remediated. Any node that hits the race and then restarts CNS (upgrade, node reboot, CNS DaemonSet rollout, OOM, crash) becomes unable to start new pods.

---

[rbtr]

Root cause

ADD mutates memory first, disk second

cns/restserver/ipam.go#L74-L107

// record a pod requesting an IP
service.podsPendingIPAssignment.Push(podInfo.Key())
podIPInfo, err := requestIPConfigsHelper(service, ipconfigsRequest)   // (1) mutate in-memory IPAM
if err != nil { ... }

...

// Check if http rest service managed endpoint state is set
if service.Options[common.OptManageEndpointState] == true {
    err = service.updateEndpointState(ipconfigsRequest, podInfo, podIPInfo)  // (2) mutate statefile
    if err != nil { ... }
}

Note there is no lock spanning (1) and (2) — each helper takes and releases service.Lock() independently.

DEL mutates disk first, memory second

cns/restserver/ipam.go#L408-L428

// Check if http rest service managed endpoint state is set
if service.Options[common.OptManageEndpointState] == true {
    if err := service.removeEndpointState(podInfo); err != nil {   // (1) mutate statefile
        ...
    }
}

if err := service.releaseIPConfigs(podInfo); err != nil {          // (2) mutate in-memory IPAM
    ...
}

A DEL for an unknown container is a no-op

cns/restserver/ipam.go#L524-L541

func (service *HTTPRestService) removeEndpointState(podInfo cns.PodInfo) error {
	if service.EndpointStateStore == nil {
		return ErrStoreEmpty
	}
	service.Lock()
	defer service.Unlock()
	logger.Printf("[removeEndpointState] Removing endpoint state for infra container %s", podInfo.InfraContainerID())
	if _, ok := service.EndpointState[podInfo.InfraContainerID()]; ok {
		delete(service.EndpointState, podInfo.InfraContainerID())
		err := service.EndpointStateStore.Write(EndpointStoreKey, service.EndpointState)
		if err != nil {
			return fmt.Errorf("failed to write endpoint state to store: %w", err)
		}
	} else { // will not fail if no endpoint state for infra container id is found
		logger.Printf("[removeEndpointState] No endpoint state found for infra container %s", podInfo.InfraContainerID())
	}
	return nil
}

This is where the bug becomes unrecoverable. DEL-before-ADD is treated as "nothing to do" and leaves no record that a delete was ever requested. A late ADD therefore has no way to know its work has already been invalidated, and happily commits.

Startup duplicate assignment check is necessary, but makes this unrecoverable

cns/stateprovider/cns/podinfoprovider.go#L40-L70

func endpointStateToPodInfoByIP(state map[string]*restserver.EndpointInfo) (map[string]cns.PodInfo, error) {
	podInfoByIP := map[string]cns.PodInfo{}
	for containerID, endpointInfo := range state { // for each endpoint
		for _, ipinfo := range endpointInfo.IfnameToIPMap {
			if !ipinfo.NICType.IsInfraOrLegacy() {
				continue
			}
			for _, ipv4conf := range ipinfo.IPv4 {
				if _, ok := podInfoByIP[ipv4conf.IP.String()]; ok {
					return nil, errors.Wrap(cns.ErrDuplicateIP, ipv4conf.IP.String())   // <-- terminal
				}
				...

ErrDuplicateIP is defined at cns/NetworkContainerContract.go#L200-L201.

Failing closed here is correct — silently last-write-wins on a duplicate IP would be worse. The defect is that we let the duplicate get written in the first place, and we provide no repair path once it exists.

Contributing factor: non-atomic statefile writes

Two secondary issues amplify the window:

• updateEndpointState mutates service.EndpointState in place and then writes, and does so inside a per-IP loop (ipam.go#L335-L395). A failure partway through a dual-stack update can leave in-memory state ahead of disk, or partial endpoint ownership on disk.

• store/json.go#L115-L128 updates the in-memory kvs.data[key] cache before attempting flush(). A failed write therefore poisons the cache, and the failed value gets persisted by the next successful write of any key.

  func (kvs *jsonFileStore) Write(key string, value interface{}) error {
      kvs.Mutex.Lock()
      defer kvs.Mutex.Unlock()
  
      var raw json.RawMessage
      raw, err := json.Marshal(value)
      if err != nil {
          return err
      }
  
      kvs.data[key] = &raw   // <-- cache updated before the write is known to succeed
  
      return kvs.flush()
  }

A second path to the same failure

Serializing ADD and DEL removes the split brain, but one ordering still leaves bad state: if a DEL runs to completion before its ADD, the DEL is a pure no-op, and the ADD that lands afterwards assigns an IP and writes endpoint state for a sandbox that no longer exists. Nothing will ever delete that record — the runtime will never issue another DEL for a container ID it has already torn down.

That leaked record leaves the endpoint record in place (internalapi.go#L364-L378) — the miss is logged and the pod is continued:

for _, ip := range ips {
    if ncReq, ok := allSecIPsIdx[ip.String()]; ok {
        desiredIPs = append(desiredIPs, ip.String())
        ncIDs = append(ncIDs, ncReq.NetworkContainerid)
    } else {
        logger.Errorf("ip %s assigned to pod %+v but not found in any nc", ip, podIPs)
    }
}

if len(desiredIPs) == 0 {
    // this may happen for pods in the host network
    continue          // <-- stale endpoint record survives untouched
}

If the pool shrinks and later reintroduces that IP, IPAM offers it as Available while the stale record still claims it. The next pod to take that IP writes the second endpoint record.

This is why the fix needs both serialization and a durable delete intent.

---

[rbtr]

Sequence diagrams

Race

sequenceDiagram
    autonumber
    participant CNI as CNI chain - cilium-cni + azure-ipam
    participant ADD as CNS ADD handler
    participant DEL as CNS DEL handler
    participant MEM as CNS in-memory IPAM - PodIPConfigState
    participant DISK as endpoint statefile - EndpointState

    CNI->>ADD: POST /network/requestipconfigs for ctr X
    Note over CNI: ADD is slow, or the client times out at 15s.<br/>The CNI chain runs its cleanup DEL for the same sandbox.
    CNI->>DEL: POST /network/releaseipconfigs for ctr X

    DEL->>DISK: removeEndpointState ctr X
    DISK-->>DEL: no entry, silent no-op

    ADD->>MEM: AssignAvailableIPConfigs ctr X
    MEM-->>ADD: 100.64.61.38 = Assigned

    ADD->>DISK: updateEndpointState ctr X owns 100.64.61.38
    DISK-->>ADD: persisted

    DEL->>MEM: releaseIPConfigs ctr X
    MEM-->>DEL: 100.64.61.38 = Available

    rect rgb(255, 235, 235)
    Note over MEM,DISK: SPLIT BRAIN<br/>MEM says 100.64.61.38 is Available and free<br/>DISK says 100.64.61.38 is owned by stale ctr X
    end

Loading

Crash

sequenceDiagram
    autonumber
    participant CNI as CNI chain
    participant CNS as CNS running
    participant MEM as CNS in-memory IPAM
    participant DISK as endpoint statefile
    participant INIT as CNS startup reconcile

    CNI->>CNS: ADD for ctr Y, a different pod
    CNS->>MEM: 100.64.61.38 is Available, assign to ctr Y
    CNS->>DISK: updateEndpointState ctr Y owns 100.64.61.38

    rect rgb(255, 235, 235)
    Note over DISK: statefile now contains BOTH<br/>ctr X owns 100.64.61.38 - stale<br/>ctr Y owns 100.64.61.38 - live
    end

    CNS-->>CNS: restart from upgrade, reboot, daemonset roll or OOM

    INIT->>DISK: read EndpointState
    INIT->>INIT: endpointStateToPodInfoByIP
    INIT--xINIT: ErrDuplicateIP for 100.64.61.38

    rect rgb(255, 235, 235)
    Note over INIT: initializer fails during reconcile.<br/>CNS never serves IPAM.<br/>Every new pod sandbox on the node fails.<br/>Restart does NOT clear it, the state is on disk.
    end

Loading

Ordering

                 ADD path                          DEL path
                 --------                          --------
   step 1    in-memory IPAM: assign            statefile: delete owner
   step 2    statefile: write owner            in-memory IPAM: free

   Because the two paths touch the same two stores in OPPOSITE order,
   and neither holds a lock across both steps, an interleave leaves:

        statefile:  ctr X owns IP        (written by ADD step 2)
        IPAM:       IP is Available      (written by DEL step 2)

[rbtr]

Why it does not self-heal

• The corrupted mapping lives in azure-endpoints.json, which survives pod restart and node reboot.
• Startup is fail-closed by design (ErrDuplicateIP), and there is no repair/reconcile path that drops a stale endpoint entry whose IP is Available in the pool or whose container no longer exists.
• The reconciler retries forever with backoff, producing the repeating error and a permanently NotReady CNS.

---

Fixes

Make managed endpoint-state ADD/DEL a single serialized transaction per infra container, and record a durable delete intent (tombstone) so a late ADD cannot resurrect an allocation the runtime has already asked us to tear down.

The fix is open as a four-PR stack. They are stacked, so each targets the one below it and they must merge bottom-up. #4698 is the PR that actually fixes the crashloop; #4696 and #4697 are the storage-correctness prerequisites it relies on, and #4502 closes the second path described in 3.6.

Order	PR	Change
1 (base)	#4696 — fix: keep JSON store cache unchanged after failed writes	jsonFileStore.Write mutated its in-memory cache before flushing, so a rejected value stayed cached and was silently persisted by the next successful write to any key. Stage the value, flush, and restore the prior value for that key if the flush fails.
2	#4697 — fix: make CNS endpoint-state writes atomic	Copy-on-write endpoint state, published only after the store write succeeds, and one write per update instead of one per IP. Also fixes a real dual-stack bug: the first already-present IP caused an early return that silently dropped remaining new addresses (IPv4 present + IPv6 new meant IPv6 was never persisted). restoreState now fails closed on an unreadable endpoint store.
3	#4698 — fix: prevent CNS duplicate-IP corruption by serializing managed IPAM and endpoint state	The fix. Hold the service lock across the whole managed ADD and the whole managed DEL, so IPAM ownership and endpoint persistence can never interleave. Roll back only IPs newly allocated by a failing ADD, preserving idempotently-returned existing assignments. Reject cancelled queued ADDs before they mutate anything. Incidentally fixes an unsynchronized read of service.state.
4 (top)	#4502 — fix: prevent leaked CNS endpoint records from late ADDs after DELETE	Durable write-ahead delete intent keyed by infra container ID, stored alongside endpoint state. DEL persists the intent, then removes endpoint state, then releases the IP. ADD and endpoint PATCH are rejected while an unexpired intent exists. Startup loads both fail-closed, replays active intents, and prunes expired ones before the pod-info provider reads the store.

Endpoint PATCH has to be gated too: UpdateEndpointHelper creates a record when one is absent, so an ungated late PATCH would resurrect exactly the leaked record the intent exists to prevent.

Resulting semantics

• DEL is still idempotent to callers, but is no longer forgetful: an unknown-container DEL now leaves a durable record.
• ADD after DEL for the same infra container ID fails fast instead of committing a doomed allocation. The runtime retries with a new sandbox / new container ID, which is unaffected.
• ADD is all-or-nothing: if the endpoint write fails, the newly allocated IPs are released, so we never leave a half-committed allocation.
• Intents are TTL-pruned opportunistically (no background goroutine). The TTL is 10 minutes — it only has to outlive a CNI request that was already in flight when the DELETE arrived, plus a CNS restart, and it is deliberately far shorter than a pod lifetime because an intent that outlives its racing request can only reject work that should have succeeded.
• An intent keeps the timestamp of the first delete and is not refreshed. Refreshing on every DELETE would be self-renewing: a rejected ADD makes the runtime retry DELETE, which would extend the intent indefinitely and permanently block that container.
• Cost is linear in the number of outstanding intents, which is the other reason the TTL is minutes rather than hours. Measured DELETE cost on a 250-endpoint node: 0.44 ms with no intents, 0.69 ms at 250 outstanding, 1.41 ms at 1 000, 99.7 ms at 86 400. A node cycling its entire 250-pod complement every 10 minutes sits at ~0.69 ms, immaterial next to the multi-second ADDs that trigger this bug class.