CNI Windows Add DelegatedNIC - scan for any left-over HNS Networks before trying to find master interface

[pjohnst5]

We saw an issue where a Windows CNI add call was killed mid-process (right after it invoked the CreateNetwork HNS call, but before that call returned to CNI), so HNS proceeded (as it was commanded) to create an HNS network for a delegated NIC

Before that CreateNetwork call finished in HNS, CNI also then got a DELETE call for the same pod (since containerd automatically invoked the CNI delete after the CNI add failed)

The CNI DELETE call basically did nothing, since CNI had not yet written anything to state since the ADD call was killed mid-process

After the CNI DELETE call finished, the background HNS createNetwork call completed, so then there was a "leaked" HNS Network (HNS Network exists on the VM, but CNI has no idea about it and did not save it to any state)

Sequence of events:

1. CNI ADD call starts
2. CNI issues CreateNetwork request

   azure-container-networking/network/network_windows.go

   Line 362 in 20d09ab

   hnsResponse, err = Hnsv2.CreateNetwork(hcnNetwork)

3. HNS begins creating the network for the delegated nic (1-1 mapping)
4. CNI ADD call is killed! The last log line we see is "Creating hcn network", nothing after that (not ADD command completed with error, nothing)
5. CNI DELETE call starts
6. CNI does not issue delete on HNS Network (because it was not able to save it to any state)
7. CNI DELETE call ends (basically did nothing)
8. HNS createNetwork call succeeds in the background (but CNI knows nothing about this network, because CNI was killed before it could save the network to any state)
9. There is now a drifting HNS Network on the VM, holding the MAC up
10. New CNI ADD call comes in for the same MAC
11. CNI cannot find the MAC, because it is now tied up to the old, leaked HNS network

One natural thought

Would be to say "Hey, let's store the HNS network that we are about to create, so that in case CNI add call gets killed mid-HNS network create, the CNI Delete will clean it up"

• Wrong, HNS does not have deterministic behavior if we issue an HNS network delete, on a HNS Network that is in progress of being created. It is then a race, as to which call finishes (could be still that HNS network delete call finishes first, and then the HNS network create call finishes last, so we still have a leaked HNS network)

Better solution

1. During a CNI Add (for windows, delegated nic), first, do an audit of all the HNS Networks on the VM
2. See if any HNS Networks don't belong to any pods (in which case, they have been leaked)
3. Delete the network
4. Then proceed with CNI ADD call as normal

5. azure-container-networking/cni/network/network.go

   Line 731 in 20d09ab

   masterIfName := plugin.findMasterInterface(opt)

[pjohnst5]

Perhaps #4290 could also fix this (if we widen the scope to non-ACI scenarios as well, AKS in this case)

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment to keep it open.

[pjohnst5]

I'm going to try and tackel this now

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment to keep it open.