Conversation
jackieluc
force-pushed
the
jackieluc/ignore-delete-notfound-endpoint
branch
from
June 22, 2026 21:33
e2811dd to
2ca6edc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for Change:
In stateless SwiftV2 (Windows), a CNI DEL ends by asking CNS to remove the endpoint state via
DeleteEndpointState. When multiple DELs race on the same pod sandbox (containerd retries + multi-NIC teardown), the first one removes the CNS state entry and the others get backNotFound. The CNS client treated thatNotFoundas a hard error, sonetwork/manager.go DeleteStatefailed,cni/network/network.gowrapped it into aRetriableError("failed to delete state: ..."), and the DEL returned a failure to containerd, which then retried, feeding the loop.A
NotFoundon delete means the state is already gone, so there is nothing to delete and the operation should succeed.Concretely, this is the
retry failedline observed in the field:Approach:
cns/client/client.goDeleteEndpointState: return(nil, nil)when CNS responds withtypes.NotFound, before the generic non-zero return-code error path. The order matters becauseNotFoundis non-zero and would otherwise be caught by the generic check. Also switched the generic guard from the magic!= 0to!= types.Successto match the adjacent constant and the rest of the file.network/manager.goDeleteState: the deadNotFound-as-success branch (which never fired because the old client discarded the typed response) is removed. The caller now just treats any returned error as a real failure and logs success otherwise.cns/restserver/util.gorestoreState: corrected a misleading log that claimed it wasRemoving endpoints.jsonon an endpoint-state restore read error when no removal actually happens. The file is intentionally kept so a transient/corrupt read does not wipe live pod state.Issue Fixed:
N/A
Requirements:
Notes:
TestDeleteEndpointStateincns/client/client_test.gocovers success, theNotFoundno-op (nil, nil, no error), a generic non-zero CNS return code, and an HTTP error.removeEndpointState), which already tolerated a missing endpoint.NotFound(accurate and observable); the client is where it is interpreted as a successful no-op.GetEndpointHelperre-reads the endpoint state file on every call without a lock, which is a redundant read and a potential data race against concurrent writers. Worth a follow-up.