Skip to content

fix: treat CNS DeleteEndpointState NotFound as successful no-op - #1

Closed
jackieluc wants to merge 2 commits into
masterfrom
jackieluc/ignore-delete-notfound-endpoint
Closed

jackieluc wants to merge 2 commits into
masterfrom
jackieluc/ignore-delete-notfound-endpoint

Conversation

@jackieluc

Copy link
Copy Markdown
Owner

Reason for Change:

In stateless SwiftV2 (Windows), a CNI DEL ends by asking CNS to remove the endpoint state via DeleteEndpointState. When multiple DELs race on the same pod sandbox (containerd retries + multi-NIC teardown), the first one removes the CNS state entry and the others get back NotFound. The CNS client treated that NotFound as a hard error, so network/manager.go DeleteState failed, cni/network/network.go wrapped it into a RetriableError("failed to delete state: ..."), and the DEL returned a failure to containerd, which then retried, feeding the loop.

A NotFound on delete means the state is already gone, so there is nothing to delete and the operation should succeed.

Concretely, this is the retry failed line observed in the field:

failed to delete state: Delete endpoint API returned with error for endpoint <id>:
[DeleteEndpointStateHandler] Failed to delete endpoint state for <id> with error:
[deleteEndpointState] endpoint <id>: endpoint state could not be found in the statefile

Approach:

  • cns/client/client.go DeleteEndpointState: return (nil, nil) when CNS responds with types.NotFound, before the generic non-zero return-code error path. The order matters because NotFound is non-zero and would otherwise be caught by the generic check. Also switched the generic guard from the magic != 0 to != types.Success to match the adjacent constant and the rest of the file.
  • network/manager.go DeleteState: the dead NotFound-as-success branch (which never fired because the old client discarded the typed response) is removed. The caller now just treats any returned error as a real failure and logs success otherwise.
  • cns/restserver/util.go restoreState: corrected a misleading log that claimed it was Removing endpoints.json on an endpoint-state restore read error when no removal actually happens. The file is intentionally kept so a transient/corrupt read does not wipe live pod state.

Issue Fixed:

N/A

Requirements:

Notes:

  • New TestDeleteEndpointState in cns/client/client_test.go covers success, the NotFound no-op (nil, nil, no error), a generic non-zero CNS return code, and an HTTP error.
  • Behavior is end-to-end idempotent now: this mirrors the existing IP-release path (removeEndpointState), which already tolerated a missing endpoint.
  • The CNS server still returns NotFound (accurate and observable); the client is where it is interpreted as a successful no-op.
  • Not addressed here (intentionally out of scope): GetEndpointHelper re-reads the endpoint state file on every call without a lock, which is a redundant read and a potential data race against concurrent writers. Worth a follow-up.

@jackieluc
jackieluc force-pushed the jackieluc/ignore-delete-notfound-endpoint branch from e2811dd to 2ca6edc Compare June 22, 2026 21:33
@jackieluc jackieluc closed this Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant