fix: [NPM] policy translation and resource-bound hardening - #4831
fix: [NPM] policy translation and resource-bound hardening#4831Isaiah Raya (rayaisaiah) wants to merge 23 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
The current error/telemetry logging behavior can produce duplicated and potentially spammy error signals (especially for repeated translation failures and over-limit pod labels), and should be tightened before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Robustness and correctness hardening for Azure NPM v2 (Linux) policy translation and related resource-bound controls, aiming to preserve behavior for well-formed NetworkPolicies while closing edge-case allow-bypasses, silent translation failures, and potential resource-exhaustion paths.
Changes:
- Fix
namespaceSelectorsemantics (NotIn conjunction, negation-only scoping to namespaces) and bound selector flattening expansion. - Canonicalize
ipBlockCIDRs and ensure translation failures are surfaced/requeued rather than being reported as successful no-ops. - Add resource bounds for per-pod label ipset creation and harden the NPM HTTP API server (timeouts, header limit, concurrency limits), plus disable debug/pprof routes by default in manifests.
File summaries
| File | Description |
|---|---|
| npm/util/util.go | Adds NormalizeCIDR and updates IPv4/CIDR validation to accept non-canonical spellings safely. |
| npm/util/util_test.go | Adds unit coverage for IsIPV4 and CIDR canonicalization behavior. |
| npm/pkg/dataplane/ipsets/ipsetmanager.go | Requires CIDR ipset members to be canonical before programming kernel sets. |
| npm/pkg/controlplane/translation/translatePolicy.go | Canonicalizes ipBlock CIDRs/Excepts and adds new translation errors for selector handling limits. |
| npm/pkg/controlplane/translation/translatePolicy_test.go | Adds regressions for NotIn semantics, negation-only scoping, and non-canonical all-addresses CIDRs. |
| npm/pkg/controlplane/translation/parseSelector.go | Fixes NotIn flattening semantics, adds expansion limit, and scopes negation-only namespace selectors. |
| npm/pkg/controlplane/translation/parseSelector_test.go | Adds unit tests for NotIn conjunction, unsupported operators, empty values, and expansion limit propagation. |
| npm/pkg/controlplane/controllers/v2/podController.go | Caps per-pod label processing to bound label-derived ipset creation. |
| npm/pkg/controlplane/controllers/v2/podController_test.go | Tests label cap behavior and cache/dataplane agreement. |
| npm/pkg/controlplane/controllers/v2/networkPolicyController.go | Surfaces translation failures (requeue) while keeping deliberate unsupported-feature suppression. |
| npm/pkg/controlplane/controllers/v2/networkPolicyController_test.go | Adds tests ensuring non-canonical CIDR policies apply and translation failures are surfaced. |
| npm/http/server/server.go | Adds HTTP server deadlines/header limit, connection ceiling, and cache-handler concurrency shedding. |
| npm/http/server/server_test.go | Adds tests for cache-handler concurrency limit and guards server timeout constants. |
| npm/deploy/npm/azure-npm.yaml | Disables pprof/debug API by default in deployment config. |
| npm/deploy/manifests/daemon/azure-npm.yaml | Disables pprof/debug API by default in daemon manifest config. |
| npm/deploy/manifests/controller/azure-npm.yaml | Disables pprof/debug API by default in controller manifest config. |
| npm/deploy/manifests/common/npm-configmap.yaml | Disables pprof/debug API by default in common configmap. |
| npm/deploy/kustomize/base/configmap.yaml | Disables pprof/debug API by default in kustomize base configmap. |
| npm/azure-npm.yaml | Disables pprof/debug API by default in top-level manifest. |
Review details
- Files reviewed: 19/19 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
81324a1 to
37fefb7
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
It makes several security- and correctness-sensitive changes across policy translation and host-network API behavior that warrant final human review despite strong test coverage.
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 4
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
The new ACL budget guard is intended as a hard resource bound, but the current checkACLBudget condition allows exceeding the declared limit by one ACL (and tests codify that), undermining the strictness/early-stop goal of the bound.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
npm/pkg/controlplane/translation/translatePolicy.go:41
- ErrEmptyMatchExpressionValues' text uses inconsistent casing ("notIn") and refers to operators by name but not in a consistent, log-friendly way. Since this string will surface in logs/events, prefer a fully-lowercase, consistently formatted message (Go convention is to start error strings with lowercase).
- Files reviewed: 18/18 changed files
- Comments generated: 1
- Review effort level: Lite
…njunction A namespaceSelector matchExpression using NotIn with more than one value is a single set-membership requirement: key NotIn [a, b] means key != a AND key != b. The v2 selector compiler flattened multi-value NotIn the same way it flattens multi-value In, emitting one selector per value. Each flattened selector becomes an independent allow decision, and allow decisions are additive (OR), so a namespace carrying one of the values could still be matched by the decision negating a different value. That does not match LabelSelectorAsSelector semantics for multi-value NotIn. Handle the two operators separately in flattenNameSpaceSelector: In still fans out into one selector per value, while NotIn keeps every value as its own single-value NotIn requirement within the same selector. When a selector mixes the two, each NotIn exclusion is carried conjunctively into every In branch. Also fail closed on unsupported operators and empty In/NotIn value lists rather than dropping the requirement, which would widen the selector. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…amespaces A namespaceSelector only ever selects namespaces, so every peer it matches must be a cluster address. A negative requirement (NotIn / DoesNotExist) renders as a negated set match, which is satisfied by every address that is not in that set, including addresses that are not cluster pods at all. When a namespaceSelector produced no positive set, the negations alone were the whole match, so the rule admitted non-cluster peers. On egress that let a selected pod reach arbitrary external hosts even though the policy named no ipBlock and no allow-all peer. Intersect with the all-namespaces set in parseNSSelector when the parsed selectors contain no positive match, mirroring allowAllInternal. Selectors that already carry a positive requirement (matchLabels, In, Exists) are unchanged, since that requirement already scopes the match to namespaces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flattening a namespaceSelector's multi-value In requirements produces the Cartesian product of their values, and the code had no ceiling on the result. The count is the product of the value counts, so it grows exponentially with the number of such requirements: 19 two-value requirements in one small, valid policy expand to 2^19 selectors. Each one is deep-copied and later becomes its own IPSet and ACL, so a single policy object could exhaust the memory of the NPM DaemonSet on every node and take policy programming down cluster-wide. Compute the product before any allocation and reject the selector once it would exceed maxFlattenedNSSelectors. The check divides instead of multiplying so it cannot overflow, and it runs per requirement, so a single very wide requirement is rejected on the first iteration too. The cap is far above any workable policy since a selector fanning out that wide would already be unusable as rules. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lation as success An ipBlock CIDR that named the all-addresses block with host bits set, such as 10.0.0.0/0, was rejected before the value was canonicalized, even though it denotes exactly the same addresses as 0.0.0.0/0. That failed the translation of the whole policy, and the v2 controller converted the failure into a successful no-op, so neither the peer rule nor the default drop the policy implies was installed. The policy looked applied while its selected pods were left with no rules at all and nothing signalled the failure. Canonicalize instead of rejecting. NormalizeCIDR clears the host bits, IsIPV4 validates through it, and the ipBlock translation compares and emits the canonical form, so a non-canonical spelling takes the same path as the canonical one. Except CIDRs are canonicalized too, so they de-duplicate correctly and are recognized by the all-addresses split rather than naming the same block twice with opposite meanings. Kernel ipset members must still be canonical, so the dataplane member check now requires that explicitly rather than relying on the old textual rejection. In the controller, a translation failure is now returned and recorded instead of reported as success, so it is visible and the key is requeued. Deliberate datapath limitations (the Windows unsupported features and NPM Lite's CIDR-only peers) cannot resolve on retry and stay suppressed with a warning as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…by default The NPM HTTP API listens on the host network of a privileged process, so any pod on the node can reach it, and its server was created with only an address and a handler: no read, write or idle deadline, no header bound, and no limit on concurrent connections. The cache route additionally serializes the entire policy cache into memory on every request while holding the cache lock. A client that opens connections and reads its responses a byte at a time could therefore hold an unbounded number of full cache copies alive until the process was OOM killed. Give the server deadlines, a header bound, and a connection ceiling, and admit only a few cache encodings at a time, shedding the rest, so neither the number of clients nor the speed at which they read decides how much memory NPM allocates. The deadlines are generous enough for a Prometheus scrape of this endpoint. Also stop enabling the debug and pprof routes by default in the deployment manifests. Prometheus metrics are unchanged; the debug routes are now opt-in for the clusters that need them, which is where the cache serialization and profiling endpoints are reachable from. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…its rendered rule The existing coverage for negation-only namespace peers used a single-value NotIn. Extend it to every operator that can produce a negation-only selector, DoesNotExist included, in both directions, and assert the shape of the resulting decision: exactly one allow ACL, the exclusion still negated, and exactly one positive set, the all-namespaces anchor. Also pin how that decision reaches the kernel. A negated set match is satisfied by every address absent from the set, including addresses that are not pods, so a rule whose peer list is only negations matches non-pod traffic. The new emission test asserts the anchor renders as a positive match-set alongside the negation, and that a namespace peer never renders as a lone negated match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ation on the Linux path Two follow-ups from review of the earlier commits. Bounding the flattened namespaceSelector count is not sufficient on its own. Every flattened branch is emitted once per port in the rule, and that product is summed across peers and rules, so a policy whose selector expansion sits comfortably under the selector limit can still multiply itself out by listing many ports: 512 branches against 512 ports is 262144 rules. Add a per-policy ACL ceiling, checked before each peer is expanded so translation stops early rather than after materializing the product, and again before the policy is returned. The earlier CIDR fix worked by broadening the shared IsIPV4 classifier, but that classifier is also consumed by the Windows and NPM Lite direct-rule paths, which write the CIDR into the ACL rather than into an ipset. Broadening it therefore changed behavior in components that are out of scope here. IsIPV4, deDuplicateExcept and the dataplane ipset member check are left exactly as they were, so those paths are byte-identical to before. The Linux ipBlock path validates through NormalizeCIDR instead, which canonicalizes first, and canonicalizes its except CIDRs through a helper used only by that path. NormalizeCIDR itself is purely additive. Finally, default the debug and profiling routes to off in DefaultConfig. The deployment manifests already disable them, but a missing or unreadable config file falls back to this struct, so the fallback must not be the configuration that exposes unauthenticated routes on the host network. Also admit a single cache encoding at a time rather than two, keeping peak memory to one copy of the cache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nces them NPM creates two ipsets per distinct pod label, on every node, and Kubernetes places no limit on how many labels a pod may carry. Applying every set to the kernel unconditionally meant one pod with tens of thousands of labels pushed thousands of sets into every node's kernel, pinned agent CPU and memory until agents were OOM killed, and delayed policy programming in unrelated namespaces while they recovered. Default to on-demand, so a set reaches the kernel only once a network policy references it. Incidental pod labels, which are the attacker-influenced input, never become kernel state at all. This removes the amplification without weakening enforcement, and specifically without the failure modes a creation limit has. The sets are still tracked and pods still join them, so a set is already populated by the time a policy references it. A limit instead has to refuse creating a set, which either blocks a policy from installing or leaves it referencing an empty set, and in both cases a pod escapes the policy that selects it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… tests meaningful Three review follow-ups. The per-policy ACL budget was still being checked in directPeerAndPortAllowRule, which is reached only under npmLite on Windows. That is an out-of-scope path, and the check was also placed before the port loop, so it could not have bounded the ports x excepts ACLs that loop appends anyway. Remove it; the budget remains on the shared peer/port path and at the end of translation. TestServerTimeoutsAreSet asserted that the timeout constants were non-zero, so removing an assignment from the server itself would not have failed it. Extract newServer and assert the server that is actually served. TestNegationOnlyPeerRendersAnchor built its own SetInfo list, used Ingress for both subtests, and asserted the match-set count was not one, which would also pass if the anchor were missing entirely. Use the real direction per subtest and assert exact counts for the anchor, the negation, and the total. Both tests now fail if their fix is reverted, which was verified by reverting each. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rs once Address review feedback on the translation and HTTP server changes. The ACL budget was checked once on entry to peerAndPortRule, before its port loop. A single peer emits one ACL per port, so one peer listing many ports could materialize every ACL and only be caught by the check at the end of translation. The budget is now checked before each port as well, so translation stops at the limit instead of after building the full product. flattenNameSpaceSelector and checkACLBudget each logged an error and also returned one, and their callers already log and record the returned error at the workqueue boundary. Both now return the error wrapped with the selector or policy context and leave recording to the caller, which is the single place that reports a translation failure. The sentinel errors are unchanged, so errors.Is comparisons still hold. The blank import of net/http/pprof is dropped; the package is already imported by name for the route handlers, which runs the same init. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Findings reported by golangci-lint on the lines this branch adds, run the way CI runs it (--new-from-rev against master). noctx: the HTTP server listener is created through net.ListenConfig, and the server tests build their requests with httptest.NewRequestWithContext. goconst: repeated literals in the added tests are named constants. The label keys, the team value and the direction names move into the existing const block in translatePolicy_test.go, the added namespace literals in parseSelector_test.go use the existing defaultNS, and the CIDRs shared between the IsIPV4 and NormalizeCIDR cases get their own constants. No behavior change; npm builds, vets and tests as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cde4271 to
7352af9
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core policy translation/enforcement semantics and introduces multiple new resource-bounding behaviors, so a final human review is warranted despite strong unit test coverage.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
npm/http/server/server.go:90
- The PathPrefix("/debug/") route is registered before the specific /debug/pprof/* routes, so it will match first and the explicit pprof handlers below become unreachable in gorilla/mux. This is confusing to maintain (it looks like those handlers matter, but they never run). Register the more specific pprof routes first, then the /debug/ prefix fallback.
- Files reviewed: 21/21 changed files
- Comments generated: 0 new
- Review effort level: Lite
NPM creates two ipsets per distinct pod label, and ipset_counts carries the set name as a label, so the number of series it reports followed workload labels rather than anything an operator controls. One pod carrying tens of thousands of labels added that many series on every node, which both retained them in the agent and inflated the response built for each scrape. Deferring kernel materialization does not help here: the sets are still tracked, which is what keeps enforcement correct, so the series were still created. The per-set breakdown now stops growing at a bound far above what a cluster's namespaces, policies and workloads produce. The aggregate counters are untouched and stay exact, and nothing NPM does reads the breakdown, so this only limits reported detail; an operator can tell it is incomplete by comparing the reported series against num_ipsets. Measured with one pod carrying 34,000 labels: agent memory for those labels drops from 88.8 MB to 14.6 MB, and the metrics response from 6.42 MB to 1.87 MB, which no longer varies with the labels a workload chooses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ad0cbe4 to
31faeea
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core NetworkPolicy translation semantics and host-network API hardening, which are security-sensitive and warrant final human review despite strong test coverage.
Review details
- Files reviewed: 21/21 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
One added unit test mutates the global http.DefaultServeMux without isolation, which can make the test suite order-dependent/flaky and should be made hermetic before approval.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
npm/http/server/server_test.go:218
- This test mutates the global http.DefaultServeMux via HandleFunc, which can leak routes into other tests in this package (and makes the test order-dependent). Prefer using a fresh *http.ServeMux instance for the test and mounting that, so the test stays hermetic.
npm/pkg/controlplane/translation/translatePolicy.go:856 - checkACLBudget’s docstring claims the per-append checks ensure translation paths “never take the policy past the ceiling”, but ingressPolicy/egressPolicy still append the default drop ACL without a budget check. That means a policy can hit the ceiling during allow-rule emission and only exceed it when adding the mandatory drop, then fail only at the end via checkACLTotal. Consider either updating the comment, or adding a budget check immediately before appending the default drop so the translator fails at the right spot.
- Files reviewed: 21/21 changed files
- Comments generated: 0 new
- Review effort level: Lite
The router matched a /debug/ prefix before the individually named pprof routes, so those named routes never ran and every handler on the default mux was reachable under /debug/, not just the profiles. The default mux is now mounted at the pprof prefix instead, which is where net/http/pprof registers, so the profiles are served, the subpaths that naming each handler missed (/debug/pprof/goroutine and the rest) are served too, and nothing else later registered on the default mux is exposed here. The import returns to a blank one because the handlers are reached through the mux rather than by name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4e49c6a to
4b767e9
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate findings remain in selector expansion, ACL budgeting, IPSet metrics, and HTTP lifecycle handling.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (4)
npm/metrics/ipsets.go:166
- This implementation deliberately stops registering
ipset_countsseries after 20,000, but the PR's known-limitations text says every label still registers a metric series and that deferring per-set metric registration is follow-up work. Please reconcile the stated limitation and operator expectations with this new incomplete-metrics behavior.
if len(inventorySeries) >= maxIPSetInventorySeries {
return false
npm/metrics/ipsets.go:151
- Once
inventorySeriesreaches the cap,updateIPSetInventoryskips this set, butGetNumEntriesForIPSetstill callsgetVecValue, whoseGaugeVec.Withcreates a series on lookup. Querying any unreported live set therefore bypasses the cap and creates a series thatinventorySeriesdoes not track, so the cardinality bound is not enforced for this exported accessor. Return the userspace count for names that are not ininventorySeries(or use a non-creating lookup) instead.
func updateIPSetInventory(setName string) {
if !canReportIPSetInventory(setName) {
return
npm/pkg/controlplane/translation/translatePolicy.go:869
- Because this is only a post-build backstop, a policy with many empty Ingress/Egress rules still enters
translateRule's zero-peer/zero-port branch, appends one ACL per rule, and is rejected only after the over-limit slice has been materialized. That leaves a valid Kubernetes object able to bypass the new allocation bound; check the budget before that append (or count this path up front) so oversized policies stop at the ceiling.
func checkACLTotal(npmNetPol *policies.NPMNetworkPolicy) error {
if len(npmNetPol.ACLs) > maxACLsPerPolicy {
return tooManyACLs(npmNetPol)
npm/pkg/controlplane/translation/translatePolicy.go:295
- This comment incorrectly says the ipset path is Linux-only.
translateRuleusesipBlockRulefor Windows v2 whenever NPM Lite is disabled, so the new CIDR normalization also affects that path; only the NPM Lite direct-rule path is unchanged. Please describe the actual ipset/direct-rule boundary so future platform changes are not based on the wrong scope.
// implies is installed and the selected pods are left with no rules at all. This is the
// ipset path, which is Linux only; the Windows direct-rule path is unchanged.
- Files reviewed: 21/21 changed files
- Comments generated: 2
- Review effort level: Lite
| srv := newServer(rs.listeningAddress, rs.router) | ||
|
|
||
| var lc net.ListenConfig | ||
| listener, err := lc.Listen(context.Background(), "tcp", rs.listeningAddress) |
Compiling a multi-value NotIn as one conjunction keeps it out of both existing bounds: it stays a single selector, so the flattened-selector count does not see it, and it produces a single rule, so the per-policy rule budget does not either. Every value still becomes its own ipset and its own condition on that rule, so a valid policy listing a long NotIn could still create thousands of sets and one enormous rule. The matches a selector expands into are now counted before anything is allocated, against the same ceiling used for the selector count. A NotIn contributes one per value, In one per branch, and Exists and DoesNotExist one each. Selectors on the bound still translate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate findings remain in selector expansion, ACL budgeting, CIDR exception handling, and HTTP server lifecycle.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
npm/http/server/server.go:107
- The new listener is created with
context.Background()even thoughNPMRestServerListenAndServeis launched as a long-lived goroutine from the startup path. That context can never be cancelled, and this function has no caller-owned shutdown path for the listener/server, so a shutdown cannot stop or gracefully close the HTTP service. Thread a lifecycle context into this function and shut the server down from its cancellation.
var lc net.ListenConfig
listener, err := lc.Listen(context.Background(), "tcp", rs.listeningAddress)
if err != nil {
klog.Errorf("Failed to start NPM HTTP Server with error: %+v", err)
return
npm/pkg/controlplane/translation/translatePolicy.go:860
- This check only counts ACLs emitted so far, but ingressPolicy and egressPolicy unconditionally append a default-drop ACL after translateRule returns. A port-only rule with exactly maxACLsPerPolicy ports therefore emits 2000 allow ACLs and then a 2001st drop before checkACLTotal rejects the policy. Reserve the mandatory drop or check that append as part of the bound so the early paths cannot exceed the stated ceiling.
// checkACLBudget reports whether there is room for another ACL. It is checked before a peer
// is expanded, before each of that peer's ports, and before each port of a port-only rule, so
// those paths never take the policy past the ceiling.
npm/pkg/controlplane/translation/translatePolicy.go:207
- Canonicalizing an exception can turn a non-canonical
/0such as10.0.0.1/0into0.0.0.0/0. The/0parent is later split into two/1members, butsplitCIDRSethas no entry for the full/0, so this value is emitted as0.0.0.0/0 nomatch; the surrounding code explicitly notes that ipset cannot add the/0block, so restoring the translated set fails instead of representing the empty result. Handle an exception that canonicalizes to the full parent before it reaches the member-building loop.
canonical, ok := util.NormalizeCIDR(except)
if !ok {
return nil, fmt.Errorf("except %q: %w", except, ErrUnsupportedIPAddress)
}
if _, exist := exceptsSet[canonical]; !exist {
- Files reviewed: 21/21 changed files
- Comments generated: 2
- Review effort level: Lite
| deDupExcepts, err := canonicalizeExcepts(ipBlockRule.Except) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| lenOfDeDupExcepts := len(deDupExcepts) |
The match bound sat below the shortcut that returns a selector carrying no match expressions, so a selector made only of matchLabels skipped it even though each of those labels becomes its own ipset and its own condition on the rule. The bound is now applied before that shortcut. Also pins the ipBlock member packing now that except CIDRs are canonicalized first. Canonicalizing can turn an except into one of the two halves that 0.0.0.0/0 is split into, which takes the branch that rewrites an existing member and shortens the list, so the test covers a split-half except first, last, between two ordinary excepts, both halves at once, and a non-canonical all-addresses block, and asserts every other except still survives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Four moderate findings remain unresolved in selector bounds, ACL ceiling enforcement, CIDR exception preservation, and metric reporting.
Review details
Suppressed comments (4)
Previously missed (1) — in code that hasn't changed since the last review.
npm/metrics/ipsets.go:167
- Once 20,000 names have claimed a slot, this silently skips
ipsetInventory.Witheven thoughipsetInventoryMapandnum_ipsetscontinue tracking the live set.GetNumEntriesForIPSettherefore returns zero andipset_countsomits real sets, so dashboards or callers lose per-set data; this also contradicts the PR description's claim that Prometheus metrics are unchanged. If this cap is required, expose an explicit dropped-series signal and update the metric contract/consumers, or preserve the existing per-set behavior.
npm/pkg/controlplane/translation/parseSelector.go:102
- This check does not bound the total generated matches. A selector with 991
MatchLabelsplus nine two-valueInrequirements passes withmatches == 1000but produces 512 branches, each with about 1000 set matches; the translation loop appends each branch's IP sets before the ACL check, so roughly 512,000 translated IPSet objects plus 512,000 SetInfo entries can be materialized under both advertised limits. It also undercounts the implicit all-namespaces anchor added byparseNSSelectorfor negation-only selectors, so 1000NotInvalues passes here and produces 1001 matches. Bound the aggregate branch×match count and include generated anchors beforenameSpaceSelectorallocation.
matches := len(nsSelector.MatchLabels)
for _, req := range nsSelector.MatchExpressions {
if req.Operator == metav1.LabelSelectorOpNotIn {
// each excluded value is carried as its own negated match
matches += len(req.Values)
continue
}
// In contributes one match per branch; Exists and DoesNotExist one each
matches++
}
if matches > maxSelectorMatches {
npm/pkg/controlplane/translation/translatePolicy.go:874
- The final check runs after the required default-drop ACL is appended. If rule translation has already produced exactly
maxACLsPerPolicyallow ACLs, the drop is materialized as ACL 2001 and only then rejected, so the hard ceiling is exceeded and an oversized policy still allocates one extra ACL. Reserve a slot for the required drop or guard those append sites before adding it.
func checkACLTotal(npmNetPol *policies.NPMNetworkPolicy) error {
if len(npmNetPol.ACLs) > maxACLsPerPolicy {
return tooManyACLs(npmNetPol)
npm/pkg/controlplane/translation/translatePolicy.go:240
- Canonicalizing exceptions can turn an ordinary entry followed by split-half entries into the special shrink path, but the existing packing loop indexes ordinary entries by the original position and then truncates
members. For input[200.0.0.0/8, 250.0.0.0/1, 10.0.0.0/1], the latter two normalize to the split halves and the first exception is overwritten/lost, contrary to the newTestIPBlockExceptCanonicalizationKeepsEveryExceptexpectation. Build the output without positional shrinking (or compact it after applying split replacements) so every canonical exception is retained.
deDupExcepts, err := canonicalizeExcepts(ipBlockRule.Except)
if err != nil {
return nil, err
- Files reviewed: 21/21 changed files
- Comments generated: 0 new
- Review effort level: Lite
…ling exact Two bounds were checkable individually but not together. A multi-value In repeats the whole selector once per value, so a selector can sit under both the match bound and the branch bound while their product is enormous: 991 matchLabels with nine two-value In requirements is 1000 matches across 512 branches, and the translator materializes an ipset and a set reference for each before the policy's rule budget is consulted. The product is now bounded too. The branch count is still checked on its own terms first, so a selector that merely fans out too far reports that rather than the total. The match count also missed the all-namespaces anchor that a selector matching only negatively is given, so such a selector could produce one match more than the bound allowed. Separately, the default drop a policy implies was appended before the check at the end of translation, so a policy that filled the budget with allow rules landed one or two ACLs past the ceiling before being refused. The per-append guard now holds back a slot per direction, so the ceiling is not exceeded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate findings affect fail-closed policy behavior and HTTP lifecycle/access controls.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
npm/http/server/server.go:104
- This new listener is created with
context.Background(), while all NPM start paths launchNPMRestServerListenAndServeas a detached goroutine and retain no server handle for shutdown. That leaves the HTTP listener and active connections disconnected from the process lifecycle during coordinated stop/restart. Pass a caller-owned cancellation signal into this function and shut down the server/listener from that lifecycle instead of creating a root context here.
srv := newServer(rs.listeningAddress, rs.router)
var lc net.ListenConfig
listener, err := lc.Listen(context.Background(), "tcp", rs.listeningAddress)
npm/http/server/server.go:135
IsLoopbackonly proves that the peer used the node's loopback address; it does not distinguish the node process from ahostNetwork: truepod, which shares the node network namespace and can connect to127.0.0.1. Such a pod can therefore still reach the policy cache and pprof routes despite the stated pod isolation. Use a Unix-socket/permission boundary or an explicit authentication/authorization mechanism if these endpoints must be node-process-only.
if ip := net.ParseIP(host); ip == nil || !ip.IsLoopback() {
http.Error(w, "forbidden", http.StatusForbidden)
return
npm/pkg/controlplane/translation/translatePolicy.go:869
- The pre-append guard reserves two drop ACLs on every translation path (
len(ACLs) >= maxACLsPerPolicy-2), even when the policy has only one directional default drop left. A one-direction Linux policy with 1,999 allow ACLs plus its one drop is exactly 2,000 ACLs, but it is rejected while it has only 1,998 ACLs; consequently the claimed exact-bound behavior is never reachable through the guarded path (the boundary test only produces 1,999). Track the number of drops still pending, or otherwise make the reservation reflect the policy's actual directions.
if len(npmNetPol.ACLs) >= maxACLsPerPolicy-reservedDropACLs {
return tooManyACLs(npmNetPol)
- Files reviewed: 21/21 changed files
- Comments generated: 1
- Review effort level: Lite
| return metrics.NoOp, fmt.Errorf("translating network policy %s/%s: %w", | ||
| netPolObj.Namespace, netPolObj.Name, err) |
Summary
Six independent correctness and robustness fixes in the NPM v2 Linux policy path. Each is
self-contained and has its own commit, so they can be reviewed one at a time. Nothing here
changes how a well-formed, ordinary NetworkPolicy is programmed.
This targets
master. Three related naming-uniqueness changes (#4574, #4587, #4593) wereforward-ported to
masterearlier in #4611; this is the follow-on set.Changes
Listed in commit order.
1. Compile multi-value
namespaceSelectorNotInas a single conjunctionkey NotIn [a, b]is one requirement meaningkey != a AND key != b. The selector compilerflattened it the way it flattens multi-value
In, emitting one selector per value. Each becomesan independent allow decision, and allow decisions are additive (OR), so a namespace carrying one
of the values was still matched by the decision negating a different value.
flattenNameSpaceSelectornow handles the operators separately:Instill fans out into oneselector per value,
NotInkeeps every value as its own single-value requirement inside the sameselector, and a selector mixing the two carries each
NotInexclusion conjunctively into everyInbranch. Unsupported operators and emptyIn/NotInvalue lists now fail closed rather thanbeing dropped, which would widen the selector.
2. Scope negation-only
namespaceSelectormatches to cluster namespacesA
namespaceSelectorselects pods in namespaces, and NPM's namespace sets hold pod IPs. Anegative requirement (
NotIn/DoesNotExist) renders as a negated set match, which is satisfiedby every address absent from that set — including addresses that are not pods at all. When the
selector produced no positive set, the negations alone were the whole match, so the rule admitted
non-cluster peers: on egress a selected pod could reach arbitrary external hosts, and on a
flat-network cluster a routable non-pod host could reach a protected pod.
parseNSSelectornow intersects with the all-namespaces set when the parsed selectors contain nopositive match, mirroring
allowAllInternal. Selectors already carrying a positive requirement(
matchLabels,In,Exists) are unchanged.3. Bound
namespaceSelectorflattening, and the rules a policy generatesFlattening multi-value
Inrequirements produced the Cartesian product with no ceiling; 19two-value requirements in one small valid policy expand to 2^19 selectors, each deep-copied and
turned into its own IPSet and ACL.
Two bounds, because the selector count alone is not the cost:
maxFlattenedNSSelectors, computed before any allocation. The check divides rather thanmultiplies so it cannot overflow, and runs per requirement so one very wide requirement is
caught on the first iteration.
maxACLsPerPolicy. Every flattened branch is emitted once per port, summed across peers andrules, so 512 selectors against 512 ports is 262144 ACLs — comfortably under the selector cap.
It is checked before a peer expands, before each of that peer's ports, and before each port of a
rule that lists ports and no peers, so those paths never materialize more than the ceiling. It is
checked once more before the policy is returned. That final check is not conditioned on the
datapath, so an oversized policy is refused however it was built; what stays out of scope is
instrumenting the Windows/NPM Lite direct-rule path with the early checks, so a policy built
there is refused at the end rather than partway through.
maxSelectorMatches, counted before anything is allocated. A multi-valueNotInis compiled bychange 1 as one conjunction, so it stays a single selector producing a single rule and is
invisible to both bounds above, yet each value still becomes its own IPSet and its own condition
on that rule. Counting the matches a selector expands into catches that:
NotIncontributes oneper value,
Inone per branch, andExists/DoesNotExistone each. It is counted before theshortcut that returns a selector with no match expressions, so a selector made only of
matchLabelsis bounded too.4. Canonicalize
ipBlockCIDRs, and stop reporting a failed translation as successTwo compounding problems. An
ipBlockCIDR naming the all-addresses block with host bits set(
10.0.0.0/0) was rejected before canonicalization, even though it denotes exactly0.0.0.0/0.That failed translation of the whole policy — and the v2 controller converted the failure into a
successful no-op, so neither the peer rule nor the policy's default drop was installed and nothing
signalled it.
util.NormalizeCIDRclears host bits. The LinuxipBlockpath validates throughit and emits the canonical form, and canonicalizes its except CIDRs through a helper used only
by that path, so they de-duplicate and are recognized by the all-addresses split.
the key is requeued. Deliberate datapath limitations (the Windows unsupported features, NPM
Lite's CIDR-only peers) cannot resolve on retry and stay suppressed with a warning as before.
IsIPV4itself is deliberately left byte-identical, because it is shared with the Windows and NPMLite direct-rule paths which are out of scope here.
5. Bound the NPM HTTP API and stop enabling its debug routes by default
The API listens on the host network of a privileged process, and its server was created with only
an address and a handler: no read, write or idle deadline, no header bound, no limit on concurrent
connections. The cache route additionally serializes the entire policy cache into memory per
request while holding the cache lock, so slow clients could hold an unbounded number of full cache
copies alive.
The server now has deadlines, a header bound, a connection ceiling, and admits a single cache
encoding at a time with the rest shed as 503. This server-side bound is the load-bearing part of
the change: it holds regardless of whether the debug route is enabled, which is the configuration
that matters for managed clusters. The debug and pprof routes are also no longer enabled by
default in the deployment manifests. Prometheus metrics are unchanged.
The routes are also no longer reachable from a pod. They are served on the host network of a
privileged process, so any pod on the node could reach them through the node address it reads from
the downward API. Both are now served only to requests that originate on the node itself: a pod has
its own network namespace and cannot reach the node's loopback, while the on-node tooling that
consumes them already connects over localhost. A refused request is answered before the cache is
encoded. The Prometheus routes are deliberately not restricted, because they are scraped from off
the node.
The profiling handlers are also mounted at the pprof prefix rather than at
/debug/. The routermatched a
/debug/prefix ahead of the individually named pprof routes, so those named routesnever ran and every handler on the default mux was reachable under
/debug/. Mounting the defaultmux at the pprof prefix serves the profiles, including the subpaths the named list missed such as
/debug/pprof/goroutine, and exposes nothing else later registered on that mux.6. Materialize ipsets in the kernel only when a policy references them
NPM creates two ipsets per distinct pod label, on every node, and applied every set to the kernel
unconditionally. Kubernetes places no limit on how many labels a pod may carry, so one pod with
tens of thousands of labels pushed thousands of sets into every node's kernel, pinned agent CPU
and memory until agents were OOM killed, and delayed policy programming in unrelated namespaces
while they recovered.
ApplyIPSetsOnNeednow defaults to true, so a set reaches the kernel only once a network policyreferences it. Incidental pod labels never become kernel state at all.
This deliberately does not use a creation limit. Three limit designs were implemented and rejected
after live testing, each failing the same way: refusing to create a label set always creates an
escape, because either the policy cannot install or it installs referencing an empty set and the
pod escapes the policy that selects it. A per-pod cap additionally does not bound anything, since
the same labels spread across more pods reproduce the blowup. On-demand materialization refuses
nothing — sets are still tracked and pods still join them, so a set is already populated by the
time a policy references it, and only kernel materialization is deferred.
Deferring kernel materialization does not on its own bound what those labels cost in the agent.
The sets are still tracked, which is what keeps enforcement correct, and
ipset_countscarries theset name as a label, so the number of series it reported followed workload labels rather than
anything an operator controls. The per-set breakdown now stops growing at a bound far above what a
cluster's namespaces, policies and workloads produce. The aggregate counters are untouched and stay
exact, and nothing NPM does reads the breakdown, so only reported detail is limited; an operator can
tell it is incomplete by comparing the reported series against
num_ipsets. The shipped scrapeconfiguration drops
npm_ipset_countsby default, so this affects only operators who opted intothat advanced metric.
Measured with one pod carrying 34,000 labels, the size named in the report: agent memory for those
labels drops from 88.8 MB to 14.6 MB against a 300 MiB container limit, and the metrics response
from 6.42 MB to 1.87 MB, which no longer varies with the labels a workload chooses.
Tests
Unit tests accompany each change, covering both the corrected behaviour and the unchanged side
(positive selectors untouched, at-limit inputs passing, invalid CIDRs still rejected,
unsupported-feature errors still suppressed, ordinary multi-peer multi-port policies well inside
the ACL budget).
The HTTP changes are covered by tests for the loopback guard (IPv4 and IPv6 loopback admitted, a
pod address and the node address refused, a malformed remote address refused, and a refused request
never reaching the cache encoder) and for the pprof mount (every profile subpath served, the routes
still behind the guard, and nothing else on the default mux reachable). The metric bound is covered
by tests that the series stop at the bound while the aggregate counters stay exact, that a freed
slot is reusable, and that behaviour below the bound is unchanged.
go build ./npm/...andgo test ./npm/...pass. The only failures in the tree are innpm/pkg/controlplane/controllers/v1andnpm/cmd, which fail identically on unmodifiedmaster(the latter shells out to iptables and needs root).
golangci-lintwas run over every changed package against an unmodifiedmasterworktree forcomparison, with no new finding.
Validation on live AKS clusters
Each change was exercised individually against the behaviour it fixes, on Linux AKS with Azure CNI
and
--network-policy azure.masterand was blocked on this branchNotInlist past the match bound is rejected while one exactly on the bound still translates/0policy installs and its default drop is enforcedipset_countsstays within its bound. Contrasted on one cluster, the same pod produces 0 kernel sets on this branch against 4,043 withApplyIPSetsOnNeedoffRegression:
port, cross-namespace denial, multi-value
Infan-out) passes 22/22, including while theoversized policies from change 3 and a many-label pod from change 6 were live. The same matrix
scores 9/22 on unmodified
master, and the 13 failures are exactly the behaviours this PRcorrects. An 8-check matrix for the reachability and cardinality bounds passes 8/8.
e2e.testmatched to the cluster version,--ginkgo.focus=NetworkPolicy --ginkgo.skip=SCTP): 45/45 passed, run against both theprevious ApplyAll configuration and the new on-demand default.
test/cyclonus/test-cyclonus.sh): 112/112 passed.Known limitations
These are stated so a reviewer does not have to discover them, and so the ICMs are not closed on
a stronger claim than the evidence supports.
holds 14.6 MB for them, down from 88.8 MB, because each label still allocates a userspace
IPSetso that a set is already populated when a policy references it. Holding unreferenced label
membership in a compact index is the remaining follow-up. A creation cap is not — three variants
were tried and each produced an enforcement escape, either blocking a policy from installing or
leaving it referencing an empty set.
ipset_countsbound is first-come. Slots are claimed as sets appear, so a workload thatfloods labels before a policy is created can occupy them and leave later, legitimate sets without
a per-set series. The aggregate counters are unaffected. Following the existing
TODO kernel-based prometheus metricsmarkers, so that the breakdown tracks kernelmaterialization instead, would bound it by policy-referenced sets rather than by arrival order;
that changes what current dashboards report and is left for a change that can be signed off on
its telemetry impact. For the same reason
ipsetInventoryMapis left unbounded: it gates theaggregate entry-count decrement, and bounding it without care makes that counter drift.
ErrUnsupportedIPAddressandthe selector validation errors are permanent functions of the policy object, but they now go
through
AddRateLimitedand recur on every node. Classifying them as terminal (log once,Forget, emit an Event on the object) is the follow-up.a pod is refused immediately, but the Prometheus routes still share the ceiling and a pod can
still hold connections against it. A shorter idle timeout, a per-source limit, or a separate
listener for metrics would address it.
reachable from off the node to be scraped, so they cannot be restricted to loopback the way the
debug routes are. Their response is now bounded rather than scalable by workload labels, but
authenticating them needs a token review and RBAC and is out of scope here.
Deployment notes for reviewers
its config file with
DefaultConfig—cmd/root.gonotes "there is no config merging withdefault, if config is loaded, options must be set", and
start.gounmarshals into a zero-valueConfig. Any cluster that mounts a configmap therefore takes Go zero values for absent keys, soApplyIPSetsOnNeedmust be set explicitly in the deployed configmap, not only inDefaultConfig. This PR sets it innpm/azure-npm.yaml; the AKS-managed configmap is generatedelsewhere and needs the same key.
DefaultConfigchanges in change 5 only cover the case where no configfile loads at all. The server-side bounds, not the toggles, are what protect a cluster whose
configmap enables the debug route.
Scope
Linux NPM v2.
util.IsIPV4,deDuplicateExcept,exceptDirectDropRules,directPeerAndPortAllowRuleand the dataplane ipset member check are byte-identical tomaster,so the Windows and NPM Lite specific paths are unchanged, and no NPM v1 file is touched. Note that
npm/pkg/controlplane/translation/is a single compiler shared by v2 on both operating systems,so changes 1-4 necessarily run on Windows v2 as well; no Windows-specific code path was added or
altered.