fix: add one InferencePool ext proc filter per pool, not per route match

[gerritvd]

Description

patchListenerWithInferencePoolFilters appends one endpoint-picker ext_proc HTTP filter
per entry in the inferencePools slice it is given. That slice is built in
maybeModifyListenerAndRoutes from routeNameToVHRouteNameToInferencePool, which is keyed
by xDS route name — so a pool referenced by N route matches appears N times, and the
listener ends up with N identically-named copies of the same filter.

There is already a guard against duplicates, but it only searches
httpConManager.HttpFilters. New filters accumulate in the local poolFilters slice and are
not spliced into httpConManager.HttpFilters until after the loop, so the guard never sees
what the loop itself just added.

Because httpFilterNameForInferencePool is derived from the pool's namespace and name, every
duplicate shares one filter name. Per-route typed_per_filter_config is keyed by filter name,
so the existing per-route disable logic cannot express "enable one of these and disable the
rest" — it can only enable or disable the whole set. On the pool's own routes the set is
enabled, so every copy runs and opens its own gRPC stream to the EPP.

Impact for a single client request, on a Gateway with two InferencePools whose HTTPRoutes each
have 16 InferencePool-backed matches:

• 32 adding inference pool ext proc filter log lines per translation pass (16 per pool)
• 32 identically-named ext_proc filters in the listener filter chain
• 16 gRPC streams to the EPP, i.e. 16 full scheduling passes, for one request

The last point is the damaging one. It is not only wasted latency and 16x EPP load: each
redundant scheduling pass independently mutates EPP scheduler state, which makes stateful
plugins unusable. In our case no-hit-lru-scorer rotated its LRU choice on every one of the
16 passes, so cold requests never actually spread across replicas, and prefix-cache affinity
accounting was meaningless.

Reducing the HTTPRoute from 16 InferencePool-backed matches to 12 reduced the EPP calls from
16 to 12, confirming the count tracks match count exactly.

Observed on Envoy AI Gateway v1.0.0 with Envoy Gateway v1.8.1 (the pinned pairing from the
v1.0.0 dependency table), Envoy Proxy v1.38.x, using KServe 0.20 LLMInferenceService, which
generates an HTTPRoute with 8 InferencePool-backed rules — one of them carrying 8 path
matches — for a total of 16. Both affected code paths are unchanged on main, and the added
test fails there without this patch.

Repro

1. Revert this patch, or apply only the added test.
2. Run go test ./internal/extensionserver -run 'TestPatchListenerWithInferencePoolFilters/repeated_inference_pool_is_added_once' -count=1.
3. The test fails: four entries for one pool produce five filters (four identical
   envoy.filters.http.ext_proc/endpointpicker/pool1_test-ns_ext_proc plus the router)
   where two are expected.

Fix

Two changes, either of which resolves the reported behavior; together they also guard the
other call path:

1. Deduplicate by namespace/name when building listenerToInferencePools, so the slice
   handed to patchListenerWithInferencePoolFilters holds each pool once per listener.
2. Track filter names added within the loop in patchListenerWithInferencePoolFilters, so the
   duplicate guard accounts for filters not yet spliced into the chain.

No behavior change for the single-match case, and none for distinct pools on a shared
listener: both existing subtests (listener with new inference pool filter, listener with multiple inference pools) pass unmodified.

Verification

• make precommit passes, with no generated-file churn.
• make test passes.
• The added test fails on main without the fix (five filters for one pool where two are
  expected) and passes with it.

Beyond the unit tests, the patched controller was deployed to a staging cluster running the
two-InferencePool KServe 0.20 setup described above, replacing the v1.0.0 controller image.
The patch applied there is identical to this one. Measured before and after, same cluster,
same HTTPRoutes:

Measurement	v1.0.0	this patch
adding inference pool ext proc filter per translation pass	32	2
EPP gRPC streams per client request	16	1
Distinct replicas chosen for 5 distinct cold prompts	1 of 2	3/2 split across both

Controller logs showed no errors across the rollout and the translation pass. The 44 per-route
disabling inference pool filter lines are unchanged and remain correct: two HTTPRoutes with
16 InferencePool-backed matches each disable the other pool's filter, and 3 Service-backed
catch-all matches each disable both.

Cache affinity is preserved rather than traded away for spreading: re-sending an already-seen
prompt three times pinned to the replica that first served it, scoring 2 points higher there
from the prefix-cache scorer, while cold prompts continued to alternate.

Related Issues/PRs (if applicable)

envoyproxy/gateway#5002 fixed the same class of defect in Envoy Gateway, where extension
filters accumulated one copy per reference and produced multiple matches while processing
routes.

Touches the same InferencePool extension-server path as #2142 (preserve InferencePool metadata
on extension hooks). Overlaps #2517 only in internal/extensionserver/extensionserver_test.go;
that PR changes inferencepool.go and post_cluster_modify.go, so the two should rebase
cleanly past each other.

For context, istio/istio#58392 describes the inverse failure in a different implementation:
multiple InferencePools on one Gateway losing the ext_proc config for all but the first.

Special notes for reviewers (if applicable)

The dedupe key is namespace/name rather than the filter name, since httpFilterNameForInferencePool
already derives from both; using the object identity keeps the intent clear at the call site.

AI-assisted patch prep; I reviewed the code and own the change.

[gerritvd]

Ping — this has been open since Aug 28 with no CI runs. As a first-time
contributor, build_and_test, precommit_check, and codeql are all waiting on
a maintainer to approve the workflow run, so there's currently no test signal
here. Could someone approve it?

Why it may be worth a look: when an InferencePool is referenced by N route
matches, the extension server adds N identically-named ext_proc filters to the
listener, and because per-route typed_per_filter_config is keyed by filter
name, all N of them run. In our setup that means one client request opens 16
gRPC streams to the endpoint picker. Beyond the wasted latency, it makes
stateful EPP plugins unusable, since every redundant scheduling pass mutates
scheduler state independently.

The change is ~17 lines plus a regression test that fails on main without it.
make precommit and make test both pass locally, and I ran the patched
controller on a two-InferencePool KServe 0.20 cluster: 32 filters became 2, and
16 EPP streams became 1, with cache affinity preserved.
@Xunzhuo this sits in the InferencePool / Gateway API Inference Extension area.
@yuzisun the reproduction is a KServe LLMInferenceService. @nacx you've reviewed
most of the recent changes under internal/extensionserver/.

Happy to adjust the approach, split it, or add cases — just let me know.

[cdxvy30]

Independent confirmation from a separate cluster, with a single InferencePool.

Environment

• Envoy AI Gateway v1.0.0
• Envoy Gateway v1.8.3
• KServe v0.20.0-rc0.1: LLMInferenceService with the default kserve-config-llm-router-route preset. The preset yields 16 InferencePool-backed matches for one pool: 4 *-path rules + v1-model-routing (8 matches) + 4 *-publisher-path rules.

Observed

A single curl to /v1/completions produces 16 EPP invocations, all sharing the same Envoy-generated x-request-id:

{"level":"info","ts":1787633904.8779063,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744"}
{"level":"info","ts":1787633904.8779678,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"}
{"level":"info","ts":1787633904.8781154,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744"}
{"level":"info","ts":1787633904.8781772,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"}
{"level":"info","ts":1787633904.8783343,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744"}
{"level":"info","ts":1787633904.8783946,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"}
...
{"level":"info","ts":1787633904.901453,"caller":"handlers/server.go:685","msg":"EPP sent response body back to proxy","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744"}
{"level":"info","ts":1787633904.9015977,"caller":"handlers/server.go:685","msg":"EPP sent response body back to proxy","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744"}
{"level":"info","ts":1787633904.9017153,"caller":"handlers/server.go:685","msg":"EPP sent response body back to proxy","x-request-id":"586beabe-8d5f-4a0d-b0c7-dafe07816744"}

Against this patch

A test driving maybeModifyListenerAndRoutes with one pool referenced by 16 routes yields 16 identically named ext_proc/endpointpicker filters on main, and exactly one with this PR applied; the existing TestPatchListenerWithInferencePoolFilters and TestMaybeModifyListenerAndRoutes still pass. Verified at the unit-test level only; I have not run the patched controller in-cluster yet.

With a single replica the routing outcome is unaffected, but EPP load and its metrics are inflated 16x. With more replicas this becomes the scheduler-state problem described in the PR.

[netlify]

✅ Deploy Preview for theagentrouter canceled.

Name	Link
🔨 Latest commit	0a8d2c4
🔍 Latest deploy log	https://app.netlify.com/projects/theagentrouter/deploys/6ac8019a7064140008486c7d

[nacx]

Thanks for the patience and apologies for the late review.
Overall LGTM. @kanurag94 mind taking a look?

[codecov]

Codecov Report

❌ Patch coverage is 90.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/extensionserver/post_translate_modify.go	90.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[william-rom]

Observed on 1.1.0 with Envoyproxy 1.39.x as well. Would be great to get this one on the next release.

[gerritvd]

/retest

[gerritvd]

@missBerg /retest hit the same flake twice: InferencePoolResolvedRefsCondition
waiting for Accepted on the secondary parent (300s timeout). Primary was
already Accepted+ResolvedRefs; the delete phases then passed. Same as
#1331, and this job was green earlier on this PR
(https://github.com/theagentrouter/agent-router/actions/runs/36282321855).

Is this a flake and should I run another retest?

[gerritvd]

/retest