feat: proposal for Weighted Routing Across Multiple InferencePools by zirain · Pull Request #2657 · theagentrouter/agent-router

zirain · 2026-09-09T02:44:00Z

Description

Add proposal for Weighted Routing Across Multiple InferencePools, this's generally for GAIE GatewayWeightedAcrossTwoInferencePools conformance test

Related Issues/PRs (if applicable)

xref: #2638

Special notes for reviewers (if applicable)

netlify · 2026-09-09T02:44:06Z

✅ Deploy Preview for theagentrouter canceled.

Name	Link
🔨 Latest commit	`9432438`
🔍 Latest deploy log	https://app.netlify.com/projects/theagentrouter/deploys/6acefa132756a429ead46206

codecov · 2026-09-09T02:47:26Z

Codecov Report

❌ Patch coverage is 92.06897% with 23 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/extensionserver/post_translate_modify.go	91.45%	20 Missing ⚠️
internal/extensionserver/inferencepool.go	93.18%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

zirain · 2026-09-09T13:55:53Z

/retest

zirain · 2026-09-10T09:27:03Z

/retest

zirain · 2026-09-13T13:19:42Z

Key changes:
LDS

              http_filters:
              - name: envoy.filters.http.ext_proc/endpointpicker/primary-inference-pool_inference-conformance-app-backend_ext_proc
                typed_config:
                  '@type': type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
                  grpc_service:
                    envoy_grpc:
                      authority: primary-endpoint-picker-svc.inference-conformance-app-backend.svc:9002
                      cluster_name: envoy.clusters.endpointpicker_primary-inference-pool_inference-conformance-app-backend_ext_proc
                  message_timeout: 300s
                  processing_mode:
                    request_body_mode: FULL_DUPLEX_STREAMED
                    request_header_mode: SEND
                    request_trailer_mode: SEND
                    response_body_mode: FULL_DUPLEX_STREAMED
                    response_header_mode: SEND
                    response_trailer_mode: SEND
              - name: envoy.filters.http.ext_proc/endpointpicker/secondary-inference-pool_inference-conformance-app-backend_ext_proc
                typed_config:
                  '@type': type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
                  grpc_service:
                    envoy_grpc:
                      authority: secondary-endpoint-picker-svc.inference-conformance-app-backend.svc:9002
                      cluster_name: envoy.clusters.endpointpicker_secondary-inference-pool_inference-conformance-app-backend_ext_proc
                  message_timeout: 300s
                  processing_mode:
                    request_body_mode: FULL_DUPLEX_STREAMED
                    request_header_mode: SEND
                    request_trailer_mode: SEND
                    response_body_mode: FULL_DUPLEX_STREAMED
                    response_header_mode: SEND
                    response_trailer_mode: SEND
              - name: envoy.filters.http.header_to_metadata
                typed_config:
                  '@type': type.googleapis.com/envoy.extensions.filters.http.header_to_metadata.v3.Config
                  request_rules:
                  - header: agent-session-id
                    on_header_present:
                      key: session.id
                      metadata_namespace: io.envoy.ai_gateway

RDS:

           weighted_clusters:
              clusters:
              - name: httproute/inference-conformance-app-backend/httproute-weighted-two-pools/rule/0/inferencepool/primary-inference-pool
                typed_per_filter_config:
                  envoy.filters.http.ext_proc/endpointpicker/secondary-inference-pool_inference-conformance-app-backend_ext_proc:
                    '@type': type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute
                    disabled: true
                weight: 70
              - name: httproute/inference-conformance-app-backend/httproute-weighted-two-pools/rule/0/inferencepool/secondary-inference-pool
                typed_per_filter_config:
                  envoy.filters.http.ext_proc/endpointpicker/primary-inference-pool_inference-conformance-app-backend_ext_proc:
                    '@type': type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute
                    disabled: true
                weight: 30

zirain · 2026-09-15T08:59:29Z

/retest

zirain · 2026-09-24T08:52:59Z

/retest

zirain · 2026-09-24T22:07:54Z

/retest

zirain · 2026-09-24T22:11:57Z

/retest

zirain · 2026-09-28T02:50:39Z

/retest

siddharth1036 · 2026-09-29T13:03:23Z

@zirain curious why we scoped this to HTTPRoute only and didn’t include AIGatewayRoute as well?

A couple of follow-ups on the AIGatewayRoute side:

Why not AIGatewayRoute in this PR?
AIGatewayRoute currently CEL-rejects >1 InferencePool per rule (only one InferencePool backend is allowed per rule). Was that a deliberate defer for GAIE conformance (GatewayWeightedAcrossTwoInferencePools is HTTPRoute-only), or are there other blockers (controller translation, EPP filter wiring, etc.) before we can lift that restriction?
modelNameOverride with multiple InferencePools
Today AIGatewayRouteRuleBackendRef.modelNameOverride is documented as ignored for InferencePool refs. If/when we allow weighted multi-pool on AIGatewayRoute, how should that interact?
- Keep ignoring override for InferencePool (pools own model selection via EPP)?
- Or support per-pool modelNameOverride the way we do for weighted AIServiceBackends (virtual model → different upstream model per destination)?
- If per-pool, would the override need to land before EPP scoring (router-level ext_proc / body mutation), and how would that compose with the per-ClusterWeight EPP disable pattern in this PR?

zirain · 2026-09-29T13:10:06Z

@zirain curious why we scoped this to HTTPRoute only and didn’t include AIGatewayRoute as well?

A couple of follow-ups on the AIGatewayRoute side:

Why not AIGatewayRoute in this PR?
AIGatewayRoute currently CEL-rejects >1 InferencePool per rule (only one InferencePool backend is allowed per rule). Was that a deliberate defer for GAIE conformance (GatewayWeightedAcrossTwoInferencePools is HTTPRoute-only), or are there other blockers (controller translation, EPP filter wiring, etc.) before we can lift that restriction?

modelNameOverride with multiple InferencePools
Today AIGatewayRouteRuleBackendRef.modelNameOverride is documented as ignored for InferencePool refs. If/when we allow weighted multi-pool on AIGatewayRoute, how should that interact?

Keep ignoring override for InferencePool (pools own model selection via EPP)?

Or support per-pool modelNameOverride the way we do for weighted AIServiceBackends (virtual model → different upstream model per destination)?

If per-pool, would the override need to land before EPP scoring (router-level ext_proc / body mutation), and how would that compose with the per-ClusterWeight EPP disable pattern in this PR?

We could do that later, the main goal is for GAIE conformance test.

nacx

Thank you!

Signed-off-by: zirain <zirain2009@gmail.com>

zirain · 2026-10-14T04:25:45Z

/retest

zirain · 2026-10-14T04:53:55Z

/retest

zirain · 2026-10-14T05:11:12Z

/retest

zirain · 2026-10-14T06:44:53Z

/retest

nacx · 2026-10-14T07:39:34Z

+	if !ok {
+		return nil, fmt.Errorf("could not determine owning HTTPRoute rule from route name")
+	}
+	httpRoute, err := s.retrieveAndCacheHTTPRoute(ctx, cache, key)


The HTTPRoute (and, at :817, each InferencePool's appProtocol) is still read through s.k8sClient. That is the cached mgr.GetClient() (cmd/controller/main.go:486), and nothing orders it against Envoy Gateway's own HTTPRoute watch. The impact is now larger, because failures here are fatal. A newly created HTTPRoute that Envoy Gateway translates before the controller cache sees it hits HTTPRoute ... not found (:806). A rule just appended can hit rule index ... out of range (:810). Either one fails the whole PostTranslateModify, not just this route. A weight-only edit can still program the old weights silently.

Use an uncached reader (mgr.GetAPIReader()) for these Gets. Or treat not-found and out-of-range as "leave this route merged and narrow it to one pool" rather than a global error. Either way, document the remaining window. I could not observe how large the lag is in practice.

nacx · 2026-10-14T08:16:43Z

+	}
+	if len(weighted) == 0 {
+		if len(otherClusterWeights) == 0 {
+			return nil, fmt.Errorf("route %q has no non-zero-weight InferencePool backendRef and no other backend to keep", route.Name)


Setting every InferencePool backendRef to weight: 0 is valid in Gateway API, and it is a normal way to drain a rule. The spec says such requests get a 500. Here it returns an error, and splitWeightedInferencePoolRoutes (:759-760) and PostTranslateModify (:148-150) pass it up. The extension hook then fails for every listener, route and cluster on the gateway, not only this rule. manifests/envoy-gateway-values.yaml doesn't set failOpen, so one tenant's HTTPRoute can block every other xDS update until it is fixed. What Envoy Gateway does on a hook error I could not determine from this repo.

Treat it as config, not an internal error. Rewrite the route to a DirectResponse 500, or to a WeightedClusters with only Envoy Gateway's invalid-backend-cluster. Set the route metadata to an empty pool list, as the drain branch already does. The "all weights zero fails the translation" table case would then become a positive test.

if len(otherClusterWeights) == 0 { // Every backendRef has weight 0: Gateway API says to return 500 for this rule only. clone.Action = &routev3.Route_DirectResponse{DirectResponse: &routev3.DirectResponseAction{Status: 500}} return clone, nil }

nacx · 2026-10-14T08:17:52Z

+	}
+	clusterWeights := make([]*routev3.WeightedCluster_ClusterWeight, 0, len(otherClusterWeights)+len(weighted))
+	for _, cw := range otherClusterWeights {
+		// A non-InferencePool backend: no endpoint picker may run for it.


This keeps the Service's ClusterWeight and turns off every EPP filter on it, so the request goes to the Service's cluster with no x-gateway-destination-endpoint header. Per the comment at :1057-1059 and the test fixture ("EG hands the route's extension resources to every per-backendRef cluster of the rule, so the Service's cluster carries the pools' metadata too"), that cluster also went through PostClusterModify. There, handleInferencePoolCluster set it to ORIGINAL_DST / CLUSTER_PROVIDED with UseHttpHeader, and set EdsClusterConfig = nil (post_cluster_modify.go:73-97). It no longer points at the Service's endpoints. Its traffic share then goes to whatever ORIGINAL_DST falls back to when the header is missing, not to the Service. splitWeightedInferencePoolRoutes never removes or restores that cluster.

The tests hide this. Both mixed-backend cases build svcCluster with createMergedInferencePoolCluster (ORIGINAL_DST) and only check the route side. Options: in PostClusterModify, only apply handleInferencePoolCluster when the cluster is actually an InferencePool backend. Envoy Gateway's per-backendRef cluster name (.../backend/<i>) together with the HTTPRoute tells you which ones are. Alternatively, reject mixed rules for now and document that. Add a test that checks the Service cluster's discovery type and EDS config after the full PostClusterModify + PostTranslateModify sequence.

nacx · 2026-10-14T08:18:32Z

+		for _, vh := range rc.VirtualHosts {
+			for i, route := range vh.Routes {
+				pools := getInferencePoolsByMetadata(route.Metadata)
+				if len(pools) < 2 {


The new mixed-backend support only starts when the route carries two or more pools. Take a rule with one InferencePool and a Service. Per the comment at :822-825, Envoy Gateway emits a WeightedClusters with only the Service, so the pool gets no traffic. The route metadata still names the pool, so patchVirtualHostWithInferencePool leaves its EPP on for the Service traffic. That behaviour predates this PR. But the PR now handles two pools plus a Service and not one pool plus a Service, which is surprising. Also enter the split when the route action is WeightedClusters and the metadata has at least one pool, or document the gap. Neither doc page mentions mixing pools with other backend kinds.

nacx · 2026-10-14T08:19:11Z

+	}
+	clone.Name = name
+	buildEPPMetadataForCluster(clone, []*gwaiev1.InferencePool{pool})
+	setClusterMetadataBackendName(clone, pool.Namespace, pool.Name, httpRouteName, routeRuleIndex, backendRefIndex)


Why stamp per_route_rule_backend_name on HTTPRoute clones? In production, only maybeModifyCluster writes it, and only for AIGatewayRoute-owned clusters (:389, :451). For a plain HTTPRoute nothing sets it on the merged cluster. The mixed-backend test sets it on its input clusters by hand, which doesn't happen in production. If a reader needs it, say which one in a comment. Note also that the namespace segment is the pool's namespace, not the route's, unlike the AIGatewayRoute callers that pass aigwRoute.Namespace. If nothing reads it, drop it.

zirain · 2026-10-15T02:37:02Z

wait for #2657, marked this as draft.

zirain marked this pull request as ready for review September 9, 2026 06:36

zirain requested a review from a team as a code owner September 9, 2026 06:36

zirain force-pushed the GatewayWeightedAcrossTwoInferencePools branch from 80b213b to 7d6a1a0 Compare September 9, 2026 13:35

zirain force-pushed the GatewayWeightedAcrossTwoInferencePools branch from 39bc12c to b7a5631 Compare September 13, 2026 11:16

missBerg added enhancement New feature or request area/routing Routing, fallback, resilience, InferencePool integration labels Sep 24, 2026

nacx reviewed Oct 13, 2026

View reviewed changes

zirain added 5 commits October 13, 2026 18:28

docs: proposal for Weighted Routing Across Multiple InferencePools

b9f1817

Signed-off-by: zirain <zirain2009@gmail.com>

impl

94175b8

Signed-off-by: zirain <zirain2009@gmail.com>

use WeightedCluster

2ab7fc6

Signed-off-by: zirain <zirain2009@gmail.com>

precommit

6f3211f

Signed-off-by: zirain <zirain2009@gmail.com>

fix

bc3c5e8

Signed-off-by: zirain <zirain2009@gmail.com>

zirain force-pushed the GatewayWeightedAcrossTwoInferencePools branch from fad2e92 to bc3c5e8 Compare October 13, 2026 13:01

zirain added 5 commits October 14, 2026 09:02

revert

a168d0d

Signed-off-by: zirain <zirain2009@gmail.com>

remove

02030a5

Signed-off-by: zirain <zirain2009@gmail.com>

Merge branch 'main' into GatewayWeightedAcrossTwoInferencePools

5bcdab4

fix

bdc92a1

Signed-off-by: zirain <zirain2009@gmail.com>

lint

8a81739

Signed-off-by: zirain <zirain2009@gmail.com>

fix conformance test

9432438

Signed-off-by: zirain <zirain2009@gmail.com>

nacx reviewed Oct 14, 2026

View reviewed changes

zirain marked this pull request as draft October 15, 2026 02:37

This was referenced Oct 15, 2026

Mixed Service + custom backendRefs in one HTTPRoute rule lose traffic, and the outcome flips depending on MergeBackends envoyproxy/gateway#10269

Open

feat: weighted routing across multiple InferencePools #2887

Draft

Conversation

zirain commented Sep 9, 2026

Uh oh!

netlify Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for theagentrouter canceled.

Uh oh!

codecov Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

zirain commented Sep 9, 2026

Uh oh!

zirain commented Sep 10, 2026

Uh oh!

zirain commented Sep 13, 2026

Uh oh!

zirain commented Sep 15, 2026

Uh oh!

zirain commented Sep 24, 2026

Uh oh!

zirain commented Sep 24, 2026

Uh oh!

zirain commented Sep 24, 2026

Uh oh!

zirain commented Sep 28, 2026

Uh oh!

siddharth1036 commented Sep 29, 2026

Uh oh!

zirain commented Sep 29, 2026

Uh oh!

nacx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zirain commented Oct 14, 2026

Uh oh!

zirain commented Oct 14, 2026

Uh oh!

zirain commented Oct 14, 2026

Uh oh!

zirain commented Oct 14, 2026

Uh oh!

nacx Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

nacx Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

nacx Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

nacx Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

nacx Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

zirain commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify Bot commented Sep 9, 2026 •

edited

Loading

codecov Bot commented Sep 9, 2026 •

edited

Loading