Weekly Update — Week of July 20, 2026

[missBerg]

Resilience was the theme of the week — stream idle timeout with model failover landed alongside a per-backend fallback fix — with the long-awaited fix for the 1MB Secret limit, steady translation-fidelity work on the Anthropic and Gemini paths, and multi-tenant MCP design work taking shape. Thanks to everyone who shipped, reviewed, and filed. 🎉

🛣️ Multi-tenant MCP — design direction forming

How do you serve thousands of tenants from one MCP gateway? From the maintainer call: two approaches are on the table, and the preferred direction is static backends plus per-request ext-proc filtering — a custom ext proc narrows the allowed backends per tenant before initialization, reusing the existing fanout logic rather than supplying dynamic route config. A proposal doc is forthcoming for async review. The first building block is already open for review in #2351, and the work also surfaced a broader upstream question: Envoy's 256-backend cap is artificially low and worth relaxing.

One date for your calendar: EnvoyCon's 10th-anniversary virtual event is October 7, and the CFP closes Wednesday, July 22 — a good entry point for first-time speakers.

✨ What's new

Bug fixes & stability

• Stream idle timeout with failover to the next model — #2169 by @albe2669. A new streamIdleTimeout on route rules resets an upstream that goes quiet before the first token, so with retries configured the request falls over to the next backend instead of hanging until the overall request timeout.
• Filter config now splits across Secrets — #1865 by @zhaohuabing. The extproc and MCP proxy config is chunked into multiple Secrets, so deployments with many routes stop hitting the 1MB Kubernetes Secret limit. Fixes #1613 and addresses the same pain raised in #2341.
• Correct translation on per-backend fallback clusters — #2348 by @0YHR0. Per-backend clusters generated for priority-based fallback now get the fallback backend's schema translation applied. Fixes #2135.

Provider & translation coverage

• Prompt caching on tool results — #2358 by @johnpanos. Message-level cache_control on OpenAI tool result messages now survives translation to Anthropic-compatible backends, so cache breakpoints on tool_result blocks work through the OpenAI endpoint.
• /tokenize for OpenAI-compatible backends — #2371 by @hustxiayang. New endpoint for counting tokens through the gateway — useful for context-window budgeting without calling the model.
• Newer Claude models unblocked for structured output and effort — #2382 by @hustxiayang. The Claude model capability list is synced, a stopgap until the capability-hints design in #2368 lands.

Developer experience

• Named route rules — #2373 by @haruyama480. An optional name on AIGatewayRoute rules propagates to the generated HTTPRoute rules — stable, human-readable rule names in downstream metrics instead of index-based labels. Fixes #2372.

💡 New design proposals & feature requests

A strong week for design threads:

• Model capability hints for translation — #2368 by @hustxiayang. Proposes a translationHints field on backendRefs to replace model-name string matching for max_tokens defaults, JSON schema support, and reasoning-effort gating — so onboarding a new model stops requiring a code change in this repo.
• AIGatewayExtensionPolicy CRD — #2364 by @siddharth1036. A declarative, header-gated way to run a router-phase ext_proc on selected routes only, replacing the EnvoyPatchPolicy workaround and shrinking the blast radius from all catch-all traffic to just the traffic that opts in. Requires Envoy ≥ 1.39. Discussed on the maintainer call — the direction was well received, including as a possible injection point for MCP token exchange.
• Cross-namespace backendRefs for MCPRoute — #2385 by @SharathPrakash. Use ReferenceGrant to aggregate MCP servers owned by different teams into a single MCP endpoint, mirroring what AIGatewayRoute already supports.
• Keyless Azure auth — #2375 by @ahmedsiddiqq proposes Azure Workload Identity for AzureCredentials, and #2378 by @Sarthak160 revives Azure Managed Identity support — both remove static client secrets from Azure deployments.

👀 PRs looking for review

If you have time to help review, these are open and waiting:

• #2351 — Per-request backend fan-out subset and allowed tools via header by @PraveenKumarInjam — the first piece of the multi-tenant MCP work discussed on the maintainer call
• #2392 — Strip content-length when switching to streamed mode by @pyhp2017 — fixes the SSE truncation bug in #2391
• #2366 — Accept thinking_blocks signatures for Gemini function calls by @a-nych — fixes Gemini 3.x rejecting tool-call follow-ups from standard OpenAI clients (#2365)
• #2374 — Example Grafana dashboard for GenAI metrics by @baiyutang — an in-repo dashboard example was a direct ask from the maintainer call, where a token/cost dashboard built on the gateway's Prometheus metrics was demoed
• #2367 — Forward proxy egress for upstream AI traffic by @kanurag94
• #2127 — OpenAI to Anthropic translator by @ajac-zero

🙏 Thanks to this week's contributors

@hustxiayang, @0YHR0, @kanurag94, @albe2669, @zhaohuabing, @johnpanos, @haruyama480, @a-nych, @ajac-zero, @PraveenKumarInjam, @siddharth1036, @Sarthak160, @pyhp2017, @SharathPrakash, @ahmedsiddiqq, @baiyutang, @pjdurden, @Aias00, @Killusions, @yuehaii, @FAUST-BENCHOU, @arthurh4, @dragon3, @jappievw, @pbonuboinasnow, @isztldav, @toffentoffen, @VadimEisenberg, @mksinghtx, @Hritik003, @nacx — and everyone who showed up to triage, review, and discuss.

See you next week!