Repository navigation
Gcp context cache - #2
Closed
sivanantha321 wants to merge 41 commits into
Closed
sivanantha321 wants to merge 41 commits into
sivanantha321 wants to merge 41 commits into
Conversation
…router#2664) **Description** `buildStreamDoneHeaderMatchAction` emits a `GenericKey` action for a `Distinct` header, whose descriptor value is a constant. The request-time builder, `buildHeaderMatchAction`, emits `RequestHeaders` for the same header, keyed on the caller's actual value. The two phases of the same rule therefore never resolve to the same bucket: | Phase | Action | Descriptor value | |---|---|---| | Request-time (pre-flight) | `RequestHeaders` | the caller's `x-tenant-id` | | Stream-done (real cost) | `GenericKey` | a constant | Each tenant's counter takes only the pre-flight increment, while the real token cost from every tenant sharing the rule settles into one pooled generic bucket. That matches the Redis counters described in the issue. This mirrors the request-time action so stream-done settles against the bucket its pre-flight check reserved. One branch in one function. Tests: `TestBuildStreamDoneHeaderMatchAction` covers a `Distinct` header producing `RequestHeaders`, and an inverted `Exact` header still reporting with `ExpectMatch=true`. `TestStreamDoneDescriptorsMatchRequestTime` asserts the request-time and stream-done actions agree on both the descriptor kind and the key. Comparing keys alone is not enough: `GenericKey` and `RequestHeaders` share a key and differ only in the value, which is exactly how this hid. **Related Issues/PRs (if applicable)** Fixes theagentrouter#2551 **Special notes for reviewers (if applicable)** The comment I removed claimed the fallback was deliberate: "Distinct headers fall back to GenericKey because per-value bucketing is not applicable at stream-done time." I think that rationale is wrong and would rather surface the disagreement than quietly flip it. `ApplyOnStreamDone` entries are `RateLimit` actions on the same route as the request-time ones, so the phase controls when the report is sent rather than which actions can be evaluated, and a `RequestHeaders` action resolves the same way it does pre-flight. If the intent really was that per-value bucketing is unsupported here, the descriptors still should not silently disagree, and the better change is the issue's second option: reject or warn on `Distinct` combined with a token-based quota mode rather than pooling the cost. Happy to take it that way instead. The old behavior had no test coverage. The existing stream-done assertions only exercise `Exact` headers and the default bucket, which is why this change breaks nothing. Both new tests fail if the `GenericKey` branch is restored, and the `Exact` subtest correctly keeps passing. `make precommit` is clean and `go test ./internal/...` passes. I have not run this against a live Redis-backed cluster, so the descriptor pairing is verified at config-generation level only. If someone has a cluster handy, confirming the per-tenant counters would be a good extra check. I used an AI assistant while investigating and writing this change, noted per the generative AI policy in the contributing guide. I have reviewed and understand the diff and can revise it in review. Signed-off-by: Vignesh Chaturvedi <vigneshchaturvedi@gmail.com>
Add support for the GCP Vertex AI context caching explicit passthrough path: - Add CachedContent field to gcp.GenerateContentRequest wire schema so the Gemini API receives the cache reference. - Add CachedContent field to GCPVertexAIVendorFields (inline on ChatCompletionRequest) so callers can supply a pre-created cache resource name via the OpenAI-compatible API. - Copy the vendor field into the Gemini request in applyVendorSpecificFields. - Validate that cache_control markers and explicit cachedContent are not used together; return ErrMalformedRequest (HTTP 400) when both are present. - Add gcpRequestHasCacheControlMarkers helper to detect Anthropic-style cache_control on system/user/tool message content parts. - Add unit tests covering passthrough, validation error, and marker detection across all message types. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Introduce the internal/gcpcache package and the GCPAuthHandler interface required to surface GCP credentials to the resolver without duplicating credential management. Changes: - internal/filterapi/runtime.go: add GCPAuthHandler interface extending BackendAuthHandler with GCPTokenSource(), GCPRegion(), GCPProject() methods so in-process components can authenticate to GCP APIs. - internal/backendauth/gcp.go: implement GCPAuthHandler on gcpHandler; add GCPTokenSource() (wraps static token in StaticTokenSource), GCPRegion() and GCPProject() accessors. - internal/translator/gemini_helper.go: export OpenAIMessagesToGeminiContents and OpenAIToolsToGeminiTools so gcpcache can build the cached prefix in Gemini wire format without duplicating conversion logic. - internal/gcpcache/gcpcache.go: new CacheResolver interface and resolver implementation with breakpoint detection, prefix split, deterministic SHA-256 cache key (stored as Google cachedContents displayName), in-memory TTL memo, Google cachedContents list/create client, and post-create re-list to converge duplicate-create races across replicas. - internal/gcpcache/gcpcache_test.go: unit tests covering findBreakpoint, extractTTL, computeCacheKey determinism, cache miss+create, cache hit from Google list, memo hit (no API calls), memo expiry/refetch, create failure, TTL default/override, and duplicate-create race convergence. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Add the GCPCacheSetter interface to the translator package so the GCP Vertex AI translator can receive a resolved cache entry before RequestBody is called. The translator injects the cachedContent resource name into the Gemini request, uses the filtered (non-cached) message remainder as the request contents, and records cache-write token counts in ResponseBody for both streaming and non-streaming responses. Populate a shared CacheResolver on RuntimeBackend (via Server.LoadConfig) for every GCP Vertex AI backend so the in-memory TTL memo is preserved across requests. In upstreamProcessor.ProcessRequestHeaders, after the translator is selected, attempt cache resolution when the resolver, a GCPAuthHandler, and a GCPCacheSetter-capable translator are all present. Resolver failures produce a 502 ImmediateResponse; nil results (no cache_control markers) are a no-op. Add unit tests Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Add a GCPContextCachingSpec to AIServiceBackend in both api/v1beta1 and api/v1alpha1. The field carries Enabled (bool) and DefaultTTL (a GCP seconds-format duration string validated by a kubebuilder regex pattern). The controller wires it into filterapi.Backend.GCPContextCaching so the extproc layer can gate cache-resolver creation on the feature flag. In extproc/server.go, the shared CacheResolver is now only attached when GCPContextCaching.Enabled is true, keeping the resolver nil (and all cachedContents API calls suppressed) for backends that do not opt in. Generated: run make apigen codegen apidoc to regenerate CRD YAML, deepcopy functions, typed clients/listers/informers, and API reference. Tests added: - tests/crdcel: two AIServiceBackend fixtures (valid and invalid defaultTTL) exercised against the live envtest apiserver to verify CRD validation. - internal/controller/gateway_test.go: two reconcileFilterConfigSecret tests asserting that GCPContextCaching is populated when set and nil when absent. - internal/extproc/server_test.go: LoadConfig test asserting resolver creation is gated on Enabled=true / absent field / non-GCP backend. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Update the context cache documentation URL to point to the new docs.cloud.google.com location under gemini-enterprise-agent-platform. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Replace the local createCacheRequest/createCacheResponse/cacheUsageMetadata types in gcpcache with shared types defined in internal/apischema/gcp: - Add CreateCachedContent, CachedContent, CachedContentUsageMetadata, and EncryptionSpec to the gcp schema package so they can be reused across the codebase. - Set CreateCachedContent.TTL as string (GCP duration format e.g. "300s") rather than time.Duration, which marshals as a nanosecond integer and would be rejected by the Vertex AI REST API. - Set CachedContent.ExpireTime as time.Time so the response can be used directly without a separate time.Parse call. - Cast CachedContentUsageMetadata.TotalTokenCount (int32) to int when populating ResolveResult.TokenCount. - Update tests to unmarshal the POST body into gcp.CreateCachedContent instead of the now-removed local type. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
…ed output and effort (theagentrouter#2673) **Description** The config setup is blocked by theagentrouter#2369 --------- Signed-off-by: yxia216 <yxia216@bloomberg.net>
…outer#2593) **Description** The controller discovers authorization server metadata by probing well-known URIs derived from the issuer. Some authorization servers publish a complete document at a location that cannot be derived that way, such as an issuer of `https://example.com/api/idp/authn` whose metadata is served only at `https://example.com/api/idp/v4/authn/.well-known/openid-configuration`. The issuer cannot be pointed at the versioned path instead, because it is matched against the `iss` claim on every token. ```yaml securityPolicy: oauth: issuer: "https://example.com/api/idp/authn" authorizationServerMetadataUrl: "https://example.com/api/idp/v4/authn/.well-known/openid-configuration" ``` The controller fetches that URL verbatim, skipping the derived variants, so `issuer` keeps its single meaning and the two concerns are no longer coupled through one field. Fixes theagentrouter#2049 --------- Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> Signed-off-by: Huabing (Robin) Zhao <huabing@tetrate.io> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
**Description** Start the controller extension gRPC server only after the controller-runtime cache has synchronized, so Envoy Gateway cannot send extension calls before the controller has observed InferencePools and other watched resources. The server is registered as a non-leader-election manager runnable. Controller-runtime starts this runnable on every replica after its local cache has synced, preserving horizontally scaled extension-server traffic on non-leader replicas. The Helm deployment adds a gRPC startup probe with enough budget for the default cache sync timeout, preventing liveness from restarting the controller while the extension server is intentionally not serving. **Related Issues/PRs (if applicable)** Fixes theagentrouter#2581 **Special notes for reviewers (if applicable)** Verification: - `go test ./cmd/controller ./internal/controller ./internal/extensionserver -count=1` - `go test -race ./cmd/controller -run TestExtensionServerRunnable -count=1` - `go vet ./cmd/controller ./internal/extensionserver` - `make helm-test` - `make lint` This change was developed with assistance from OpenAI Codex. I reviewed the implementation and take responsibility for it. --------- Signed-off-by: liuhy <liuhongyu@apache.org> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…router#2699) **Description** Adds a capabilities page, "Agent2Agent (A2A) Traffic", that shows how to route A2A agents through the gateway today using only Envoy Gateway resources plus Envoy's native `envoy.filters.http.a2a` filter: an HTTPRoute for the agent card and JSON-RPC endpoint, an EnvoyPatchPolicy that enables the filter, a SecurityPolicy on the RPC rule, and an EnvoyProxy access-log format for the A2A fields. The page is marked as a preview and points early adopters to the ongoing work on first-class A2A support. Verified with Envoy Gateway v1.9.0 and Agent Router v1.1.0 on kind. **Related Issues/PRs (if applicable)** Related: theagentrouter#2070 **Special notes for reviewers (if applicable)** Docs only. No API or code changes. 🤖 Generated with Claude Code [1] 1: https://claude.com/claude-code --------- Signed-off-by: Erica Hughberg <erica.sundberg.90@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…agentrouter#2636) **Description** The MCP proxy's per-backend response collector (`session.go`) and the response proxy (`handlers.go`) decided whether to parse a backend response as JSON via an **exact string comparison** on the `Content-Type` header (`== "application/json"`). A backend responding with `Content-Type: application/json;charset=UTF-8` — the servlet-container default emitted by the official MCP Java SDK's servlet transports (`HttpServletStatelessServerTransport`, which does `setContentType("application/json")` + `setCharacterEncoding("UTF-8")`) — failed that check. The body fell through to the SSE parser, which found no `data:` lines and yielded **zero events and zero errors**, so the backend's `tools/list` response was silently discarded (connected-but-toolless, nothing logged). This parses the media type with `mime.ParseMediaType` and matches on the `application/json` media type, ignoring parameters like `charset` (per RFC 9110 §8.3; `charset` is meaningless for `application/json` per RFC 8259 §8.1). A small shared `isJSONContentType` helper replaces the three exact-match sites in the package (`sendRequestPerBackend`, `copyProxyHeaders`, `proxyResponseBody`). **Related Issues/PRs (if applicable)** Fixes theagentrouter#2568 **Special notes for reviewers (if applicable)** Added a regression test `TestSendRequestPerBackend_JSONContentTypeWithCharset` mirroring the existing `TestSendRequestPerBackend_BOMPrefixedJSON`, using a backend that returns `application/json;charset=UTF-8`. Verification (Go 1.26.6, `go test ./internal/mcpproxy/`): - New test passes with the fix. - RED→GREEN confirmed: reverting the `session.go` dispatch back to the exact string match makes the new test fail with `Diff: -message / +` (0 events collected); restoring the fix makes it pass. - Full `internal/mcpproxy` package suite green (`ok ... 2.980s`), `go vet` and `gofmt -l` clean on the touched files. AI-assisted; all results above are from real local runs. Happy to adjust. --------- Signed-off-by: Vladimir Babin <vovababin@gmail.com> Signed-off-by: chiliec <vovababin@gmail.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…uter#2678) **Description** Anthropic stream folding repeatedly concatenates growing text, thinking, and tool-argument strings. Each delta copies the entire accumulated prefix, causing quadratic allocation for long responses, including GenAI traces with message-content capture disabled. Accumulate fragments in per-block `strings.Builder` buffers while preserving initial content, interleaved indices, block replacement, and tool argument parsing at block stop. Add regression tests and a fold-only benchmark. This removes repeated prefix copies but does not implement a bounded-memory recorder or stop retaining source chunks. **Related Issues/PRs (if applicable)** Concrete performance improvement supporting theagentrouter#2668. Independent of the proposed configuration and recorder-interface changes, and of the usage-alias fix in theagentrouter#2635. **Special notes for reviewers (if applicable)** Using the same sampled-span benchmark from theagentrouter#2668 (1 MiB text, 8,192 deltas, content capture off), cumulative allocation fell from 4,330,508,344 B/op to 8,048,784 B/op, approximately a 99.8% reduction. These are second runs with `-benchtime=1x -count=2`, not peak heap or statistically rigorous latency claims. Local validation uses Go 1.26.6 / Windows amd64 in the isolated `1899d8d4` checkout used for the proposal baseline, with this implementation and tests applied. `go test ./internal/apischema/anthropic ./internal/tracing -count=1` and the full-span benchmark pass. The PR is based on current main; its Go 1.27.1 validation remains for CI. Developed with assistance from OpenAI Codex. Signed-off-by: 0YHR0 <97868579@qq.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…r#2299) **Description** Adds OpenRouter as an OpenAI-compatible real-provider data-plane test target. This wires TEST_OPENROUTER_API_KEY into the credential context, adds an OpenRouter backend using the OpenAI-compatible v1 schema, adds one gated chat-completions health-check case with openai/gpt-4o-mini, and exposes ENVOY_AI_GATEWAY_OPENROUTER_API_KEY to the data-plane CI job. Closes theagentrouter#2272. **Testing** - go test ./internal/testing - go test ./tests/data-plane -run '^$' - TEST_OPENROUTER_API_KEY= go test ./tests/data-plane -run '^TestWithRealProviders/health-checking/chat/completions/openrouter$' -count=1 -v - make precommit - make test - make test-data-plane **Notes** This PR is additive test infrastructure only; no production code paths are changed. The OpenRouter test skips unless TEST_OPENROUTER_API_KEY is set. I used Codex/AI assistance to help prepare this change and reviewed the implementation and test results before submitting. --------- Signed-off-by: Amrinder Randhawa <272048731+amarrtech@users.noreply.github.com> Signed-off-by: amarrtech <272048731+amarrtech@users.noreply.github.com> Co-authored-by: Amrinder Randhawa <272048731+amarrtech@users.noreply.github.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…oice of gcp to make it compatible with gcp (theagentrouter#2516) **Description** vLLM v0.12.0 removed the legacy top-level guided-decoding request fields (`guided_json`, `guided_regex`, `guided_choice`, `guided_grammar`, …) and replaced them with a single nested `structured_outputs` object: https://github.com/vllm-project/vllm/blob/v0.12.0/vllm/sampling_params.py#L33 . Clients migrating to the new format got no guided decoding on GCP/Gemini models, because the Gemini translator only understood the deprecated fields. This adds `structured_outputs` support so the new format is honored on the GCP path, while keeping the legacy `guided_*` fields working for backward compatity. --------- Signed-off-by: yxia216 <yxia216@bloomberg.net> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
**Description** Folding Anthropic stream chunks currently aliases usage from `message_start` (or the first `message_delta` when start usage is absent). Later deltas overwrite the original chunk's output token count. Folding a prefix of the same stream can also change the usage in an already returned response. Copy usage before accumulating output tokens so each folded response owns its mutable usage. Regression tests cover streams with and without a message start: the original token counts stay unchanged, a complete stream reports 7 output tokens, and folding its prefix reports 4 without modifying the complete result. **Related Issues/PRs (if applicable)** Independently discovered while inspecting the stream folding used by tracing. **Special notes for reviewers (if applicable)** Validation: the new regression test fails on the base commit for both cases. After the fix, `go test ./internal/apischema/anthropic -count=1`, `go vet ./internal/apischema/anthropic`, and `git diff --check` pass. Developed with assistance from OpenAI Codex. Signed-off-by: 0YHR0 <97868579@qq.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…entrouter#2601) **Description** This commit skips filter chains without an HTTP connection manager in `insertRequestHeaderToMetadataFilter` and `insertRouterLevelAIGatewayExtProc` instead of returning an error that aborts the whole `PostTranslateModify` hook. Like the existing skip paths (`patchListenerWithInferencePoolFilters`, `findListenerRouteConfigs`, `insertMCPBackendFilters`, `insertQuotaRateLimitFilter`), chains without an HCM are skipped rather than failing the hook. Per review feedback, the skip keys on the `findHCM` sentinel (`hcmIndex == -1`) instead of the returned error, so a future genuine error is not silenced; the error is still logged. Chains without an HCM (TCPRoute/UDPRoute listeners) cannot host HTTP filters, so finding no HCM in them is normal, not an error. A regression test is added: a listener mixing a TCP-proxy chain with an HCM chain must translate, with the filter injected only into the HTTP chain. This hits stock installs: `logRequestHeaderAttributes` defaults to `agent-session-id:session.id` when the flag is unset (`internal/requestheaderattrs/resolve.go`), and `insertRequestHeaderToMetadataFilters` walks every listener unconditionally — so any Gateway carrying TCP/UDP listeners makes xDS translation fail for the entire control plane (all Gateways `Programmed=False`, dataplane crash-looping on rejected config). **Related Issues/PRs (if applicable)** Fixes theagentrouter#2600 **Special notes for reviewers (if applicable)** `go test ./internal/extensionserver/`, `go vet`, and `gofmt` pass locally. Tested against a live fleet where this exact topology (HTTP inference Gateway + Syncthing TCP/UDP Gateway on one control plane) reproduced the failure. --------- Signed-off-by: Shikanime Deva <william.phetsinorath@shikanime.studio> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
) **Description** `gen_ai.provider.name` is derived from the backend API schema, so backends that share a schema are indistinguishable in metrics: a route fronting several OpenAI-compatible backends reports every request under the same provider value. Since backendRefs exist to route across multiple backends, this means the metrics cannot attribute error rate, latency, or token usage to the backend that served the request. This change adds a `gen_ai.backend` attribute to the GenAI metrics, set to the `namespace/name` of the AIServiceBackend that served the request. The attribute is not part of the GenAI semantic conventions. unknown is reported when no backend has been selected, and cardinality is bounded by the number of configured AIServiceBackends. **Related Issues/PRs (if applicable)** **Special notes for reviewers (if applicable)** --------- Signed-off-by: Zixin Zhou <zhouzixin@apache.org> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…2355) **Description** Preserve Anthropic's `web_search_20260209` tool when an OpenAI-compatible chat-completions request is routed to an Anthropic backend. The translator now retains the tool name and maps it to Anthropic's built-in web search parameter instead of silently dropping it. Fixes theagentrouter#2124 **Testing** - go test ./internal/apischema/openai ./internal/translator ./internal/extproc -count=1 Signed-off-by: Yang Haoran <97868579@qq.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
**Description** Make the OpenAI Responses and Speech translators recognize complete SSE events framed with either LF or CRLF line endings while inspecting streamed responses for observability data. **Root Cause** Both parsers buffered events by searching only for `\n\n`. Standards-compliant backends using `\r\n\r\n` therefore left complete events buffered and prevented token usage, response model, or tracing data from being extracted. Parsing of the optional space after `data:` is already handled by the shared field helper on current main and is not changed here. **Impact** Client response bytes remain unchanged. The bug only affects Gateway-side inspection of CRLF-framed streams. **Fix** Use a shared event-boundary helper that preserves partial input and accepts both LF and CRLF framing in the Responses and Speech parsers. Fixes theagentrouter#2463 **Testing** - `go test ./internal/translator -count=1` --------- Signed-off-by: 0YHR0 <97868579@qq.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…entrouter#2741) **Description** When CEL expressions cannot be properly evaluated, it is safer to fail closed than to silently bypass Deny rules. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
**Description** Adds `excludeFromModelsEndpoint` to `AIGatewayRoute` rules. Models matched by a rule with this option remain routeable and support `modelNameOverride`, but are excluded from the OpenAI-compatible `/v1/models` response. This supports backwards-compatible aliases and aliases only intended for selected clients. **Related Issues/PRs (if applicable)** Fixes theagentrouter#2445 **Special notes for reviewers (if applicable)** Tested manually. ⚒️ with ❤️ by @siemens Signed-off-by: Linus Schlumberger <linus.schlumberger@siemens.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
…heagentrouter#2742) **Description** If CEL expressions are configured to extract information from the JWT token, then the route should declare an OAuth security policy so that the token can be previously validated. Otherwise, that would open the door to parse unvalidated tokens in CEL expressions. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
…ss-namespce secrets (theagentrouter#2746) **Description** ReferenceGrants should be required when referencing secrets in other namespaces from BackendSecurityPolicies. Note this may be a breaking change that is worth mentioning in the release notes. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** N/A Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
…theagentrouter#2740) **Description** Update the MCP implementation to always check the request subject when resuming MCP sessions. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
…agentrouter#2755) **Description** The chart shipped "default-insecure-seed" as the MCP session encryption seed. Anyone who knows that public value can decrypt an observed Mcp-Session-Id, swap in their own subject, and encrypt it again, which bypasses the subject check added in theagentrouter#2740. This makes the chart generate a random 64-character seed on install when controller.mcp.sessionEncryption.seed is empty (the new default). The seed is stored in a Secret named "<controller fullname>-mcp-session-encryption" and reused on upgrades through lookup, the same pattern the chart already uses for the webhook certificate. The controller reads the seed from that Secret through the MCP_SESSION_ENCRYPTION_SEED environment variable, so it no longer appears as a plain value in the Deployment args. - An explicit seed value still works (it is written to the same Secret). - A new existingSecret value lets users point at their own Secret with a "seed" key. This matters for tools that render with "helm template" (GitOps), where lookup returns nothing and a new seed would be generated on every render. - helm-test gains checks for the generated Secret, the env-based arg, the absence of the default seed, an explicit seed, and existingSecret. **Related Issues/PRs (if applicable)** Follow-up to theagentrouter#2740 (seed part split out as agreed in the advisory discussion). **Special notes for reviewers (if applicable)** Upgrading from a chart version that used the default seed changes the seed and ends existing MCP sessions once. The values.yaml comment documents setting fallback.seed to "default-insecure-seed" for a smooth rollover. The controller and extproc flag defaults are unchanged in this PR (extproc already warns on the default); refusing to start on the default seed could be a separate change if you want it. Signed-off-by: aeonframework <aeonframework@users.noreply.github.com> Co-authored-by: aeonframework <aeonframework@users.noreply.github.com>
theagentrouter#2758) **Description** This adds four anthropic-beta flags to the AWS Anthropic allowlist (`awsBedrockSupportedAnthropicBetas`): `afk-mode-2026-01-31`, `dangerous-tool-use-2026-09-03`, `thinking-binding-controls-2026-08-01` and `thinking-token-count-2026-05-13`. Bedrock accepts all four, but the allowlist strips them from the `anthropic-beta` header before the translator writes the `anthropic_beta` body field, so requests that depend on them fail with a 400: - `dangerous-tool-use-2026-09-03` gates the `safeguards` body field that Claude Code sends in auto mode. Without the flag, Bedrock rejects the request with `safeguards: Extra inputs are not permitted`. - `afk-mode-2026-01-31` is the other flag Claude Code sends in auto mode. - `thinking-binding-controls-2026-08-01` gates `thinking.block_binding`, which clients such as OpenCode now send for newer Claude models. Without the flag, Bedrock rejects the request with `thinking.adaptive.block_binding: Extra inputs are not permitted`. - `thinking-token-count-2026-05-13` is listed in the doc comment as probed and rejected, which was true when theagentrouter#2512 added the allowlist. Bedrock accepts it now, so it moves to the allowlist. Claude Code sends it on every request. Unit tests cover the four new flags, on their own and together with an existing allowlisted flag. The header value filter test case that used `thinking-token-count-2026-05-13` as its example of a flag already stripped by the allowlist now uses `prompt-caching-2024-07-31`, which Bedrock still rejects. I used Claude Code, an AI coding assistant, to help write this change and run the Bedrock probes below. I have reviewed the change and the probe results and take ownership of them. Verification: each flag was sent directly to Bedrock with `aws bedrock-runtime invoke-model` on 2026-09-29, in eu-west-1 against EU cross-region inference profiles, with this body: ```json {"anthropic_version":"bedrock-2023-05-31","max_tokens":16,"messages":[{"role":"user","content":"hi"}],"anthropic_beta":["<flag>"]} ``` A made-up flag sent the same way is the negative control. Bedrock rejects it on every model with ``Unexpected value(s) `bogus-flag-2099-01-01` for the `anthropic-beta` header.``, so it does validate `anthropic_beta` and the 200s are meaningful. | Model | `afk-mode-2026-01-31` | `dangerous-tool-use-2026-09-03` | `thinking-binding-controls-2026-08-01` | `thinking-token-count-2026-05-13` | `bogus-flag-2099-01-01` | | --- | --- | --- | --- | --- | --- | | `eu.anthropic.claude-opus-5-5` | 200 | 200 | 200 | 200 | ValidationException | | `eu.anthropic.claude-opus-5` | 200 | 200 | 200 | 200 | ValidationException | | `eu.anthropic.claude-sonnet-5` | 200 | 200 | 200 | 200 | ValidationException | | `eu.anthropic.claude-opus-4-8` | 200 | 200 | 200 | 200 | ValidationException | | `eu.anthropic.claude-sonnet-4-6` | 200 | 200 | 200 | 200 | ValidationException | | `eu.anthropic.claude-haiku-4-5-20251001-v1:0` | 200 | 200 | 200 | 200 | ValidationException | The paired body fields were checked on `eu.anthropic.claude-opus-5-5` with the same request: - `"safeguards": [{"type": "dangerous_tool_use", "classifier_context": {}}]` returns 200 with a `safeguard_results` entry when `dangerous-tool-use-2026-09-03` is in `anthropic_beta`, and ValidationException `safeguards: Extra inputs are not permitted` without it. - `"thinking": {"type": "adaptive", "block_binding": {"prefix_mismatch_behavior": "drop_block"}}` returns 200 when `thinking-binding-controls-2026-08-01` is in `anthropic_beta`, and ValidationException `thinking.adaptive.block_binding: Extra inputs are not permitted` without it. Through an AWSAnthropic backend, the same requests fail when the flag is sent only in the `anthropic-beta` header, because the allowlist strips it. **Related Issues/PRs (if applicable)** Related PR: theagentrouter#2512 Related PR: theagentrouter#2420 **Special notes for reviewers (if applicable)** The per-backend `headerValueFilters` from theagentrouter#2420 can't be used to enable these flags instead. They run after the static allowlist and can only remove flags. The probes were only run in eu-west-1. 🤖 Generated with Claude Code (https://claude.com/claude-code) Signed-off-by: Patricio Danos <danos@zorgdomein.nl> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
**Description** Adds Bitdeer AI to the homepage provider strip and supported-provider documentation, with a connection guide and Kubernetes configuration example. Uses the icon-only SVG supplied in the review thread, with its canvas trimmed to fit the other homepage provider icons. Fixes Markdown formatting in the provider table and connection guide so it matches the precommit formatter. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** Documentation and static assets only. Validation passed: production site build (`npm run build`), Prettier checks for the changed Markdown files, SVG parsing, `git diff --check`, and visual inspection of the homepage icon in light and dark modes. The build reports existing broken-anchor warnings in API reference pages. Full Go precommit and unit-test runs could not complete locally because Go tools and test executables were terminated with `signal: killed`; the unit-test run was stopped after repeated terminations. GitHub Actions will validate the updated branch. AI assistance: Codex helped apply the supplied icon, correct formatting, and verify the changes. --------- Signed-off-by: tinatsou <tinatsou6@gmail.com> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>
**Description** Add KubeCon India 2026 talk. Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>
…clude_usage (theagentrouter#2747) **Description** Make sure we don't leave duplicate keys in the JSON payload after injecting the `stream_options.include_usage`. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io> Co-authored-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>
The context cache previously kept its resolved cachedContents names in a per-replica in-memory map, so two gateway replicas seeing the same prefix each created their own Google cache. Replace the map with a CacheStore interface and a Redis-backed implementation that all replicas share. This changes the AIServiceBackend API in both v1alpha1 and v1beta1: gcpContextCaching is renamed to contextCache, its enabled field is removed, and a required url field gives the Redis address, either as host:port or as a redis:// URL. Setting contextCache now enables caching; leaving it unset disables it and cache_control markers are ignored. The filterapi config and controller wiring follow the same shape. Creation is arbitrated with SET NX PX: the replica that claims a key creates the cache while the others poll for its result, so a cold prefix arriving at N replicas produces one create rather than N. A waiter that times out resolves against Google itself, which degrades a stuck leader into an extra list rather than a duplicate create. Store failures are never fatal to a request. An unreachable Redis, or a url that does not parse, is treated as a cache miss and the request is served uncached, since caching is an optimization. Google-side failures keep failing fast, because those are actionable by the user: a create rejected for being below the model's minimum token count means the cache_control markers are misplaced, and silently serving the request uncached would hide a cost increase. Resolvers are now carried across config reloads, keyed by backend name and Redis URL, so a filterapi update does not tear down and rebuild each backend's connection pool. Adds github.com/redis/go-redis/v9, and github.com/alicebob/miniredis/v2 for tests. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Creation was arbitrated with a Redis SET NX PX lock: one replica created the cache while the others polled for up to 5s inside ProcessRequestHeaders. The lock was not a guarantee, since a waiter that timed out went on to list and create anyway. It cost a blocking wait on the request path, up to ~100 Redis round trips per waiting request, and timeout constants that were never measured. Resolution on a store miss is now: create the cache, publish it to the store, and return it. There is no waiting on other replicas and no cachedContents.list call. Replicas that miss the same cold prefix at the same moment each create a cache and each keep their own. This is an accepted cost. Once any replica publishes a name, all replicas reuse it. Every create now reports Created with its real TokenCount. Previously the post-create re-list could switch to another replica's cache and report Created=false, which dropped a billed write from the cache-write token metrics. Duplicate creates now show up there. A create response without an expireTime is now an error. Its store TTL would be negative, so the write was skipped silently and every later request for that prefix created again. A leftover lock sentinel from an older replica fails to decode and is treated as a miss, so rolling deploys need no migration. The list code is removed from the request path. A background reconciler that repairs drift between Redis and GCP will follow separately. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
The request path no longer lists cachedContents, so a Google cache that the store has forgotten (after a Redis flush or eviction, a dropped write, or a cache created before Redis was configured) is not found again. Add a reconciler that repairs this off the request path. Each round lists cachedContents for one project and region and writes every gateway-created entry to the store with SET NX and a TTL pinned to its expireTime. SET NX means it only fills gaps and never overwrites a name that a request just published. Entries that are not keyed by a cache key were created by some other client and are skipped, as are entries near or past expiry and entries whose expireTime does not parse. The list API has no server-side filter, so the walk reads every entry. It requests the maximum page size, follows nextPageToken, treats a repeated token as the end, and stops after a fixed number of pages. A per-project/region gate, taken with SET NX and the interval as TTL, limits listing to one replica per interval across the fleet without leader election. A missed round has no effect on correctness. The reconciler never serves requests and never returns an error; a failed round is logged and the next one retries. Stores that cannot back it, such as the no-op store, have no reconciler. Nothing starts the reconciler yet; that comes with the resolver lifecycle change. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
The shared store and the reconcile loop do not depend on any provider API, but they lived in gcpcache. Move them into a provider-neutral internal/contextcache package so a second provider only has to supply its own API calls. contextcache now holds Store, Entry, NoopStore, the Redis store, and a ReconcileStore interface with SetNX and AcquireGate for stores that can back a reconciler. It also holds Reconciler, which runs the ticker, the per-scope gate, and the SET NX writes, and StaleThreshold. A provider plugs in through Source: GateKey names the scope's gate, and List reports every gateway-created cache in the scope with its cache key. gcpcache keeps the resolver, the cache key, createCache, and a Source that walks cachedContents.list with pagination and the page limit. That source now drops caches created by other clients, and caches whose expireTime does not parse, before the reconciler sees them, so they no longer count as seen or skipped in round stats. Redis keys, the stored value format, and the gcpcache:reconcile: gate prefix are unchanged, so old and new replicas interoperate during a rolling deploy. Store errors now start with "contextcache:" instead of "gcpcache:". Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
The reconciler existed but nothing ran it. Tie its lifetime to the resolver that owns it. The resolver gains Start and Close behind a new Background interface, so CacheResolver and the request path are unchanged. Start builds the reconciler for the backend's project and region and runs it under a context the resolver owns. It cannot use the LoadConfig context: the config watcher cancels that when each reload finishes, which would stop the reconciler within seconds. Start does nothing if the resolver is already running, has been closed, or has a store that cannot reconcile. Close cancels the context and waits for the goroutine to exit, and is safe to call more than once. LoadConfig calls Start with the backend's GCPAuthHandler. Resolvers carried across a reload are already running and are not restarted. After a reload, LoadConfig closes every resolver that was not carried over, because its backend was removed, lost its cache config, or was repointed at another Redis. Without this, each such reload would leak a goroutine. Server.Close closes all resolvers, and extproc calls it on shutdown after the gRPC server stops. Rounds run every 60s, starting as soon as the reconciler starts. A round against an unreachable Redis fails and is logged at warn level. Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
…filterconfig (theagentrouter#2750) **Description** GatewayController resolved cross-namespace AIServiceBackend/InferencePool backends (and their BackendSecurityPolicy credentials) into the shared extproc filter-config Secret without checking a ReferenceGrant, even though AIGatewayRouteController already enforces one for HTTPRoute generation. Gate both backend kinds on the same check. Also reconcile BackendSecurityPolicy on grant changes. ReferenceGrantController only requeued AIGatewayRoutes when a ReferenceGrant changed, so a BackendSecurityPolicy left NotAccepted for lacking a grant to its Secret wouldn't re-reconcile until something else touched it. Requeue affected BackendSecurityPolicies too. **Related Issues/PRs (if applicable)** N/A **Special notes for reviewers (if applicable)** N/A --------- Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
…er#2347) **Description** - explain the intent split between the platform's upstream provider/model budget and each client's route allowance - compare configuration, scope, token accounting, client budgets, enforcement semantics, and time windows - document the dedicated AI Gateway quota rate limit service, Redis dependency, and fail-open default - clarify the current one-day `QuotaPolicy` limit and the longer windows available through usage-based rate limiting - keep the cross-reference in the usage-based rate limiting guide consistent ## Why The existing comparison incorrectly described usage-based rate limiting as request velocity even though it can charge response token metadata against cumulative limits. Both features can count tokens and return 429, but they answer different operational questions and enforce matched limits differently. The updated guide frames `QuotaPolicy` as a platform budget attached to an `AIServiceBackend`, and usage-based rate limiting as a consumer allowance attached to route traffic. ## User impact Users can choose the policy that matches who owns the budget, where it applies, how multiple matched limits are enforced, and which time windows are required. Operators also get explicit deployment and failure-mode prerequisites for quota enforcement. ## Validation - `git diff --check` - `npm ci` - `npm run build` - repository precommit checks on Ubuntu and macOS ## AI assistance This documentation update was prepared with OpenAI Codex assistance. The statements were checked against the `QuotaPolicy` API and implementation in this repository and the Envoy Gateway API version used by the project. Fixes theagentrouter#2340 --------- Signed-off-by: Yang Haoran <97868579@qq.com> Signed-off-by: 0YHR0 <97868579@qq.com> Signed-off-by: Erica Hughberg <erica.sundberg.90@gmail.com> Co-authored-by: Erica Hughberg <erica.sundberg.90@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…heagentrouter#2767) **Description** The extproc's memory grows significantly when metrics carry high-cardinality attributes **for Inference requests with high context length**. We expected the increase in memory because of cardinality but the correlation with context length was unexpected. The cause is string aliasing, not the metric data itself. `internal/json` uses sonic without `CopyString`, so decoded fields such as `req.Model` are substrings of a copy of the whole request body. That model string is passed to `SetOriginalModel` / `SetRequestModel`, ends up in the metric `attribute.Set`, and cumulative histograms keep every attribute set for the process lifetime. So the first request that creates each new attribute combination keeps its entire body in memory for good. The same applies to `SetResponseModel` and response bodies. Heap profiles from the load test support this. About 76% of live heap was `sonic.frozenConfig.Unmarshal` allocations from `ChatCompletionsEndpointSpec.ParseBody`. **Change** This change: - Clones the model name once in `ProcessRequestBody`, right after `ParseBody` /`ParseMultipartBody`. This covers everything downstream of it: the `x-ai-eg-model` request header, the router's `originalModel`, and the request metrics. - Clones the model name in the `metricsImpl` model setters, so metric attributes never keep a body alive whatever the caller passes. `SetResponseModel` runs on every response body chunk for streaming responses, so it clones only when the value changes. *The first time a request creates a new label combination, the OTel SDK stores that combination for the life of the process, as it always has. The value it keeps now is the cloned model name, a few bytes like loadtest-uep1. Before this change it was the whole request body. Later requests with the same combination match the stored entry, so their clones are simply freed.* A local check shows the difference: 200 attribute sets built from `ParseBody` output on a ~300 KiB body retained 59.7 MiB without the clone and 0.1 MiB with it. **Related Issues/PRs (if applicable)** **Special notes for reviewers (if applicable)** - New tests, both of which fail without this change: - `Test_chatCompletionProcessorRouterFilter_ProcessRequestBody/original_model_does_not_alias_the_request_body` checks that `originalModel` and the `x-ai-eg-model` header don't share memory with the parsed request body. - `TestSetModel_DoesNotRetainSourceBuffer` checks the same for the three metrics setters. - The clones in `SetOriginalModel` / `SetRequestModel` are redundant with the clone after `ParseBody` on the current call path. They're kept as a safety net for future callers. --------- Signed-off-by: Aishwarya <aishraimule@gmail.com>
…he backend schema (theagentrouter#2765) **Description** When a request's endpoint has no translator for the selected backend's API schema (for example `/v1/responses` to a `GCPVertexAI` or `GCPAnthropic` backend), `SetBackend` fails the ext_proc stream and Envoy answers with HTTP 500. The combination is known to be unsupported from configuration alone and the request never reaches an upstream, so this is a client/configuration error, not an internal failure. Returning 5xx lets any client burn an operator's 5xx-based availability SLO. This change answers it with a 422 instead: - The unsupported API schema errors returned by every `GetTranslator` are wrapped with `internalapi.ErrInvalidRequestBody`, whose doc already covers requests that don't "match the expected schema for the target API". The error text is unchanged. - In `upstreamProcessor.SetBackend`, a user-facing `GetTranslator` error is kept on the processor instead of failing the stream. `ProcessRequestHeaders` then returns it through the existing `respondLocally` path, so metrics, the span and the `localReplyEmitted` handling are the same as for the other 422s. `upstreamFilter` still stays nil, so the theagentrouter#1941 fix is unaffected. - The client gets `{"type":"error","error":{"type":"UnprocessableEntity","code":"422","message":"invalid request body: unsupported API schema: backend={GCPVertexAI ...}"}}`. The backend, namespace and route names appear only in the logs. - Any other `GetTranslator` error still fails the stream as before. **Related Issues/PRs (if applicable)** Fixes theagentrouter#2764 Related: theagentrouter#1941, theagentrouter#1621, theagentrouter#2197, theagentrouter#2752 **Special notes for reviewers (if applicable)** - The status is 422, reusing `ErrInvalidRequestBody` rather than adding a new sentinel, so it follows the existing user-facing error framework from theagentrouter#1621. Happy to switch to a dedicated sentinel and/or a 400/404/501 if you prefer. The message prefix "invalid request body:" is the part that reads least naturally. - Behaviour change to consider: a route with fallback backends and a retry-on-5xx policy may currently retry past an unsupported backend on the 500. A 422 will not. That matches the existing 422 path for translator request errors, but I have not verified how Envoy treats the upstream-filter 500 for retries. - Verified locally with `go test ./internal/...`, `golangci-lint fmt` and `golangci-lint run` on the changed packages. I did not run `make test-data-plane`. - Generative AI (Claude Code) was used to help write this change; I have reviewed and understand the code. --------- Signed-off-by: Ryuzo Yamamoto <ryuzo.yamamoto@gmail.com>
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
sivanantha321
force-pushed
the
gcp-context-cache
branch
from
October 2, 2026 16:35
44332c5 to
599fdf8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issues/PRs (if applicable)
Special notes for reviewers (if applicable)