From 9416c8a20cd578be26d6961b2970bb1a6c6cd234 Mon Sep 17 00:00:00 2001 From: Erica Hughberg Date: Sun, 4 Oct 2026 21:57:59 -0400 Subject: [PATCH 1/5] docs: add v1.2 release notes Add the v1.2 release notes, the first release under the Agent Router name: site data and page, the plain-markdown copy for the GitHub release body, and the release-notes index and navigation. Also fix release-notes pages in dark mode (hard-coded white backgrounds), render the v1.1 breaking change and bug fixes that were in its data but not on its page, and update the release-notes command for the Agent Router name and the latest Envoy Gateway release. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Erica Hughberg --- .claude/commands/release-notes | 143 +++++---- release-notes/v1.2.0.md | 233 ++++++++++++++ .../ReleaseNotes/ReleaseNotes.module.css | 12 +- site/src/data/releases/index.json | 4 +- site/src/data/releases/v1.1.json | 5 +- site/src/data/releases/v1.2.json | 285 ++++++++++++++++++ site/src/pages/release-notes/index.mdx | 18 +- site/src/pages/release-notes/v1.1.mdx | 13 +- site/src/pages/release-notes/v1.2.mdx | 229 ++++++++++++++ 9 files changed, 870 insertions(+), 72 deletions(-) create mode 100644 release-notes/v1.2.0.md create mode 100644 site/src/data/releases/v1.2.json create mode 100644 site/src/pages/release-notes/v1.2.mdx diff --git a/.claude/commands/release-notes b/.claude/commands/release-notes index 686ca4213b..c4a2a9f3a4 100644 --- a/.claude/commands/release-notes +++ b/.claude/commands/release-notes @@ -1,18 +1,31 @@ --- -description: Generate release notes for Envoy AI Gateway following the established style and structure. +description: Generate release notes for Agent Router (formerly Envoy AI Gateway) following the established style and structure. --- -Generate release notes for Envoy AI Gateway following the established style and structure. +> Mirrored in `.claude/commands/release-notes` (git-tracked) and `.cursor/commands/generate-release-notes.md` — keep all three in sync when editing. + +Generate release notes for Agent Router following the established style and structure. ## Instructions -You are generating release notes for the Envoy AI Gateway project. Follow this workflow: +You are generating release notes for the Agent Router project. Follow this workflow: + +### Project Naming + +The project was renamed from Envoy AI Gateway to **Agent Router** in September 2026 (an Agentic AI Foundation project). + +- **Use "Agent Router"** for the project in every title, subtitle, overview, and prose sentence. Do not write "Envoy AI Gateway" except in a deliberate "formerly Envoy AI Gateway" mention. +- **Site:** `https://theagentrouter.ai` (release notes at `https://theagentrouter.ai/release-notes/vX.Y`). `aigateway.envoyproxy.io` only redirects; never link to it. +- **Repository:** `https://github.com/theagentrouter/agent-router`. +- **Keep these identifiers unchanged.** They were not renamed, and users type them literally: the `aigateway.envoyproxy.io` API group, CRD kinds (`AIGatewayRoute`, `AIServiceBackend`, `BackendSecurityPolicy`, `MCPRoute`, …), the `aigw` CLI, the `envoy-ai-gateway-system` namespace, Helm charts (`oci://docker.io/envoyproxy/ai-gateway-helm`, `ai-gateway-crds-helm`), container images (`docker.io/envoyproxy/ai-gateway-*`), `AI_GATEWAY_*` environment variables, and the Go module path. +- **Envoy and Envoy Gateway keep their names.** Agent Router is built on them. ### Step 1: Gather Context 1. **Identify the previous release tag:** ```bash + git fetch upstream --tags git describe --tags --abbrev=0 git tag --sort=-v:refname | head -10 ``` @@ -31,8 +44,24 @@ You are generating release notes for the Envoy AI Gateway project. Follow this w ``` 4. **Review the existing release notes for style reference:** - - Read `site/src/data/releases/v0.4.json` and `site/src/data/releases/v0.3.json` - - Read `site/src/pages/release-notes/v0.4.mdx` and `site/src/pages/release-notes/v0.3.mdx` + - Read the JSON for the two most recent series in `site/src/data/releases/` (for example `v1.1.json` and `v1.0.json`) + - Read the matching pages in `site/src/pages/release-notes/` and the latest `release-notes/vX.Y.Z.md` + - `site/src/pages/release-notes/v0.7.mdx` is the reference for rendering `bugFixes` and `breakingChanges` on the main release + +5. **Check the latest Envoy Gateway release.** The release notes always document the **latest Envoy Gateway patch release**. If `go.mod` is behind, assume the bump lands before the release, use the latest version in the notes, and tell the user the bump is still needed: + + ```bash + grep -E "envoyproxy/gateway |sigs.k8s.io/gateway-api" go.mod + gh release list -R envoyproxy/gateway --limit 8 + # Envoy Proxy version shipped by that EG release + gh api "repos/envoyproxy/gateway/contents/api/v1alpha1/shared_types.go?ref=" -q .content | base64 -d | grep DefaultEnvoyProxyImage + # Gateway API version used by that EG release + gh api "repos/envoyproxy/gateway/contents/go.mod?ref=" -q .content | base64 -d | grep "sigs.k8s.io/gateway-api " + # Breaking changes or security flags in that EG release that operators must know about + gh api "repos/envoyproxy/gateway/contents/release-notes/.yaml?ref=" -q .content | base64 -d + ``` + + Take the Envoy Proxy and Gateway API versions from that EG release, not from `.envoy-version`. If the EG release has its own breaking changes, mention them in Upgrade Guidance. ### Step 2: Verify Features Against Code @@ -145,12 +174,14 @@ Verify actual dependency versions from source files: grep -E "^go " go.mod # Key dependencies -grep -E "envoyproxy/gateway|sigs.k8s.io/gateway-api" go.mod +grep -E "envoyproxy/gateway|sigs.k8s.io/gateway-api|modelcontextprotocol/go-sdk" go.mod -# Envoy version +# Envoy version pinned for the standalone aigw CLI (not necessarily what EG ships) cat .envoy-version ``` +For Envoy Gateway, Envoy Proxy, and Gateway API, document the **latest EG patch release** (see Step 1.5), not just what `go.mod` pins. + #### Test Coverage Review test files to understand feature scope and edge cases: @@ -197,6 +228,10 @@ List deprecated features with: Document fixes for user-facing bugs +#### Security Hardening + +Security fixes that change behavior (stricter validation, newly required ReferenceGrants, fail-closed defaults) belong in `breakingChanges` with a migration step. Fixes that don't change behavior belong in `bugFixes`. Describe the new behavior and what operators must do, not how the old behavior could be exploited. Reference GHSA IDs only after the advisory is published. + #### Breaking Changes (`breakingChanges`) List changes that require user action: @@ -223,7 +258,7 @@ Create `site/src/data/releases/vX.Y.json` following this exact schema: { "series": { "version": "vX.Y", - "title": "Envoy AI Gateway vX.Y.x", + "title": "Agent Router vX.Y.x", "subtitle": "Release introducing [key features summary].", "badge": "Latest", "badgeType": "milestone" @@ -276,12 +311,12 @@ Create `site/src/data/releases/vX.Y.json` following this exact schema: "description": "Updated to Go X.Y.Z for improved performance and security." }, { - "title": "Envoy Gateway vX.Y", - "description": "Built on Envoy Gateway vX.Y for proven data plane capabilities." + "title": "Envoy Gateway vX.Y.Z", + "description": "Built on Envoy Gateway vX.Y.Z (latest patch release) for proven data plane capabilities." }, { "title": "Envoy vX.Y", - "description": "Leveraging Envoy Proxy's battle-tested networking capabilities." + "description": "Leveraging Envoy Proxy vX.Y.Z, as shipped by Envoy Gateway vX.Y.Z." }, { "title": "Gateway API vX.Y.Z", @@ -299,27 +334,29 @@ Create `site/src/data/releases/vX.Y.json` following this exact schema: ### Step 5: Generate the MDX Page -Create `site/src/pages/release-notes/vX.Y.mdx` using this template: +Create `site/src/pages/release-notes/vX.Y.mdx` using this template. Keep the established section order: New Features → API Updates → Deprecations → Breaking Changes → Bug Fixes → Upgrade Guidance → Dependencies → Patch Releases → Acknowledgements → What's Next. Every release since v0.2 opens with New Features. Don't move Breaking Changes to the top; if the release has breaking changes, say so in the last sentence of the `overview` instead. Bug Fixes render on the main release, not only on patch releases. + +Write section headings as plain markdown at the top level, never inside a JSX expression or fragment. MDX does not parse `## Heading` inside `{cond && (<>…)}`: it renders the literal text `## ⚠️ Breaking Changes` and leaves the heading out of the table of contents. If a release has no breaking changes or no deprecations, delete that heading and its `` from the page instead of wrapping them in a condition. ```mdx --- -title: Envoy AI Gateway vX.Y.x Release Series +title: Agent Router vX.Y.x Release Series description: [Brief description of major features] toc_min_heading_level: 2 toc_max_heading_level: 4 --- -import Link from "@docusaurus/Link"; +import Link from '@docusaurus/Link'; import { ReleaseSeriesLayout, ReleaseHeader, FeatureSectionCard, PatchRelease, ItemList, - Dependencies, -} from "../../components/ReleaseNotes"; -import releaseData from "../../data/releases/vX.Y.json"; -import React from "react"; + Dependencies +} from '../../components/ReleaseNotes'; +import releaseData from '../../data/releases/vX.Y.json'; +import React from 'react'; export const { series, releases, navigation } = releaseData; export const mainRelease = releases[0]; @@ -348,39 +385,27 @@ export const patchReleases = releases.slice(1).reverse(); ## ✨ New Features {mainRelease.features.map((featureSection, index) => ( - -{" "} - - + ))} ## 🔗 API Updates {mainRelease.apiChanges.length > 0 && ( - -{" "} - - + )} -{mainRelease.deprecations?.length > 0 && ( +### Deprecations -{" "} + -<> - ### Deprecations - - -)} +## ⚠️ Breaking Changes -{mainRelease.breakingChanges?.length > 0 && ( + -{" "} +## 🐛 Bug Fixes -<> - ## ⚠️ Breaking Changes - - +{mainRelease.bugFixes?.length > 0 && ( + )} ## 📖 Upgrade Guidance @@ -394,17 +419,16 @@ export const patchReleases = releases.slice(1).reverse(); ## ⏩ Patch Releases {patchReleases.map((release, index) => ( - - + ))} ## 🙏 Acknowledgements @@ -425,8 +449,8 @@ Create `release-notes/vX.Y.Z.md` (at the repo root, sibling to `RELEASES.md`). T Rules: - **No MDX.** No imports, no React components, no `` HTML tags — use markdown backticks instead. -- **Same sections, same order** as the MDX page, so the GitHub release body and the site stay aligned: short overview paragraph → Breaking Changes → New Features (with the same feature-group subheadings) → API Updates → Bug Fixes → Upgrade Guidance (full migration steps and code blocks) → Dependency Versions → Acknowledgements → What's Next. -- **Self-contained.** A reader pasting this into GitHub should see the full release without needing to follow any link, but include a single link near the top to the rendered version on the site (`https://aigateway.envoyproxy.io/release-notes/vX.Y`). +- **Same sections, same order** as the MDX page, so the GitHub release body and the site stay aligned: short overview paragraph → New Features (with the same feature-group subheadings) → API Updates → Deprecations → Breaking Changes → Bug Fixes → Upgrade Guidance (full migration steps and code blocks) → Dependency Versions → Acknowledgements → What's Next. Older copies (`v0.6.0.md`, `v1.1.0.md`) put Breaking Changes first; don't copy that. +- **Self-contained.** A reader pasting this into GitHub should see the full release without needing to follow any link, but include a single link near the top to the rendered version on the site (`https://theagentrouter.ai/release-notes/vX.Y`). Title the file `# Agent Router vX.Y.Z`. - **Filename includes the patch version** (`v0.6.0.md`, not `v0.6.md`) so future patch releases get their own files. ### Step 6: Update Navigation @@ -440,7 +464,9 @@ Update the previous release's JSON to add the `next` navigation: } ``` -Update `site/src/data/releases/index.json` with the new `lastUpdated` date. +In the same previous-series JSON, change `series.badge` from `"Latest"` to `"Stable"` (keep its `badgeType`). Only the newest series carries the `Latest` badge. + +Update `site/src/data/releases/index.json` with the new `lastUpdated` date. Its `metadata.description` should read `Release Notes for Agent Router`. ### Step 6b: Update the Release Notes Index Page @@ -490,6 +516,11 @@ Change the previous "Latest" release card: - Set `featured={false}` - Update the `date` range to end at the new release date (e.g., "November 7, 2025 - January 16, 2026") +- Its `badge` now reads `Stable`, because Step 6 changed the badge in its JSON + +#### 5. Check the docs version cut + +If the release also cuts a docs version, `lastVersion` in `site/docusaurus.config.ts` and the homepage "vX.Y Release now available" card in `site/src/components/HomepageFeatures/index.tsx` must point at the new version. Otherwise the site keeps showing the previous release. Flag it to the user if the docs cut hasn't happened yet. ### Step 7: Verify Each Item Against Code @@ -601,7 +632,7 @@ Ask these questions: - [ ] Verified CLI changes against `cmd/aigw/` source files - [ ] Verified authentication changes against `internal/backendauth/` - [ ] Cross-referenced features with test files for accuracy -- [ ] Verified dependency versions from `go.mod` and `.envoy-version` +- [ ] Verified dependency versions from `go.mod`, and checked the latest Envoy Gateway release (Step 1.5) #### Content Generation @@ -610,6 +641,10 @@ Ask these questions: - [ ] Created MDX page file - [ ] Updated previous release navigation - [ ] Updated `site/src/data/releases/index.json` lastUpdated date +- [ ] Previous series badge changed from `Latest` to `Stable` +- [ ] Created `release-notes/vX.Y.Z.md` plain-markdown copy +- [ ] Project called "Agent Router" everywhere; API group, CRD kinds, chart and image names left unchanged +- [ ] Envoy Gateway listed at its latest patch release, with Envoy and Gateway API versions taken from that EG release - [ ] Updated `site/src/pages/release-notes/index.mdx` with new import, allReleases array, and ReleaseCard #### Item-by-Item Verification (Step 7) diff --git a/release-notes/v1.2.0.md b/release-notes/v1.2.0.md new file mode 100644 index 0000000000..76c6196e40 --- /dev/null +++ b/release-notes/v1.2.0.md @@ -0,0 +1,233 @@ +# Agent Router v1.2.0 + +> Plain-markdown copy of the v1.2.0 release notes, suitable for pasting into the GitHub release body. The full rendered version lives at https://theagentrouter.ai/release-notes/v1.2. + +Agent Router v1.2.0 is the first release under the project's new name (formerly Envoy AI Gateway). Nothing you deploy is renamed: the aigateway.envoyproxy.io API group, the CRDs, the Helm charts, the container images, and the aigw CLI all stay the same. This release hardens MCP authentication and cross-namespace references. It adds bare tool names, caller-supplied credentials, and policy merging to MCPRoute, plus two new backend schemas (AWSOpenAI and TypeSafe). It also moves to Envoy Gateway v1.9.2, Gateway API v1.6, and Gateway API Inference Extension v1.6. Several security fixes change behavior, so read the breaking changes before you upgrade. + +## ✨ New Features + +### Agent Router + +- **Same project, new name** — Envoy AI Gateway is now Agent Router, an Agentic AI Foundation project with the same code, maintainers, and Apache 2.0 license. The website moved to [theagentrouter.ai](https://theagentrouter.ai) and the repository to [theagentrouter/agent-router](https://github.com/theagentrouter/agent-router); old links redirect. Your manifests, Helm values, and automation keep working unchanged. + +### MCP Gateway + +- **Bare tool and prompt names with `prefixMode: Never`** — Clients that hardcode tool names, such as interactive MCP Apps, no longer have to see the `__` prefix. Set `prefixMode: Never` on the route or on individual `backendRefs`. A Never-mode backend must list its tools in `toolSelector.include`, and the controller marks the route NotAccepted if two Never-mode backends expose the same name. Prompts are exposed bare only when they are listed in the new `promptSelector.include`. Resources always keep the prefix. The default is still `Always`. +- **Caller-supplied credentials with `injectionPolicy: IfNotPresent`** — Let users bring their own upstream token, for example a personal GitHub PAT forwarded with `forwardHeaders`, and fall back to a shared service key when they don't. With `IfNotPresent`, the configured API key is injected only when the target header is missing. This applies to header injection only. +- **Inherit Gateway-level policies with `mergeType`** — The SecurityPolicy and BackendTrafficPolicy that Agent Router generates for an MCPRoute used to replace any Gateway- or listener-level policy. Set `securityPolicy.mergeType` or `backendTrafficPolicy.mergeType` to `StrategicMerge` or `JSONMerge` to keep a Gateway-wide rate limit or ext-auth policy in force alongside MCP OAuth. +- **OAuth for identity providers with non-standard metadata paths** — `oauth.authorizationServerMetadataUrl` points the controller at the exact RFC 8414 metadata document when the issuer URL doesn't lead to it. The controller fetches this document while reconciling, so it needs network access to the identity provider. If the fetch fails, reconciliation fails. +- **Groundwork for the next MCP spec** — The `initialize` response now negotiates the protocol version with the client and backends instead of always answering 2025-06-18. Merged list responses also carry `ttlMs` and `cacheScope` caching hints. Support for the stateless 2026-07-28 MCP specification is in progress and not yet enabled. + +### Providers & API Compatibility + +- **AWS OpenAI-compatible endpoint (`AWSOpenAI`)** — Send OpenAI-format `/v1/chat/completions` and `/v1/responses` requests to Bedrock Runtime's OpenAI-compatible `/openai/v1` endpoint without a translation hop. Requests are signed with SigV4 through an AWS `BackendSecurityPolicy`. The prefix defaults to `openai/v1`. Other endpoints return 422. +- **TypeSafe System One (`TypeSafe`)** — Route TypeSafe's Jev decision model through the gateway at `/typesafe/v1/systemone` with API-key auth. Request bodies pass through unchanged apart from a model name override, and token usage feeds the standard metrics and `llmRequestCosts`. Streaming is not supported. +- **Per-backend `anthropic-beta` filtering** — One `anthropic-beta` value that a provider rejects no longer fails the whole request. `AIServiceBackend.spec.headerValueFilters` drops listed values (`Denylist`, the default) or keeps only listed values (`Allowlist`). It applies to `/v1/messages` traffic to `GCPAnthropic` and `AWSAnthropic` backends. Bedrock also accepts four more beta flags, including `thinking-token-count-2026-05-13`. +- **Claude and Gemini request coverage** — Claude Opus 5 and 5.5 accept `reasoning_effort` on Vertex AI and Bedrock, and structured output (`response_format`) on Vertex AI. Chat-completion tools forward `eager_input_streaming` to Claude. Gemini on Vertex AI accepts vLLM-style `structured_outputs` (`json`, `regex`, `choice`). +- **OpenAI `cache_write_tokens`** — Prompt-cache writes reported as `cache_write_tokens` by OpenAI Chat Completions and the Responses API now count toward cache-creation token metrics and costs. Translated Claude and Bedrock responses report the same value. + +### Model Catalog & Observability + +- **Hide aliases from `/v1/models`** — Set `excludeFromModelsEndpoint: true` on an `AIGatewayRoute` rule to keep internal or backwards-compatible model aliases out of `/v1/models` and `/anthropic/v1/models`. Requests to those models still route normally. +- **Per-backend GenAI metrics** — All four `gen_ai.*` metrics now carry a `gen_ai.backend` attribute (`gen_ai_backend` in Prometheus) set to the backend's `namespace/name`. This lets you compare latency, time to first token, and token usage across providers serving the same model. + +### Inference & Operations + +- **Gateway API Inference Extension v1.6** — An `InferencePool` without `endpointPickerRef` is now reported as `Accepted=False` (`EndpointPickerRefMissing`) instead of breaking routing. `appProtocol: kubernetes.io/h2c` is honored for cleartext HTTP/2 model servers. +- **Safer controller rollouts** — The controller reports Ready only after its caches sync, so Envoy Gateway no longer calls the extension server with an empty view of your resources during a rollout. The chart adds a startup probe and a `controller.priorityClassName` value. + +## 🔗 API Updates + +- **`AIGatewayRouteRule.excludeFromModelsEndpoint`** — Optional boolean (default `false`) in v1alpha1 and v1beta1. +- **`AIServiceBackend.spec.headerValueFilters`** — Optional list (max 16, one per header) of `{name, mode, values}`. `mode` is `Denylist` (default) or `Allowlist`; `values` holds up to 64 entries. A CEL rule rejects the field unless the schema is `GCPAnthropic` or `AWSAnthropic`, and only `anthropic-beta` is honored today. v1beta1 only. +- **`VersionedAPISchema.name`: `AWSOpenAI` and `TypeSafe`** — Two new schema values. `TypeSafe` defaults its version to `v1`. +- **`GatewayConfig.spec.extProc.metadataForwardingNamespaces`** — Optional list (max 32) of dynamic metadata namespaces that Envoy forwards to the external processor. It is required for `credentialOverride.fromDynamicMetadata`. Under Envoy Gateway `mergeGateways`, every Gateway in the class shares the union of these namespaces. +- **`MCPRoute.spec.prefixMode` and `backendRefs[].prefixMode`** — `Always` or `Never`; unset means `Always`. The per-backend value takes precedence. v1beta1 only. +- **`MCPRoute.spec.backendRefs[].promptSelector`** — Filters prompts with `include`, `includeRegex`, `exclude`, or `excludeRegex` (max 32 each). Exclude rules win over include rules. v1beta1 only. +- **`MCPRoute.spec.securityPolicy.mergeType` and `spec.backendTrafficPolicy.mergeType`** — Envoy Gateway `MergeType`; `Replace` is rejected. Unset keeps the previous override behavior. +- **`MCPRoute.spec.backendRefs[].securityPolicy.apiKey.injectionPolicy`** — `Always` (default) or `IfNotPresent`. `IfNotPresent` cannot be combined with `queryParam`. v1beta1 only. +- **`MCPRoute.spec.securityPolicy.oauth.authorizationServerMetadataUrl`** — Optional URI, up to 1024 characters. +- **New `MCPRoute` validation for JWT-based CEL** — `securityPolicy.oauth` is now required when an authorization rule or `backendSelector` rule's `cel` expression references `auth.jwt`. See Breaking Changes. + +### Deprecations + +- **`cache_creation_input_tokens` in translated OpenAI-format responses** — Responses translated from Claude and Bedrock now report `cache_write_tokens` alongside the legacy `cache_creation_input_tokens` field. Read `cache_write_tokens` instead; the legacy field will be removed in v1.3.0. + +## ⚠️ Breaking Changes + +- **Random MCP session encryption seed** — The Helm chart no longer defaults to the well-known `default-insecure-seed`. It generates a random seed, stores it in the Secret `-mcp-session-encryption`, and reuses it on later upgrades. Upgrading therefore ends active MCP sessions, and clients must re-initialize. To keep sessions during the upgrade, set `controller.mcp.sessionEncryption.fallback.seed=default-insecure-seed` and remove it once clients have reconnected. If you render the chart with `helm template` (common with GitOps), set `controller.mcp.sessionEncryption.existingSecret` or `seed`. Otherwise every render generates a new seed. +- **MCP OAuth tokens must match `oauth.issuer`** — The generated JWT provider now checks the token's `iss` claim. Tokens with a missing or different issuer, including a trailing-slash mismatch, get 401. Make sure `securityPolicy.oauth.issuer` matches your identity provider's `iss` exactly. +- **JWT data in MCP CEL requires `securityPolicy.oauth`** — The gateway reads `request.auth.jwt` claims and scopes, and the session subject, only from a JWT that Envoy has verified. Without `securityPolicy.oauth`, claims are empty, and new CRD validation rejects authorization or `backendSelector` CEL that references `auth.jwt`. Existing routes that break this rule keep their stored spec, but every update is rejected. At runtime their claim lookups fail, and therefore deny, and their scope checks evaluate to false. Add `oauth` or remove the JWT references. MCP sessions are now bound to the verified subject; reusing a session ID as a different user returns 401. +- **MCP CEL evaluation errors now deny** — An authorization or `backendSelector` CEL expression that fails at runtime, for example by reading a header that isn't present, used to be skipped. It now denies: `tools/call` returns 403, the tool is hidden from `tools/list`, and the backend is left out of the session. Write expressions defensively, for example `"x-team" in request.headers && request.headers["x-team"] == "blocked"`. +- **MCP OAuth discovery honors `spec.headers`** — On an MCPRoute with both `headers` matches and `oauth`, the OAuth well-known endpoints now apply the same header matches. Clients that run OAuth discovery without those headers get 404. +- **Cross-namespace Secrets in `BackendSecurityPolicy` need a ReferenceGrant** — `azureCredentials.clientSecretRef`, `gcpCredentials.credentialsFile.secretRef`, and the OIDC `clientSecret` under AWS, Azure, and GCP `oidcExchangeToken` were read from other namespaces without any check. A missing grant now sets the policy to NotAccepted. Already-issued tokens keep working until they expire, so the failure may appear late. Create a ReferenceGrant in the Secret's namespace (see Upgrade Guidance). +- **Ungranted cross-namespace backends no longer reach the data plane** — An `AIGatewayRoute` `backendRef` to an `AIServiceBackend` or `InferencePool` in another namespace without a ReferenceGrant already marked the route NotAccepted, but its credentials were still written to the external processor's config. Such backends are now left out. Working setups already have the grant; this affects only routes that were already NotAccepted. +- **Per-tenant token limits are now enforced** — With the `QuotaPolicy` `Distinct` fix, tenants can start receiving 429 at the limits you configured. Review per-tenant limits before upgrading. +- **External processor `-configPath` removed** — The external processor reads only its config bundle (`-configBundlePath`), and the controller no longer writes the legacy `filter-config.yaml` Secret. Existing legacy Secrets are left in place and can be deleted. This matters only if you run the external processor yourself, or still have Envoy pods created before config bundles existed (v1.0.x era). Restart those pods before upgrading. + +## 🐛 Bug Fixes + +- **Credential rotation no longer serves stale tokens** — Fixes a v1.1.0 regression: the controller read backend credential Secrets from its informer cache, so a rotated credential could miss the filter config and upstream calls failed until the next reconcile. Secrets are read from the API server again. +- **Gateways with TCP or TLS listeners translate again** — A listener without an HTTP connection manager, such as a TCPRoute or TLS-passthrough listener, made the extension server fail translation for the whole Gateway. Those listeners are now skipped. +- **Controller crash on combined priority backends** — Adding a `backendRef` before its `AIServiceBackend` existed could panic and crash-loop the controller. The controller now logs the mismatch and continues. +- **Self-signed webhook survives `helm upgrade`** — With the default self-signed webhook certificate, an upgrade or Argo CD sync wiped the webhook's `caBundle`, so Envoy pods failed admission until the controller restarted. The chart now embeds the CA bundle. +- **`credentialOverride.fromDynamicMetadata` now receives metadata** — In v1.1.0 Envoy never forwarded the metadata namespace to the external processor. Requests silently used the configured credential, or failed with 401 when `fallbackToConfigured: false`. List the namespace in `GatewayConfig.spec.extProc.metadataForwardingNamespaces` (see Upgrade Guidance). +- **Token usage is always captured on streams** — Duplicate `stream_options` in a streaming request could stop `include_usage` from reaching the provider, which bypassed token accounting. Usage metadata is now also recorded when a client disconnects right after the final SSE frame. Responses API streams framed with CRLF now report usage, model, and tracing data. +- **Clear 422 errors instead of 500s** — Calling an endpoint that a backend's schema doesn't support (for example `/v1/responses` to Vertex AI Gemini) returns 422 Unprocessable Entity. Oversized JSON schemas sent to Vertex AI return 422 once they exceed 50,000 nodes. +- **Lower external processor memory** — Model-name strings used as metric labels no longer keep entire request and response bodies in memory, which was most noticeable with large-context requests. +- **Per-tenant token limits charge the right tenant** — With `QuotaPolicy` `Distinct` header selectors, the end-of-stream token cost went into one shared bucket instead of the caller's own. See Breaking Changes for the effect on tenants. +- **Claude translation fixes** — Structured outputs keep the property order you declared. Streamed extended-thinking text is no longer empty. The `web_search_20260209` tool on `/v1/chat/completions` is now forwarded to Claude instead of rejected with 422; use streaming, because non-streaming responses currently return only the first text block. Tracing of long Claude streams no longer allocates quadratically or panics on malformed delta indices. +- **MCP interoperability** — Backends that send `application/json; charset=utf-8` (for example the MCP Java SDK) no longer have their list results dropped. OAuth metadata discovery now treats an empty or non-JSON document as a miss and tries the next well-known URL. +- **`aigw run` base URL handling** — A path in `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` (for example OpenRouter's `/api/v1`) is now used as the request prefix instead of being dropped. Base URLs that aren't http or https are rejected. +- **Unresolvable `InferencePool` routes are rejected** — When Envoy Gateway couldn't resolve an `InferencePool`, the extension server used to accept a route that only returned 500. It now rejects that route so the error is visible. + +## 📖 Upgrade Guidance + +Upgrading from **v1.1** needs no CRD storage migration, and every new field is optional. Several security fixes change behavior, though, so work through the steps that apply to you **before** you upgrade. Upgrade from v1.1.x, not directly from v1.0.x. + +### 1. Upgrade Gateway API and Envoy Gateway first + +Envoy Gateway v1.9 requires the Gateway API **v1.6** CRDs. If you use the standard channel, move any `TCPRoute` and `UDPRoute` manifests to `gateway.networking.k8s.io/v1` before you upgrade the CRDs. Then upgrade the CRDs and Envoy Gateway. The Agent Router values file is unchanged: + +```shell +helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \ + --version v1.9.2 \ + --set crds.gatewayAPI.enabled=true \ + --set crds.envoyGateway.enabled=true \ + | kubectl apply --force-conflicts --server-side -f - + +helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \ + --version v1.9.2 \ + --namespace envoy-gateway-system \ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/v1.2.0/manifests/envoy-gateway-values.yaml +``` + +Envoy Gateway v1.9 has its own breaking changes. Read its [v1.9.0](https://gateway.envoyproxy.io/news/releases/notes/v1.9.0/) and [v1.9.2](https://gateway.envoyproxy.io/news/releases/notes/v1.9.2/) release notes. The ones most likely to affect AI traffic: + +- Lua `EnvoyExtensionPolicy` is disabled by default. Enable it with `extensionApis.enableLua`. +- `mergeType` on a `SecurityPolicy` or `BackendTrafficPolicy` is only accepted on route targets. +- Tracing client sampling now defaults to 0%. +- EndpointSlice indexing can increase controller memory. + +Do **not** enable Envoy Gateway's new `EnvoyProxy` `mergeBackends` option with Agent Router yet. It shares clusters between routes, which breaks how Agent Router resolves the backend for a request. + +### 2. Plan for the MCP session seed change + +The chart now generates a random MCP session encryption seed, so upgrading ends active MCP sessions. Pick one: + +- **Accept a reconnect.** Do nothing. Clients re-initialize their sessions. +- **Keep sessions during the upgrade.** Add the old seed as a fallback in your values for this upgrade, then remove it once clients have reconnected: + + ```yaml + controller: + mcp: + sessionEncryption: + fallback: + seed: default-insecure-seed + ``` + +- **GitOps or `helm template`.** The chart can't read back a seed it generated earlier, so it generates a new one on every render. Supply your own Secret instead: + + ```shell + kubectl create secret generic mcp-session-seed -n envoy-ai-gateway-system \ + --from-literal=seed="$(openssl rand -hex 32)" + ``` + + Then set `controller.mcp.sessionEncryption.existingSecret: mcp-session-seed`. + +### 3. Check your MCPRoutes + +- `securityPolicy.oauth.issuer` must match your identity provider's `iss` claim exactly, including any trailing slash. +- Any authorization or `backendSelector` CEL that references `request.auth.jwt` needs `securityPolicy.oauth` on the same route. +- Make CEL expressions null-safe, because evaluation errors now deny. For example, use `"x-team" in request.headers && request.headers["x-team"] == "blocked"` instead of `request.headers["x-team"] == "blocked"`. +- On routes with both `headers` and `oauth`, make sure clients send those headers during OAuth discovery. + +### 4. Add ReferenceGrants for cross-namespace credential Secrets + +If a `BackendSecurityPolicy` references an Azure client secret, a GCP credentials file, or an OIDC client secret in another namespace, create a grant in the **Secret's** namespace: + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: allow-backend-security-policy-secrets + namespace: credentials # the Secret's namespace +spec: + from: + - group: aigateway.envoyproxy.io + kind: BackendSecurityPolicy + namespace: ai-backends # the BackendSecurityPolicy's namespace + to: + - group: "" + kind: Secret +``` + +### 5. Turn on metadata forwarding for `credentialOverride.fromDynamicMetadata` + +If you use per-request credentials from dynamic metadata, list the metadata namespace in a `GatewayConfig`, and reference that `GatewayConfig` from the Gateway with the `aigateway.envoyproxy.io/gateway-config` annotation: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: my-gateway-config + namespace: default # same namespace as the Gateway +spec: + extProc: + metadataForwardingNamespaces: + - envoy.filters.http.ext_authz +``` + +### 6. Upgrade Agent Router + +```shell +helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \ + --version v1.2.0 \ + --namespace envoy-ai-gateway-system + +helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \ + --version v1.2.0 \ + --namespace envoy-ai-gateway-system +``` + +The chart and image names keep the `ai-gateway` prefix. Only the project name changed. + +### 7. After the upgrade + +- **Dashboards:** GenAI metrics now carry a `gen_ai_backend` label. Panels that don't aggregate with `sum by (...)` will split into one series per backend. +- **Legacy Secrets:** the controller no longer writes the old `filter-config.yaml` Secrets. You can delete them. +- **Inference Extension:** if you use `InferencePool`, upgrade the Gateway API Inference Extension CRDs to v1.6.x and redeploy your endpoint picker. + +## 📦 Dependency Versions + +| Dependency | Version | +| ------------------------------- | ------- | +| Go | 1.27.1 | +| Envoy Gateway | v1.9.2 | +| Envoy Proxy | v1.39.1 | +| Gateway API | v1.6.1 | +| Gateway API Inference Extension | v1.6.2 | +| MCP Go SDK | v1.7.0 | + +## 🙏 Acknowledgements + +v1.2 came from 39 contributors, many of them contributing for the first time. Special thanks to: + +- **Ignasi Barrera**, who led the security hardening across MCP authentication, session binding, and cross-namespace references. +- **Hritik Raj**, who built the groundwork for the 2026-07-28 MCP specification. +- **Deepika Agrawal**, **Aishwarya Raimule**, and **Huabing (Robin) Zhao**, who added MCP prefix mode, credential injection, policy merging, and OAuth metadata URLs. +- **Yang Haoran**, **hustxiayang**, and **Xiaolin Lin**, who added provider coverage and fixed streaming bugs. +- **zirain**, who handled the Gateway API Inference Extension v1.6 upgrade and conformance. +- Everyone who reported bugs, reviewed PRs, and joined the community meetings during the move to Agent Router. + +## 🔮 What's Next + +Work already in flight for upcoming releases: + +- **The stateless 2026-07-28 MCP specification**, switched on end to end. +- **OAuth token exchange for MCP backends**, so the gateway can swap a client token for a backend-scoped one. +- **A dedicated `MCPBackend` CRD**, decoupling backend configuration from `MCPRoute`. +- **Longer quota windows** (weekly, monthly, and yearly) and **quota-aware routing**. +- **Agent-to-agent (A2A) traffic**, building on the A2A preview guide. + +The roadmap is community-driven. [Join us](https://github.com/theagentrouter/agent-router) and help shape it. diff --git a/site/src/components/ReleaseNotes/ReleaseNotes.module.css b/site/src/components/ReleaseNotes/ReleaseNotes.module.css index 4e8b80897f..ba41789579 100644 --- a/site/src/components/ReleaseNotes/ReleaseNotes.module.css +++ b/site/src/components/ReleaseNotes/ReleaseNotes.module.css @@ -108,7 +108,7 @@ /* Release Header */ .releaseHeader { - background: white; + background: var(--ifm-background-surface-color); border: 1px solid var(--ifm-color-emphasis-300); border-radius: 8px; padding: 1.5rem; @@ -223,7 +223,7 @@ align-items: flex-start; gap: 1rem; padding: 1rem; - background: white; + background: var(--ifm-background-surface-color); border: 1px solid var(--ifm-color-emphasis-300); border-radius: 8px; box-shadow: 0 2px 8px rgba(0, 0, 0, 0.05); @@ -257,7 +257,7 @@ border-radius: 8px; padding: 1.5rem; margin: 1.5rem 0; - background: white; + background: var(--ifm-background-surface-color); border: 1px solid var(--ifm-color-emphasis-300); } @@ -305,7 +305,7 @@ /* Release Cards */ .releaseCard { - background: white; + background: var(--ifm-background-surface-color); border: 1px solid var(--ifm-color-emphasis-300); border-radius: 8px; padding: 2rem; @@ -607,7 +607,7 @@ .timelineContent { flex: 1; - background: white; + background: var(--ifm-background-surface-color); border: 1px solid var(--ifm-color-emphasis-300); border-radius: 8px; padding: 1rem; @@ -746,7 +746,7 @@ font-family: monospace; color: var(--ifm-color-emphasis-900); display: inline; - background: #fff; + background: var(--ifm-code-background); padding: 0.01rem 0.5rem; word-break: break-word; white-space: normal; diff --git a/site/src/data/releases/index.json b/site/src/data/releases/index.json index 73f34e1827..43162afa4e 100644 --- a/site/src/data/releases/index.json +++ b/site/src/data/releases/index.json @@ -1,8 +1,8 @@ { "metadata": { "title": "Release Notes", - "description": "Release Notes for Envoy AI Gateway", - "lastUpdated": "2026-08-21" + "description": "Release Notes for Agent Router", + "lastUpdated": "2026-10-05" }, "community": { "discord": "https://discord.gg/xuxtPq43gZ", diff --git a/site/src/data/releases/v1.1.json b/site/src/data/releases/v1.1.json index 4ba467246b..a5c063b83c 100644 --- a/site/src/data/releases/v1.1.json +++ b/site/src/data/releases/v1.1.json @@ -3,7 +3,7 @@ "version": "v1.1", "title": "Envoy AI Gateway v1.1.x", "subtitle": "Token counting, per-request credentials, stream idle timeout, MCP hostname routing, OpenTelemetry GenAI tracing, and HTTP CONNECT egress.", - "badge": "Latest", + "badge": "Stable", "badgeType": "milestone" }, "releases": [ @@ -203,6 +203,7 @@ } ], "navigation": { - "previous": { "version": "v1.0.x Series", "path": "/release-notes/v1.0" } + "previous": { "version": "v1.0.x Series", "path": "/release-notes/v1.0" }, + "next": { "version": "v1.2.x Series", "path": "/release-notes/v1.2" } } } diff --git a/site/src/data/releases/v1.2.json b/site/src/data/releases/v1.2.json new file mode 100644 index 0000000000..a8849a2dca --- /dev/null +++ b/site/src/data/releases/v1.2.json @@ -0,0 +1,285 @@ +{ + "series": { + "version": "v1.2", + "title": "Agent Router v1.2.x", + "subtitle": "First release as Agent Router: MCP security hardening, bare MCP tool names, caller-supplied MCP credentials, AWS OpenAI-compatible and TypeSafe backends, and Envoy Gateway v1.9.", + "badge": "Latest", + "badgeType": "milestone" + }, + "releases": [ + { + "version": "v1.2.0", + "date": "October 5, 2026", + "type": "minor", + "tags": [ + { "text": "Agent Router", "type": "feature" }, + { "text": "MCP Security Hardening", "type": "feature" }, + { "text": "MCP Prefix Mode", "type": "feature" }, + { "text": "MCP Credential Passthrough", "type": "feature" }, + { "text": "MCP Policy Merging", "type": "feature" }, + { "text": "AWS OpenAI-Compatible", "type": "feature" }, + { "text": "TypeSafe System One", "type": "feature" }, + { "text": "Header Value Filters", "type": "feature" }, + { "text": "Per-Backend Metrics", "type": "feature" }, + { "text": "Envoy Gateway v1.9", "type": "feature" } + ], + "overview": "Agent Router v1.2.0 is the first release under the project's new name (formerly Envoy AI Gateway). Nothing you deploy is renamed: the aigateway.envoyproxy.io API group, the CRDs, the Helm charts, the container images, and the aigw CLI all stay the same. This release hardens MCP authentication and cross-namespace references. It adds bare tool names, caller-supplied credentials, and policy merging to MCPRoute, plus two new backend schemas (AWSOpenAI and TypeSafe). It also moves to Envoy Gateway v1.9.2, Gateway API v1.6, and Gateway API Inference Extension v1.6. Several security fixes change behavior, so read the breaking changes before you upgrade.", + "features": [ + { + "title": "Agent Router", + "items": [ + { + "title": "Same project, new name", + "description": "Envoy AI Gateway is now Agent Router, an Agentic AI Foundation project with the same code, maintainers, and Apache 2.0 license. The website moved to theagentrouter.ai and the repository to theagentrouter/agent-router; old links redirect. Your manifests, Helm values, and automation keep working unchanged." + } + ] + }, + { + "title": "MCP Gateway", + "items": [ + { + "title": "Bare tool and prompt names with prefixMode: Never", + "description": "Clients that hardcode tool names, such as interactive MCP Apps, no longer have to see the <backend>__ prefix. Set prefixMode: Never on the route or on individual backendRefs. A Never-mode backend must list its tools in toolSelector.include, and the controller marks the route NotAccepted if two Never-mode backends expose the same name. Prompts are exposed bare only when they are listed in the new promptSelector.include. Resources always keep the prefix. The default is still Always." + }, + { + "title": "Caller-supplied credentials with injectionPolicy: IfNotPresent", + "description": "Let users bring their own upstream token, for example a personal GitHub PAT forwarded with forwardHeaders, and fall back to a shared service key when they don't. With IfNotPresent, the configured API key is injected only when the target header is missing. This applies to header injection only." + }, + { + "title": "Inherit Gateway-level policies with mergeType", + "description": "The SecurityPolicy and BackendTrafficPolicy that Agent Router generates for an MCPRoute used to replace any Gateway- or listener-level policy. Set securityPolicy.mergeType or backendTrafficPolicy.mergeType to StrategicMerge or JSONMerge to keep a Gateway-wide rate limit or ext-auth policy in force alongside MCP OAuth." + }, + { + "title": "OAuth for identity providers with non-standard metadata paths", + "description": "oauth.authorizationServerMetadataUrl points the controller at the exact RFC 8414 metadata document when the issuer URL doesn't lead to it. The controller fetches this document while reconciling, so it needs network access to the identity provider. If the fetch fails, reconciliation fails." + }, + { + "title": "Groundwork for the next MCP spec", + "description": "The initialize response now negotiates the protocol version with the client and backends instead of always answering 2025-06-18. Merged list responses also carry ttlMs and cacheScope caching hints. Support for the stateless 2026-07-28 MCP specification is in progress and not yet enabled." + } + ] + }, + { + "title": "Providers & API Compatibility", + "items": [ + { + "title": "AWS OpenAI-compatible endpoint (AWSOpenAI)", + "description": "Send OpenAI-format /v1/chat/completions and /v1/responses requests to Bedrock Runtime's OpenAI-compatible /openai/v1 endpoint without a translation hop. Requests are signed with SigV4 through an AWS BackendSecurityPolicy. The prefix defaults to openai/v1. Other endpoints return 422." + }, + { + "title": "TypeSafe System One (TypeSafe)", + "description": "Route TypeSafe's Jev decision model through the gateway at /typesafe/v1/systemone with API-key auth. Request bodies pass through unchanged apart from a model name override, and token usage feeds the standard metrics and llmRequestCosts. Streaming is not supported." + }, + { + "title": "Per-backend anthropic-beta filtering", + "description": "One anthropic-beta value that a provider rejects no longer fails the whole request. AIServiceBackend.spec.headerValueFilters drops listed values (Denylist, the default) or keeps only listed values (Allowlist). It applies to /v1/messages traffic to GCPAnthropic and AWSAnthropic backends. Bedrock also accepts four more beta flags, including thinking-token-count-2026-05-13." + }, + { + "title": "Claude and Gemini request coverage", + "description": "Claude Opus 5 and 5.5 accept reasoning_effort on Vertex AI and Bedrock, and structured output (response_format) on Vertex AI. Chat-completion tools forward eager_input_streaming to Claude. Gemini on Vertex AI accepts vLLM-style structured_outputs (json, regex, choice)." + }, + { + "title": "OpenAI cache_write_tokens", + "description": "Prompt-cache writes reported as cache_write_tokens by OpenAI Chat Completions and the Responses API now count toward cache-creation token metrics and costs. Translated Claude and Bedrock responses report the same value." + } + ] + }, + { + "title": "Model Catalog & Observability", + "items": [ + { + "title": "Hide aliases from /v1/models", + "description": "Set excludeFromModelsEndpoint: true on an AIGatewayRoute rule to keep internal or backwards-compatible model aliases out of /v1/models and /anthropic/v1/models. Requests to those models still route normally." + }, + { + "title": "Per-backend GenAI metrics", + "description": "All four gen_ai.* metrics now carry a gen_ai.backend attribute (gen_ai_backend in Prometheus) set to the backend's namespace/name. This lets you compare latency, time to first token, and token usage across providers serving the same model." + } + ] + }, + { + "title": "Inference & Operations", + "items": [ + { + "title": "Gateway API Inference Extension v1.6", + "description": "An InferencePool without endpointPickerRef is now reported as Accepted=False (EndpointPickerRefMissing) instead of breaking routing. appProtocol: kubernetes.io/h2c is honored for cleartext HTTP/2 model servers." + }, + { + "title": "Safer controller rollouts", + "description": "The controller reports Ready only after its caches sync, so Envoy Gateway no longer calls the extension server with an empty view of your resources during a rollout. The chart adds a startup probe and a controller.priorityClassName value." + } + ] + } + ], + "apiChanges": [ + { + "title": "AIGatewayRouteRule.excludeFromModelsEndpoint", + "description": "Optional boolean (default false) in v1alpha1 and v1beta1." + }, + { + "title": "AIServiceBackend.spec.headerValueFilters", + "description": "Optional list (max 16, one per header) of {name, mode, values}. mode is Denylist (default) or Allowlist; values holds up to 64 entries. A CEL rule rejects the field unless the schema is GCPAnthropic or AWSAnthropic, and only anthropic-beta is honored today. v1beta1 only." + }, + { + "title": "VersionedAPISchema.name: AWSOpenAI and TypeSafe", + "description": "Two new schema values. TypeSafe defaults its version to v1." + }, + { + "title": "GatewayConfig.spec.extProc.metadataForwardingNamespaces", + "description": "Optional list (max 32) of dynamic metadata namespaces that Envoy forwards to the external processor. It is required for credentialOverride.fromDynamicMetadata. Under Envoy Gateway mergeGateways, every Gateway in the class shares the union of these namespaces." + }, + { + "title": "MCPRoute.spec.prefixMode and backendRefs[].prefixMode", + "description": "Always or Never; unset means Always. The per-backend value takes precedence. v1beta1 only." + }, + { + "title": "MCPRoute.spec.backendRefs[].promptSelector", + "description": "Filters prompts with include, includeRegex, exclude, or excludeRegex (max 32 each). Exclude rules win over include rules. v1beta1 only." + }, + { + "title": "MCPRoute.spec.securityPolicy.mergeType and spec.backendTrafficPolicy.mergeType", + "description": "Envoy Gateway MergeType; Replace is rejected. Unset keeps the previous override behavior." + }, + { + "title": "MCPRoute.spec.backendRefs[].securityPolicy.apiKey.injectionPolicy", + "description": "Always (default) or IfNotPresent. IfNotPresent cannot be combined with queryParam. v1beta1 only." + }, + { + "title": "MCPRoute.spec.securityPolicy.oauth.authorizationServerMetadataUrl", + "description": "Optional URI, up to 1024 characters." + }, + { + "title": "New MCPRoute validation for JWT-based CEL", + "description": "securityPolicy.oauth is now required when an authorization rule or backendSelector rule's cel expression references auth.jwt. See Breaking Changes." + } + ], + "deprecations": [ + { + "title": "cache_creation_input_tokens in translated OpenAI-format responses", + "description": "Responses translated from Claude and Bedrock now report cache_write_tokens alongside the legacy cache_creation_input_tokens field. Read cache_write_tokens instead; the legacy field will be removed in v1.3.0." + } + ], + "bugFixes": [ + { + "title": "Credential rotation no longer serves stale tokens", + "description": "Fixes a v1.1.0 regression: the controller read backend credential Secrets from its informer cache, so a rotated credential could miss the filter config and upstream calls failed until the next reconcile. Secrets are read from the API server again." + }, + { + "title": "Gateways with TCP or TLS listeners translate again", + "description": "A listener without an HTTP connection manager, such as a TCPRoute or TLS-passthrough listener, made the extension server fail translation for the whole Gateway. Those listeners are now skipped." + }, + { + "title": "Controller crash on combined priority backends", + "description": "Adding a backendRef before its AIServiceBackend existed could panic and crash-loop the controller. The controller now logs the mismatch and continues." + }, + { + "title": "Self-signed webhook survives helm upgrade", + "description": "With the default self-signed webhook certificate, an upgrade or Argo CD sync wiped the webhook's caBundle, so Envoy pods failed admission until the controller restarted. The chart now embeds the CA bundle." + }, + { + "title": "credentialOverride.fromDynamicMetadata now receives metadata", + "description": "In v1.1.0 Envoy never forwarded the metadata namespace to the external processor. Requests silently used the configured credential, or failed with 401 when fallbackToConfigured: false. List the namespace in GatewayConfig.spec.extProc.metadataForwardingNamespaces (see Upgrade Guidance)." + }, + { + "title": "Token usage is always captured on streams", + "description": "Duplicate stream_options in a streaming request could stop include_usage from reaching the provider, which bypassed token accounting. Usage metadata is now also recorded when a client disconnects right after the final SSE frame. Responses API streams framed with CRLF now report usage, model, and tracing data." + }, + { + "title": "Clear 422 errors instead of 500s", + "description": "Calling an endpoint that a backend's schema doesn't support (for example /v1/responses to Vertex AI Gemini) returns 422 Unprocessable Entity. Oversized JSON schemas sent to Vertex AI return 422 once they exceed 50,000 nodes." + }, + { + "title": "Lower external processor memory", + "description": "Model-name strings used as metric labels no longer keep entire request and response bodies in memory, which was most noticeable with large-context requests." + }, + { + "title": "Per-tenant token limits charge the right tenant", + "description": "With QuotaPolicy Distinct header selectors, the end-of-stream token cost went into one shared bucket instead of the caller's own. See Breaking Changes for the effect on tenants." + }, + { + "title": "Claude translation fixes", + "description": "Structured outputs keep the property order you declared. Streamed extended-thinking text is no longer empty. The web_search_20260209 tool on /v1/chat/completions is now forwarded to Claude instead of rejected with 422; use streaming, because non-streaming responses currently return only the first text block. Tracing of long Claude streams no longer allocates quadratically or panics on malformed delta indices." + }, + { + "title": "MCP interoperability", + "description": "Backends that send application/json; charset=utf-8 (for example the MCP Java SDK) no longer have their list results dropped. OAuth metadata discovery now treats an empty or non-JSON document as a miss and tries the next well-known URL." + }, + { + "title": "aigw run base URL handling", + "description": "A path in OPENAI_BASE_URL or ANTHROPIC_BASE_URL (for example OpenRouter's /api/v1) is now used as the request prefix instead of being dropped. Base URLs that aren't http or https are rejected." + }, + { + "title": "Unresolvable InferencePool routes are rejected", + "description": "When Envoy Gateway couldn't resolve an InferencePool, the extension server used to accept a route that only returned 500. It now rejects that route so the error is visible." + } + ], + "breakingChanges": [ + { + "title": "Random MCP session encryption seed", + "description": "The Helm chart no longer defaults to the well-known default-insecure-seed. It generates a random seed, stores it in the Secret <controller fullname>-mcp-session-encryption, and reuses it on later upgrades. Upgrading therefore ends active MCP sessions, and clients must re-initialize. To keep sessions during the upgrade, set controller.mcp.sessionEncryption.fallback.seed=default-insecure-seed and remove it once clients have reconnected. If you render the chart with helm template (common with GitOps), set controller.mcp.sessionEncryption.existingSecret or seed. Otherwise every render generates a new seed." + }, + { + "title": "MCP OAuth tokens must match oauth.issuer", + "description": "The generated JWT provider now checks the token's iss claim. Tokens with a missing or different issuer, including a trailing-slash mismatch, get 401. Make sure securityPolicy.oauth.issuer matches your identity provider's iss exactly." + }, + { + "title": "JWT data in MCP CEL requires securityPolicy.oauth", + "description": "The gateway reads request.auth.jwt claims and scopes, and the session subject, only from a JWT that Envoy has verified. Without securityPolicy.oauth, claims are empty, and new CRD validation rejects authorization or backendSelector CEL that references auth.jwt. Existing routes that break this rule keep their stored spec, but every update is rejected. At runtime their claim lookups fail, and therefore deny, and their scope checks evaluate to false. Add oauth or remove the JWT references. MCP sessions are now bound to the verified subject; reusing a session ID as a different user returns 401." + }, + { + "title": "MCP CEL evaluation errors now deny", + "description": "An authorization or backendSelector CEL expression that fails at runtime, for example by reading a header that isn't present, used to be skipped. It now denies: tools/call returns 403, the tool is hidden from tools/list, and the backend is left out of the session. Write expressions defensively, for example \"x-team\" in request.headers && request.headers[\"x-team\"] == \"blocked\"." + }, + { + "title": "MCP OAuth discovery honors spec.headers", + "description": "On an MCPRoute with both headers matches and oauth, the OAuth well-known endpoints now apply the same header matches. Clients that run OAuth discovery without those headers get 404." + }, + { + "title": "Cross-namespace Secrets in BackendSecurityPolicy need a ReferenceGrant", + "description": "azureCredentials.clientSecretRef, gcpCredentials.credentialsFile.secretRef, and the OIDC clientSecret under AWS, Azure, and GCP oidcExchangeToken were read from other namespaces without any check. A missing grant now sets the policy to NotAccepted. Already-issued tokens keep working until they expire, so the failure may appear late. Create a ReferenceGrant in the Secret's namespace (see Upgrade Guidance)." + }, + { + "title": "Ungranted cross-namespace backends no longer reach the data plane", + "description": "An AIGatewayRoute backendRef to an AIServiceBackend or InferencePool in another namespace without a ReferenceGrant already marked the route NotAccepted, but its credentials were still written to the external processor's config. Such backends are now left out. Working setups already have the grant; this affects only routes that were already NotAccepted." + }, + { + "title": "Per-tenant token limits are now enforced", + "description": "With the QuotaPolicy Distinct fix, tenants can start receiving 429 at the limits you configured. Review per-tenant limits before upgrading." + }, + { + "title": "External processor -configPath removed", + "description": "The external processor reads only its config bundle (-configBundlePath), and the controller no longer writes the legacy filter-config.yaml Secret. Existing legacy Secrets are left in place and can be deleted. This matters only if you run the external processor yourself, or still have Envoy pods created before config bundles existed (v1.0.x era). Restart those pods before upgrading." + } + ], + "dependencies": [ + { + "title": "Go 1.27.1", + "description": "Updated from Go 1.26.4." + }, + { + "title": "Envoy Gateway v1.9.2", + "description": "Built on Envoy Gateway v1.9.2, up from v1.8.1." + }, + { + "title": "Envoy v1.39", + "description": "Envoy Proxy v1.39.1, as shipped by Envoy Gateway v1.9.2." + }, + { + "title": "Gateway API v1.6.1", + "description": "Updated from v1.5.1. Envoy Gateway v1.9 requires the v1.6 CRDs." + }, + { + "title": "Gateway API Inference Extension v1.6.2", + "description": "Updated from v1.0.2." + }, + { + "title": "MCP Go SDK v1.7.0", + "description": "Unchanged from v1.1." + } + ] + } + ], + "navigation": { + "previous": { "version": "v1.1.x Series", "path": "/release-notes/v1.1" } + } +} diff --git a/site/src/pages/release-notes/index.mdx b/site/src/pages/release-notes/index.mdx index 93fd17e8bb..f52d7c4b2f 100644 --- a/site/src/pages/release-notes/index.mdx +++ b/site/src/pages/release-notes/index.mdx @@ -14,10 +14,12 @@ import v06Data from '../../data/releases/v0.6.json'; import v07Data from '../../data/releases/v0.7.json'; import v10Data from '../../data/releases/v1.0.json'; import v11Data from '../../data/releases/v1.1.json'; +import v12Data from '../../data/releases/v1.2.json'; import indexData from '../../data/releases/index.json'; import styles from '../../components/ReleaseNotes/ReleaseNotes.module.css'; export const allReleases = [ + ...v12Data.releases, ...v11Data.releases, ...v10Data.releases, ...v07Data.releases, @@ -45,15 +47,27 @@ export const getMarkerStyle = (type) => { Stay up-to-date with the latest improvements, features, and fixes in Agent Router.
+ release.version).join(', ')} + /> + release.version).join(', ')} /> diff --git a/site/src/pages/release-notes/v1.1.mdx b/site/src/pages/release-notes/v1.1.mdx index d482495e95..c853a2d3aa 100644 --- a/site/src/pages/release-notes/v1.1.mdx +++ b/site/src/pages/release-notes/v1.1.mdx @@ -53,12 +53,13 @@ export const patchReleases = releases.slice(1).reverse(); )} -{mainRelease.deprecations?.length > 0 && ( - <> - ### Deprecations - - -)} +## ⚠️ Breaking Changes + + + +## 🐛 Bug Fixes + + ## 📖 Upgrade Guidance diff --git a/site/src/pages/release-notes/v1.2.mdx b/site/src/pages/release-notes/v1.2.mdx new file mode 100644 index 0000000000..c02dd021b0 --- /dev/null +++ b/site/src/pages/release-notes/v1.2.mdx @@ -0,0 +1,229 @@ +--- +title: Agent Router v1.2.x Release Series +description: First release as Agent Router — MCP security hardening, bare MCP tool names, caller-supplied MCP credentials, AWS OpenAI-compatible and TypeSafe backends, and Envoy Gateway v1.9 +toc_min_heading_level: 2 +toc_max_heading_level: 4 +--- + +import Link from '@docusaurus/Link'; +import { + ReleaseSeriesLayout, + ReleaseHeader, + FeatureSectionCard, + PatchRelease, + ItemList, + Dependencies +} from '../../components/ReleaseNotes'; +import releaseData from '../../data/releases/v1.2.json'; +import React from 'react'; + +export const { series, releases, navigation } = releaseData; +export const mainRelease = releases[0]; +export const patchReleases = releases.slice(1).reverse(); + + + + + +{mainRelease.overview} + + + +## ✨ New Features + +{mainRelease.features.map((featureSection, index) => ( + +))} + +## 🔗 API Updates + +{mainRelease.apiChanges.length > 0 && ( + +)} + +### Deprecations + + + +## ⚠️ Breaking Changes + + + +## 🐛 Bug Fixes + +{mainRelease.bugFixes?.length > 0 && ( + +)} + +## 📖 Upgrade Guidance + +Upgrading from **v1.1** needs no CRD storage migration, and every new field is optional. Several security fixes change behavior, though, so work through the steps that apply to you **before** you upgrade. Upgrade from v1.1.x, not directly from v1.0.x. + +### 1. Upgrade Gateway API and Envoy Gateway first + +Envoy Gateway v1.9 requires the Gateway API **v1.6** CRDs. If you use the standard channel, move any `TCPRoute` and `UDPRoute` manifests to `gateway.networking.k8s.io/v1` before you upgrade the CRDs. Then upgrade the CRDs and Envoy Gateway. The Agent Router values file is unchanged: + +```shell +helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \ + --version v1.9.2 \ + --set crds.gatewayAPI.enabled=true \ + --set crds.envoyGateway.enabled=true \ + | kubectl apply --force-conflicts --server-side -f - + +helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \ + --version v1.9.2 \ + --namespace envoy-gateway-system \ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/v1.2.0/manifests/envoy-gateway-values.yaml +``` + +Envoy Gateway v1.9 has its own breaking changes. Read its [v1.9.0](https://gateway.envoyproxy.io/news/releases/notes/v1.9.0/) and [v1.9.2](https://gateway.envoyproxy.io/news/releases/notes/v1.9.2/) release notes. The ones most likely to affect AI traffic: + +- Lua `EnvoyExtensionPolicy` is disabled by default. Enable it with `extensionApis.enableLua`. +- `mergeType` on a `SecurityPolicy` or `BackendTrafficPolicy` is only accepted on route targets. +- Tracing client sampling now defaults to 0%. +- EndpointSlice indexing can increase controller memory. + +Do **not** enable Envoy Gateway's new `EnvoyProxy` `mergeBackends` option with Agent Router yet. It shares clusters between routes, which breaks how Agent Router resolves the backend for a request. + +### 2. Plan for the MCP session seed change + +The chart now generates a random MCP session encryption seed, so upgrading ends active MCP sessions. Pick one: + +- **Accept a reconnect.** Do nothing. Clients re-initialize their sessions. +- **Keep sessions during the upgrade.** Add the old seed as a fallback in your values for this upgrade, then remove it once clients have reconnected: + + ```yaml + controller: + mcp: + sessionEncryption: + fallback: + seed: default-insecure-seed + ``` + +- **GitOps or `helm template`.** The chart can't read back a seed it generated earlier, so it generates a new one on every render. Supply your own Secret instead: + + ```shell + kubectl create secret generic mcp-session-seed -n envoy-ai-gateway-system \ + --from-literal=seed="$(openssl rand -hex 32)" + ``` + + Then set `controller.mcp.sessionEncryption.existingSecret: mcp-session-seed`. + +### 3. Check your MCPRoutes + +- `securityPolicy.oauth.issuer` must match your identity provider's `iss` claim exactly, including any trailing slash. +- Any authorization or `backendSelector` CEL that references `request.auth.jwt` needs `securityPolicy.oauth` on the same route. +- Make CEL expressions null-safe, because evaluation errors now deny. For example, use `"x-team" in request.headers && request.headers["x-team"] == "blocked"` instead of `request.headers["x-team"] == "blocked"`. +- On routes with both `headers` and `oauth`, make sure clients send those headers during OAuth discovery. + +### 4. Add ReferenceGrants for cross-namespace credential Secrets + +If a `BackendSecurityPolicy` references an Azure client secret, a GCP credentials file, or an OIDC client secret in another namespace, create a grant in the **Secret's** namespace: + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: allow-backend-security-policy-secrets + namespace: credentials # the Secret's namespace +spec: + from: + - group: aigateway.envoyproxy.io + kind: BackendSecurityPolicy + namespace: ai-backends # the BackendSecurityPolicy's namespace + to: + - group: "" + kind: Secret +``` + +### 5. Turn on metadata forwarding for `credentialOverride.fromDynamicMetadata` + +If you use per-request credentials from dynamic metadata, list the metadata namespace in a `GatewayConfig`, and reference that `GatewayConfig` from the Gateway with the `aigateway.envoyproxy.io/gateway-config` annotation: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: my-gateway-config + namespace: default # same namespace as the Gateway +spec: + extProc: + metadataForwardingNamespaces: + - envoy.filters.http.ext_authz +``` + +### 6. Upgrade Agent Router + +```shell +helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \ + --version v1.2.0 \ + --namespace envoy-ai-gateway-system + +helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \ + --version v1.2.0 \ + --namespace envoy-ai-gateway-system +``` + +The chart and image names keep the `ai-gateway` prefix. Only the project name changed. + +### 7. After the upgrade + +- **Dashboards:** GenAI metrics now carry a `gen_ai_backend` label. Panels that don't aggregate with `sum by (...)` will split into one series per backend. +- **Legacy Secrets:** the controller no longer writes the old `filter-config.yaml` Secrets. You can delete them. +- **Inference Extension:** if you use `InferencePool`, upgrade the Gateway API Inference Extension CRDs to v1.6.x and redeploy your endpoint picker. + +## 📦 Dependencies Versions + + + +## ⏩ Patch Releases + +{patchReleases.map((release, index) => ( + +))} + +## 🙏 Acknowledgements + +v1.2 came from 39 contributors, many of them contributing for the first time. Special thanks to: + +- **Ignasi Barrera**, who led the security hardening across MCP authentication, session binding, and cross-namespace references. +- **Hritik Raj**, who built the groundwork for the 2026-07-28 MCP specification. +- **Deepika Agrawal**, **Aishwarya Raimule**, and **Huabing (Robin) Zhao**, who added MCP prefix mode, credential injection, policy merging, and OAuth metadata URLs. +- **Yang Haoran**, **hustxiayang**, and **Xiaolin Lin**, who added provider coverage and fixed streaming bugs. +- **zirain**, who handled the Gateway API Inference Extension v1.6 upgrade and conformance. +- Everyone who reported bugs, reviewed PRs, and joined the community meetings during the move to Agent Router. + +## 🔮 What's Next + +Work already in flight for upcoming releases: + +- **The stateless 2026-07-28 MCP specification**, switched on end to end. +- **OAuth token exchange for MCP backends**, so the gateway can swap a client token for a backend-scoped one. +- **A dedicated `MCPBackend` CRD**, decoupling backend configuration from `MCPRoute`. +- **Longer quota windows** (weekly, monthly, and yearly) and **quota-aware routing**. +- **Agent-to-agent (A2A) traffic**, building on the A2A preview guide. + +The roadmap is community-driven. [Join us](https://github.com/theagentrouter/agent-router) and help shape it. + + From 0082d7c465322be1c3b56be1808025b53376b8c4 Mon Sep 17 00:00:00 2001 From: Ignasi Barrera Date: Tue, 6 Oct 2026 10:07:36 +0200 Subject: [PATCH 2/5] add reference grant notes Signed-off-by: Ignasi Barrera --- release-notes/v1.2.0.md | 11 ++++++++--- site/src/data/releases/v1.2.json | 12 ++++++++++-- site/src/pages/release-notes/v1.2.mdx | 5 ++++- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/release-notes/v1.2.0.md b/release-notes/v1.2.0.md index 76c6196e40..6085f75193 100644 --- a/release-notes/v1.2.0.md +++ b/release-notes/v1.2.0.md @@ -60,14 +60,16 @@ Agent Router v1.2.0 is the first release under the project's new name (formerly - **JWT data in MCP CEL requires `securityPolicy.oauth`** — The gateway reads `request.auth.jwt` claims and scopes, and the session subject, only from a JWT that Envoy has verified. Without `securityPolicy.oauth`, claims are empty, and new CRD validation rejects authorization or `backendSelector` CEL that references `auth.jwt`. Existing routes that break this rule keep their stored spec, but every update is rejected. At runtime their claim lookups fail, and therefore deny, and their scope checks evaluate to false. Add `oauth` or remove the JWT references. MCP sessions are now bound to the verified subject; reusing a session ID as a different user returns 401. - **MCP CEL evaluation errors now deny** — An authorization or `backendSelector` CEL expression that fails at runtime, for example by reading a header that isn't present, used to be skipped. It now denies: `tools/call` returns 403, the tool is hidden from `tools/list`, and the backend is left out of the session. Write expressions defensively, for example `"x-team" in request.headers && request.headers["x-team"] == "blocked"`. - **MCP OAuth discovery honors `spec.headers`** — On an MCPRoute with both `headers` matches and `oauth`, the OAuth well-known endpoints now apply the same header matches. Clients that run OAuth discovery without those headers get 404. -- **Cross-namespace Secrets in `BackendSecurityPolicy` need a ReferenceGrant** — `azureCredentials.clientSecretRef`, `gcpCredentials.credentialsFile.secretRef`, and the OIDC `clientSecret` under AWS, Azure, and GCP `oidcExchangeToken` were read from other namespaces without any check. A missing grant now sets the policy to NotAccepted. Already-issued tokens keep working until they expire, so the failure may appear late. Create a ReferenceGrant in the Secret's namespace (see Upgrade Guidance). +- **Cross-namespace Secrets in `BackendSecurityPolicy` need a ReferenceGrant** — `azureCredentials.clientSecretRef`, `gcpCredentials.credentialsFile.secretRef`, and the OIDC `clientSecret` under AWS, Azure, and GCP `oidcExchangeToken` were read from other namespaces without any check. The `secretRef` of `apiKey`, `azureAPIKey`, `anthropicAPIKey`, and `awsCredentials.credentialsFile` ignored its namespace, so a cross-namespace reference failed or silently used a Secret with the same name from the policy's own namespace. All of these now honor the namespace, and a missing grant sets the policy to NotAccepted. Already-issued tokens keep working until they expire, so the failure may appear late. Create a ReferenceGrant in the Secret's namespace (see Upgrade Guidance). - **Ungranted cross-namespace backends no longer reach the data plane** — An `AIGatewayRoute` `backendRef` to an `AIServiceBackend` or `InferencePool` in another namespace without a ReferenceGrant already marked the route NotAccepted, but its credentials were still written to the external processor's config. Such backends are now left out. Working setups already have the grant; this affects only routes that were already NotAccepted. +- **ReferenceGrant `to.name` is now enforced** — A ReferenceGrant `to` entry that sets `name` used to authorize every object of that group and kind in its namespace. It now authorizes only the named object. References to other objects, such as another Secret, `AIServiceBackend`, or `InferencePool`, are NotAccepted and left out of the data plane. Check grants that set `name` before upgrading. - **Per-tenant token limits are now enforced** — With the `QuotaPolicy` `Distinct` fix, tenants can start receiving 429 at the limits you configured. Review per-tenant limits before upgrading. - **External processor `-configPath` removed** — The external processor reads only its config bundle (`-configBundlePath`), and the controller no longer writes the legacy `filter-config.yaml` Secret. Existing legacy Secrets are left in place and can be deleted. This matters only if you run the external processor yourself, or still have Envoy pods created before config bundles existed (v1.0.x era). Restart those pods before upgrading. ## 🐛 Bug Fixes -- **Credential rotation no longer serves stale tokens** — Fixes a v1.1.0 regression: the controller read backend credential Secrets from its informer cache, so a rotated credential could miss the filter config and upstream calls failed until the next reconcile. Secrets are read from the API server again. +- **Credential rotation no longer serves stale tokens** — Fixes a v1.1.0 regression: the controller read backend credential Secrets from its informer cache, so a rotated credential could miss the filter config and upstream calls failed until the next reconcile. Secrets are read from the API server again. Credential changes in a `BackendSecurityPolicy` that targets an `InferencePool` now also reach the Gateway's filter config. +- **Revoking a ReferenceGrant takes effect immediately** — Deleting a ReferenceGrant, or removing a `from` entry, now reconciles the affected `AIGatewayRoute`s (including `InferencePool` backends) and `BackendSecurityPolicy`s (including OIDC client secrets). Access they relied on used to stay in place until an unrelated reconcile. - **Gateways with TCP or TLS listeners translate again** — A listener without an HTTP connection manager, such as a TCPRoute or TLS-passthrough listener, made the extension server fail translation for the whole Gateway. Those listeners are now skipped. - **Controller crash on combined priority backends** — Adding a `backendRef` before its `AIServiceBackend` existed could panic and crash-loop the controller. The controller now logs the mismatch and continues. - **Self-signed webhook survives `helm upgrade`** — With the default self-signed webhook certificate, an upgrade or Argo CD sync wiped the webhook's `caBundle`, so Envoy pods failed admission until the controller restarted. The chart now embeds the CA bundle. @@ -144,7 +146,7 @@ The chart now generates a random MCP session encryption seed, so upgrading ends ### 4. Add ReferenceGrants for cross-namespace credential Secrets -If a `BackendSecurityPolicy` references an Azure client secret, a GCP credentials file, or an OIDC client secret in another namespace, create a grant in the **Secret's** namespace: +If a `BackendSecurityPolicy` references a credential Secret in another namespace, create a grant in the **Secret's** namespace. This now applies to every credential type: API keys, AWS credentials files, Azure client secrets, GCP credentials files, and OIDC client secrets. ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 @@ -160,8 +162,11 @@ spec: to: - group: "" kind: Secret + name: openai-api-key # optional: limit the grant to one Secret ``` +A `to` entry with `name` now authorizes only that object. Without `name`, it covers every object of that kind in the namespace. If an existing grant sets `name`, make sure it lists every Secret, `AIServiceBackend`, or `InferencePool` that other namespaces reference. + ### 5. Turn on metadata forwarding for `credentialOverride.fromDynamicMetadata` If you use per-request credentials from dynamic metadata, list the metadata namespace in a `GatewayConfig`, and reference that `GatewayConfig` from the Gateway with the `aigateway.envoyproxy.io/gateway-config` annotation: diff --git a/site/src/data/releases/v1.2.json b/site/src/data/releases/v1.2.json index a8849a2dca..36fce6ebc3 100644 --- a/site/src/data/releases/v1.2.json +++ b/site/src/data/releases/v1.2.json @@ -162,7 +162,11 @@ "bugFixes": [ { "title": "Credential rotation no longer serves stale tokens", - "description": "Fixes a v1.1.0 regression: the controller read backend credential Secrets from its informer cache, so a rotated credential could miss the filter config and upstream calls failed until the next reconcile. Secrets are read from the API server again." + "description": "Fixes a v1.1.0 regression: the controller read backend credential Secrets from its informer cache, so a rotated credential could miss the filter config and upstream calls failed until the next reconcile. Secrets are read from the API server again. Credential changes in a BackendSecurityPolicy that targets an InferencePool now also reach the Gateway's filter config." + }, + { + "title": "Revoking a ReferenceGrant takes effect immediately", + "description": "Deleting a ReferenceGrant, or removing a from entry, now reconciles the affected AIGatewayRoutes (including InferencePool backends) and BackendSecurityPolicys (including OIDC client secrets). Access they relied on used to stay in place until an unrelated reconcile." }, { "title": "Gateways with TCP or TLS listeners translate again", @@ -236,12 +240,16 @@ }, { "title": "Cross-namespace Secrets in BackendSecurityPolicy need a ReferenceGrant", - "description": "azureCredentials.clientSecretRef, gcpCredentials.credentialsFile.secretRef, and the OIDC clientSecret under AWS, Azure, and GCP oidcExchangeToken were read from other namespaces without any check. A missing grant now sets the policy to NotAccepted. Already-issued tokens keep working until they expire, so the failure may appear late. Create a ReferenceGrant in the Secret's namespace (see Upgrade Guidance)." + "description": "azureCredentials.clientSecretRef, gcpCredentials.credentialsFile.secretRef, and the OIDC clientSecret under AWS, Azure, and GCP oidcExchangeToken were read from other namespaces without any check. The secretRef of apiKey, azureAPIKey, anthropicAPIKey, and awsCredentials.credentialsFile ignored its namespace, so a cross-namespace reference failed or silently used a Secret with the same name from the policy's own namespace. All of these now honor the namespace, and a missing grant sets the policy to NotAccepted. Already-issued tokens keep working until they expire, so the failure may appear late. Create a ReferenceGrant in the Secret's namespace (see Upgrade Guidance)." }, { "title": "Ungranted cross-namespace backends no longer reach the data plane", "description": "An AIGatewayRoute backendRef to an AIServiceBackend or InferencePool in another namespace without a ReferenceGrant already marked the route NotAccepted, but its credentials were still written to the external processor's config. Such backends are now left out. Working setups already have the grant; this affects only routes that were already NotAccepted." }, + { + "title": "ReferenceGrant to.name is now enforced", + "description": "A ReferenceGrant to entry that sets name used to authorize every object of that group and kind in its namespace. It now authorizes only the named object. References to other objects, such as another Secret, AIServiceBackend, or InferencePool, are NotAccepted and left out of the data plane. Check grants that set name before upgrading." + }, { "title": "Per-tenant token limits are now enforced", "description": "With the QuotaPolicy Distinct fix, tenants can start receiving 429 at the limits you configured. Review per-tenant limits before upgrading." diff --git a/site/src/pages/release-notes/v1.2.mdx b/site/src/pages/release-notes/v1.2.mdx index c02dd021b0..74d228e198 100644 --- a/site/src/pages/release-notes/v1.2.mdx +++ b/site/src/pages/release-notes/v1.2.mdx @@ -130,7 +130,7 @@ The chart now generates a random MCP session encryption seed, so upgrading ends ### 4. Add ReferenceGrants for cross-namespace credential Secrets -If a `BackendSecurityPolicy` references an Azure client secret, a GCP credentials file, or an OIDC client secret in another namespace, create a grant in the **Secret's** namespace: +If a `BackendSecurityPolicy` references a credential Secret in another namespace, create a grant in the **Secret's** namespace. This now applies to every credential type: API keys, AWS credentials files, Azure client secrets, GCP credentials files, and OIDC client secrets. ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 @@ -146,8 +146,11 @@ spec: to: - group: "" kind: Secret + name: openai-api-key # optional: limit the grant to one Secret ``` +A `to` entry with `name` now authorizes only that object. Without `name`, it covers every object of that kind in the namespace. If an existing grant sets `name`, make sure it lists every Secret, `AIServiceBackend`, or `InferencePool` that other namespaces reference. + ### 5. Turn on metadata forwarding for `credentialOverride.fromDynamicMetadata` If you use per-request credentials from dynamic metadata, list the metadata namespace in a `GatewayConfig`, and reference that `GatewayConfig` from the Gateway with the `aigateway.envoyproxy.io/gateway-config` annotation: From 3fc7af41a402cf66d5feb9c3952b0bb4d3c6a887 Mon Sep 17 00:00:00 2001 From: Ignasi Barrera Date: Tue, 6 Oct 2026 10:19:01 +0200 Subject: [PATCH 3/5] create versioned-docs for 1.2 Signed-off-by: Ignasi Barrera --- RELEASES.md | 11 +- site/docs/compatibility.md | 1 + site/docusaurus.config.ts | 9 +- site/versioned_docs/version-1.2/_vars.json | 8 + site/versioned_docs/version-1.2/api/api.mdx | 5323 +++++++++++++++++ .../version-1.2/capabilities/a2a/index.md | 337 ++ .../capabilities/gateway-config.md | 293 + .../version-1.2/capabilities/index.md | 66 + .../capabilities/inference/_setup.mdx | 48 + .../inference/aigatewayroute-inferencepool.md | 264 + .../inference/httproute-inferencepool.md | 108 + .../capabilities/inference/index.md | 27 + .../inference/inferencepool-support.md | 145 + .../llm-integrations/connect-providers.md | 572 ++ .../capabilities/llm-integrations/index.md | 22 + .../llm-integrations/prompt-caching.md | 174 + .../llm-integrations/supported-endpoints.md | 672 +++ .../llm-integrations/supported-providers.md | 45 + .../vendor-specific-fields.md | 207 + .../version-1.2/capabilities/mcp/index.md | 475 ++ .../capabilities/observability/accesslogs.md | 275 + .../capabilities/observability/index.md | 16 + .../capabilities/observability/metrics.md | 124 + .../capabilities/observability/tracing.md | 366 ++ .../version-1.2/capabilities/scaling.md | 117 + .../capabilities/security/index.md | 32 + .../capabilities/security/upstream-auth.mdx | 194 + .../traffic/header-body-mutations.md | 187 + .../version-1.2/capabilities/traffic/index.md | 6 + .../traffic/model-virtualization.md | 156 + .../capabilities/traffic/provider-fallback.md | 112 + .../capabilities/traffic/quota-policy.md | 369 ++ .../traffic/usage-based-ratelimiting.md | 336 ++ site/versioned_docs/version-1.2/cli/index.md | 17 + .../version-1.2/cli/installation.md | 100 + site/versioned_docs/version-1.2/cli/run.md | 383 ++ .../version-1.2/compatibility.md | 79 + .../concepts/architecture/control-plane.md | 135 + .../concepts/architecture/data-plane.md | 134 + .../concepts/architecture/index.md | 47 + .../architecture/system-architecture.md | 66 + .../version-1.2/concepts/index.md | 19 + .../version-1.2/concepts/resources.md | 55 + .../getting-started/basic-usage.md | 302 + .../connect-providers/anthropic.md | 149 + .../connect-providers/aws-bedrock.md | 385 ++ .../connect-providers/azure-openai.md | 138 + .../connect-providers/bitdeer.md | 163 + .../connect-providers/gcp-vertexai.md | 220 + .../connect-providers/index.md | 51 + .../connect-providers/openai.md | 166 + .../getting-started/connect-providers/tars.md | 124 + .../version-1.2/getting-started/index.md | 69 + .../getting-started/installation.md | 80 + .../getting-started/prerequisites.md | 196 + site/versioned_docs/version-1.2/index.md | 103 + .../versioned_docs/version-1.2/terminology.md | 160 + .../version-1.2-sidebars.json | 8 + site/versions.json | 1 + 59 files changed, 14444 insertions(+), 3 deletions(-) create mode 100644 site/versioned_docs/version-1.2/_vars.json create mode 100644 site/versioned_docs/version-1.2/api/api.mdx create mode 100644 site/versioned_docs/version-1.2/capabilities/a2a/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/gateway-config.md create mode 100644 site/versioned_docs/version-1.2/capabilities/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/inference/_setup.mdx create mode 100644 site/versioned_docs/version-1.2/capabilities/inference/aigatewayroute-inferencepool.md create mode 100644 site/versioned_docs/version-1.2/capabilities/inference/httproute-inferencepool.md create mode 100644 site/versioned_docs/version-1.2/capabilities/inference/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/inference/inferencepool-support.md create mode 100644 site/versioned_docs/version-1.2/capabilities/llm-integrations/connect-providers.md create mode 100644 site/versioned_docs/version-1.2/capabilities/llm-integrations/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/llm-integrations/prompt-caching.md create mode 100644 site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-endpoints.md create mode 100644 site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-providers.md create mode 100644 site/versioned_docs/version-1.2/capabilities/llm-integrations/vendor-specific-fields.md create mode 100644 site/versioned_docs/version-1.2/capabilities/mcp/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/observability/accesslogs.md create mode 100644 site/versioned_docs/version-1.2/capabilities/observability/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/observability/metrics.md create mode 100644 site/versioned_docs/version-1.2/capabilities/observability/tracing.md create mode 100644 site/versioned_docs/version-1.2/capabilities/scaling.md create mode 100644 site/versioned_docs/version-1.2/capabilities/security/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/security/upstream-auth.mdx create mode 100644 site/versioned_docs/version-1.2/capabilities/traffic/header-body-mutations.md create mode 100644 site/versioned_docs/version-1.2/capabilities/traffic/index.md create mode 100644 site/versioned_docs/version-1.2/capabilities/traffic/model-virtualization.md create mode 100644 site/versioned_docs/version-1.2/capabilities/traffic/provider-fallback.md create mode 100644 site/versioned_docs/version-1.2/capabilities/traffic/quota-policy.md create mode 100644 site/versioned_docs/version-1.2/capabilities/traffic/usage-based-ratelimiting.md create mode 100644 site/versioned_docs/version-1.2/cli/index.md create mode 100644 site/versioned_docs/version-1.2/cli/installation.md create mode 100644 site/versioned_docs/version-1.2/cli/run.md create mode 100644 site/versioned_docs/version-1.2/compatibility.md create mode 100644 site/versioned_docs/version-1.2/concepts/architecture/control-plane.md create mode 100644 site/versioned_docs/version-1.2/concepts/architecture/data-plane.md create mode 100644 site/versioned_docs/version-1.2/concepts/architecture/index.md create mode 100644 site/versioned_docs/version-1.2/concepts/architecture/system-architecture.md create mode 100644 site/versioned_docs/version-1.2/concepts/index.md create mode 100644 site/versioned_docs/version-1.2/concepts/resources.md create mode 100644 site/versioned_docs/version-1.2/getting-started/basic-usage.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/anthropic.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/aws-bedrock.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/azure-openai.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/bitdeer.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/gcp-vertexai.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/index.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/openai.md create mode 100644 site/versioned_docs/version-1.2/getting-started/connect-providers/tars.md create mode 100644 site/versioned_docs/version-1.2/getting-started/index.md create mode 100644 site/versioned_docs/version-1.2/getting-started/installation.md create mode 100644 site/versioned_docs/version-1.2/getting-started/prerequisites.md create mode 100644 site/versioned_docs/version-1.2/index.md create mode 100644 site/versioned_docs/version-1.2/terminology.md create mode 100644 site/versioned_sidebars/version-1.2-sidebars.json diff --git a/RELEASES.md b/RELEASES.md index 4226ec20c0..e39f7a2e9c 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -76,7 +76,16 @@ Each non-patch release should start with Release Candidate (RC) phase as follows The main branch should only accept the bug fixes, the security fixes, and documentation changes. The release candidate should always be cut from the main branch. -2. Prepare the docs site with the new version, following the process described in the [site/README.md](site/README.md) +2. Prepare the docs site with the new version: + + ``` + cd site + npm run docusaurus docs:version 0.50 + ``` + + This will create the new folder for the release, `versioned_docs/release-0.50`. You'll have to update the `_vars.json` and `compatibility.md` and + make sure all the versions are correct. + You also need to udpate the version list in `docusaurus.config.ts` and point the `latestVersion` to the versino being released. 3. Cut the request candidate tag from the main branch. The tag should be v0.50.0-rc1. Assuming the remote `origin` is the main envoyproxy/ai-gateway repository, the command to cut the tag is: diff --git a/site/docs/compatibility.md b/site/docs/compatibility.md index dee1fefb34..3ec7741756 100644 --- a/site/docs/compatibility.md +++ b/site/docs/compatibility.md @@ -11,6 +11,7 @@ This document provides compatibility information for Agent Router releases with | Agent Router | Envoy Gateway | Kubernetes | Gateway API | Inference Extension | Support Status | | ------------ | ----------------------------- | ---------- | ----------- | ------------------- | -------------- | | main | v1.9.2+ (Envoy Proxy v1.39.x) | v1.32+ | v1.6.1 | v1.6.2 | Development | +| v1.2.x | v1.9.2+ (Envoy Proxy v1.39.x) | v1.32+ | v1.6.1 | v1.6.2 | Supported | | v1.1.x | v1.8.1+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.6.0 | Supported | | v1.0.x | v1.8.1+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.0.2 | Supported | | v0.7.x | v1.8.x+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.0.2 | Supported | diff --git a/site/docusaurus.config.ts b/site/docusaurus.config.ts index 8092ab0cdb..2658c596d0 100644 --- a/site/docusaurus.config.ts +++ b/site/docusaurus.config.ts @@ -116,7 +116,7 @@ const config: Config = { remarkPlugins: [ [require('@docusaurus/theme-mermaid'), {}], ], - lastVersion: '1.1', + lastVersion: '1.2', versions: { current: { label: 'Next', @@ -125,9 +125,14 @@ const config: Config = { // Intentionally indexable: LLMs and search may see the latest // unreleased docs. Only the unmaintained versions are noindexed. }, + '1.2': { + label: '1.2', + path: '/', + banner: 'none' + }, '1.1': { label: '1.1', - path: '/', + path: '1.1', banner: 'none' }, '1.0': { diff --git a/site/versioned_docs/version-1.2/_vars.json b/site/versioned_docs/version-1.2/_vars.json new file mode 100644 index 0000000000..6b883b94d6 --- /dev/null +++ b/site/versioned_docs/version-1.2/_vars.json @@ -0,0 +1,8 @@ +{ + "aigwVersion": "1.2.0", + "aigwGitRef": "main", + "egVersion": "1.9.2", + "egMinVersion": "1.9.2", + "k8sMinVersion": "1.32", + "gaieVersion": "v1.6.2" +} diff --git a/site/versioned_docs/version-1.2/api/api.mdx b/site/versioned_docs/version-1.2/api/api.mdx new file mode 100644 index 0000000000..03621506b7 --- /dev/null +++ b/site/versioned_docs/version-1.2/api/api.mdx @@ -0,0 +1,5323 @@ +--- +id: api_references +title: API Reference +toc_min_heading_level: 2 +toc_max_heading_level: 4 +--- + +## Versions +* [aigateway.envoyproxy.io/v1alpha1](#aigatewayenvoyproxyiov1alpha1) +* [aigateway.envoyproxy.io/v1beta1](#aigatewayenvoyproxyiov1beta1) + + +## aigateway.envoyproxy.io/v1alpha1 + +Package v1alpha1 contains API schema definitions for the aigateway.envoyproxy.io +API group. + + +## Resource Kinds + +### Available Kinds +- [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroute) +- [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutelist) +- [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackend) +- [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendlist) +- [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicy) +- [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicylist) +- [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfig) +- [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfiglist) +- [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproute) +- [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutelist) +- [QuotaPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicy) +- [QuotaPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicylist) + +### Kind Definitions +#### AIGatewayRoute + + + +**Appears in:** +- [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutelist) + +AIGatewayRoute combines multiple AIServiceBackends and attaching them to Gateway(s) resources. + +This serves as a way to define a "unified" AI API for a Gateway which allows downstream +clients to use a single schema API to interact with multiple AI backends. + +Envoy AI Gateway will generate the following k8s resources corresponding to the AIGatewayRoute: + + - HTTPRoute of the Gateway API as a top-level resource to bind all backends. + The name of the HTTPRoute is the same as the AIGatewayRoute. + - HTTPRouteFilter of the Envoy Gateway API per namespace for automatic hostname rewrite. + The name of the HTTPRouteFilter is `ai-eg-host-rewrite-${AIGatewayRoute.Name}`. + +All of these resources are created in the same namespace as the AIGatewayRoute. Note that this is the implementation +detail subject to change. If you want to customize the default behavior of the Envoy AI Gateway, you can use these +resources as a reference and create your own resources. Alternatively, you can use EnvoyPatchPolicy API of the Envoy +Gateway to patch the generated resources. For example, you can configure the retry fallback behavior by attaching +BackendTrafficPolicy API of Envoy Gateway to the generated HTTPRoute. + +##### Fields + + + + + + + + +#### AIGatewayRouteList + + + + +AIGatewayRouteList contains a list of AIGatewayRoute. + +##### Fields + + + + + + + + +#### AIServiceBackend + + + +**Appears in:** +- [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendlist) + +AIServiceBackend is a resource that represents a single backend for AIGatewayRoute. +A backend is a service that handles traffic with a concrete API specification. + +A AIServiceBackend is "attached" to a Backend which is either a k8s Service or a Backend resource of the Envoy Gateway. + +When a backend with an attached AIServiceBackend is used as a routing target in the AIGatewayRoute (more precisely, the +HTTPRouteSpec defined in the AIGatewayRoute), the ai-gateway will generate the necessary configuration to do +the backend specific logic in the final HTTPRoute. + +##### Fields + + + + + + + + +#### AIServiceBackendList + + + + +AIServiceBackendList contains a list of AIServiceBackends. + +##### Fields + + + + + + + + +#### BackendSecurityPolicy + + + +**Appears in:** +- [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicylist) + +BackendSecurityPolicy specifies configuration for authentication and authorization rules on the traffic +exiting the gateway to the backend. + +##### Fields + + + + + + + + +#### BackendSecurityPolicyList + + + + +BackendSecurityPolicyList contains a list of BackendSecurityPolicy + +##### Fields + + + + + + + + +#### GatewayConfig + + + +**Appears in:** +- [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfiglist) + +GatewayConfig provides configuration for the AI Gateway external processor +container that is deployed alongside the Gateway. + +A GatewayConfig is referenced by a Gateway via the annotation +"aigateway.envoyproxy.io/gateway-config". The GatewayConfig must be in the +same namespace as the Gateway that references it. + +This allows gateway-level configuration of the external processor, including +environment variables (e.g., for tracing configuration) and resource requirements. + +Multiple Gateways can reference the same GatewayConfig to share configuration. + +Environment Variable Precedence: +When merging environment variables, the following precedence applies (highest to lowest): + 1. GatewayConfig.Spec.ExtProc.Kubernetes.Env (this resource) + 2. Global controller flags (extProcExtraEnvVars) + +If the same environment variable name exists in both sources, the GatewayConfig +value takes precedence. + +##### Fields + + + + + + + + +#### GatewayConfigList + + + + +GatewayConfigList contains a list of GatewayConfig. + +##### Fields + + + + + + + + +#### MCPRoute + + + +**Appears in:** +- [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutelist) + +MCPRoute defines how to route MCP requests to the backend MCP servers. + +This serves as a way to define a "unified" AI API for a Gateway which allows downstream +clients to use a single schema API to interact with multiple MCP backends. + +##### Fields + + + + + + + + +#### MCPRouteList + + + + +MCPRouteList contains a list of MCPRoute. + +##### Fields + + + + + + + + +#### QuotaPolicy + + + +**Appears in:** +- [QuotaPolicyList](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicylist) + +QuotaPolicy specifies token quota configuration for inference services. +Generates rate limit configuration and tracks quota usage. +Reject requests with 429 once all related quota to that request has been exceeded. + + +##### Fields + + + + + + + + +#### QuotaPolicyList + + + + +QuotaPolicyList contains a list of QuotaPolicy + +##### Fields + + + + + + + + +## Supporting Types + +### Available Types +- [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterule) +- [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulebackendref) +- [AIGatewayRouteRuleMatch](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulematch) +- [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutespec) +- [AIGatewayRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutestatus) +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) +- [AIServiceBackendStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendstatus) +- [APISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-apischema) +- [AWSCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awscredentialsfile) +- [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awsoidcexchangetoken) +- [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-azureoidcexchangetoken) +- [BackendSecurityPolicyAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyapikey) +- [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyawscredentials) +- [BackendSecurityPolicyAnthropicAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyanthropicapikey) +- [BackendSecurityPolicyAzureAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazureapikey) +- [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazurecredentials) +- [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicygcpcredentials) +- [BackendSecurityPolicyOIDC](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyoidc) +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) +- [BackendSecurityPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicystatus) +- [BackendSecurityPolicyType](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicytype) +- [GCPCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpcredentialsfile) +- [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpoidcexchangetoken) +- [GCPServiceAccountImpersonationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpserviceaccountimpersonationconfig) +- [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityfederationconfig) +- [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityprovider) +- [GatewayConfigExtProc](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigextproc) +- [GatewayConfigForwardProxy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigforwardproxy) +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) +- [GatewayConfigStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigstatus) +- [HTTPBodyField](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodyfield) +- [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodymutation) +- [HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpheadermutation) +- [JWKS](#github-com-envoyproxy-ai-gateway-api-v1alpha1-jwks) +- [JWTSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-jwtsource) +- [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcost) +- [LLMRequestCostType](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcosttype) +- [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationsource) +- [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationtarget) +- [MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendapikey) +- [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendsecuritypolicy) +- [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselector) +- [MCPBackendSelectorRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselectorrule) +- [MCPHeaderForward](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpheaderforward) +- [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorization) +- [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorizationrule) +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) +- [MCPRouteBackendTrafficPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendtrafficpolicy) +- [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteoauth) +- [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutesecuritypolicy) +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) +- [MCPRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutestatus) +- [MCPToolFilter](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcptoolfilter) +- [PerModelQuota](#github-com-envoyproxy-ai-gateway-api-v1alpha1-permodelquota) +- [ProtectedResourceMetadata](#github-com-envoyproxy-ai-gateway-api-v1alpha1-protectedresourcemetadata) +- [QuotaBucketMode](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotabucketmode) +- [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) +- [QuotaPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicyspec) +- [QuotaPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicystatus) +- [QuotaRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotarule) +- [QuotaValue](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotavalue) +- [ServiceQuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-servicequotadefinition) +- [ToolCall](#github-com-envoyproxy-ai-gateway-api-v1alpha1-toolcall) +- [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-versionedapischema) + +### Type Definitions +#### AIGatewayRouteRule + + + +**Appears in:** +- [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutespec) + +AIGatewayRouteRule is a rule that defines the routing behavior of the AIGatewayRoute. + +##### Fields + + + + + + +#### AIGatewayRouteRuleBackendRef + + + +**Appears in:** +- [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterule) + +AIGatewayRouteRuleBackendRef is a reference to a backend with a weight. +It can reference either an AIServiceBackend or an InferencePool resource. + +##### Fields + + + + + + +#### AIGatewayRouteRuleMatch + + + +**Appears in:** +- [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterule) + + + +##### Fields + + + + + + +#### AIGatewayRouteSpec + + + +**Appears in:** +- [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroute) + +AIGatewayRouteSpec details the AIGatewayRoute configuration. + +##### Fields + + + + + + +#### AIGatewayRouteStatus + + + +**Appears in:** +- [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroute) + +AIGatewayRouteStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### AIServiceBackendSpec + + + +**Appears in:** +- [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackend) + +AIServiceBackendSpec details the AIServiceBackend configuration. + +##### Fields + + + + + + +#### AIServiceBackendStatus + + + +**Appears in:** +- [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackend) + +AIServiceBackendStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### APISchema + +**Underlying type:** string + +**Appears in:** +- [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1alpha1-versionedapischema) + +APISchema defines the API schema. + + + +##### Possible Values + + +#### AWSCredentialsFile + + + +**Appears in:** +- [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyawscredentials) + +AWSCredentialsFile specifies the credentials file to use for the AWS provider. +Envoy reads the secret file, and the profile to use is specified by the Profile field. + +##### Fields + + + + + + +#### AWSOIDCExchangeToken + + + +**Appears in:** +- [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyawscredentials) + +AWSOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. +For AWS, the controller will query STS to obtain AWS AccessKeyId, SecretAccessKey, and SessionToken, +and store them in a temporary credentials file. + +##### Fields + + + + + + +#### AzureOIDCExchangeToken + + + +**Appears in:** +- [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyazurecredentials) + +AzureOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. +For Azure, the controller will query Azure Entra ID to get an Azure Access Token, +and store them in a secret. + +##### Fields + + + + + + +#### BackendSecurityPolicyAPIKey + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) + +BackendSecurityPolicyAPIKey specifies the API key. + +##### Fields + + + + + + +#### BackendSecurityPolicyAWSCredentials + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) + +BackendSecurityPolicyAWSCredentials contains the supported authentication mechanisms to access AWS. + +When neither CredentialsFile nor OIDCExchangeToken is specified, the AWS SDK's default credential chain +will be used. This automatically supports various authentication methods in the following order: + 1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) + 2. EKS Pod Identity - automatically rotates credentials for pods in EKS clusters + 3. IAM Roles for Service Accounts (IRSA) - injects credentials via mounted service account tokens + 4. EC2 instance metadata (IAM instance roles) + 5. ECS task roles + +The default credential chain is recommended for Kubernetes deployments as it supports automatic +credential rotation without manual configuration. Credentials are refreshed automatically when +they approach expiration (typically hourly for IRSA and Pod Identity). + +##### Fields + + + + + + +#### BackendSecurityPolicyAnthropicAPIKey + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) + +BackendSecurityPolicyAnthropicAPIKey specifies the Anthropic API key. + +##### Fields + + + + + + +#### BackendSecurityPolicyAzureAPIKey + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) + +BackendSecurityPolicyAzureAPIKey specifies the Azure OpenAI API key. + +##### Fields + + + + + + +#### BackendSecurityPolicyAzureCredentials + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) + +BackendSecurityPolicyAzureCredentials contains the supported authentication mechanisms to access Azure. +Only one of ClientSecretRef or OIDCExchangeToken must be specified. Credentials will not be generated if +neither are set. + +##### Fields + + + + + + +#### BackendSecurityPolicyGCPCredentials + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) + +BackendSecurityPolicyGCPCredentials contains the supported authentication mechanisms to access GCP. + +##### Fields + + + + + + +#### BackendSecurityPolicyOIDC + + + +**Appears in:** +- [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-awsoidcexchangetoken) +- [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-azureoidcexchangetoken) +- [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpoidcexchangetoken) +- [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityprovider) + +BackendSecurityPolicyOIDC specifies OIDC related fields. + +##### Fields + + + + + + +#### BackendSecurityPolicySpec + + + +**Appears in:** +- [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicy) + +BackendSecurityPolicySpec specifies authentication rules on access the provider from the Gateway. +Only one mechanism to access a backend(s) can be specified. + +Only one type of BackendSecurityPolicy can be defined. + +##### Fields + + + + + + +#### BackendSecurityPolicyStatus + + + +**Appears in:** +- [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicy) + +BackendSecurityPolicyStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### BackendSecurityPolicyType + +**Underlying type:** string + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicyspec) + +BackendSecurityPolicyType specifies the type of auth mechanism used to access a backend. + + + +##### Possible Values + + +#### GCPCredentialsFile + + + +**Appears in:** +- [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicygcpcredentials) + +GCPCredentialsFile specifies the service account key json file to authenticate with GCP provider. + +##### Fields + + + + + + +#### GCPOIDCExchangeToken + + + +**Appears in:** +- [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityfederationconfig) + + + +##### Fields + + + + + + +#### GCPServiceAccountImpersonationConfig + + + +**Appears in:** +- [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gcpworkloadidentityfederationconfig) + + + +##### Fields + + + + + + +#### GCPWorkloadIdentityFederationConfig + + + +**Appears in:** +- [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1alpha1-backendsecuritypolicygcpcredentials) + + + +##### Fields + + + + + + + +#### GatewayConfigExtProc + + + +**Appears in:** +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) + +GatewayConfigExtProc holds runtime-specific configuration for the external processor. + +##### Fields + + + + + + +#### GatewayConfigForwardProxy + + + +**Appears in:** +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) + +GatewayConfigForwardProxy configures an HTTP CONNECT forward proxy for upstream egress. + +##### Fields + + + + + + +#### GatewayConfigSpec + + + +**Appears in:** +- [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfig) + +GatewayConfigSpec defines the configuration for the AI Gateway. + +##### Fields + + + + + + +#### GatewayConfigStatus + + + +**Appears in:** +- [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfig) + +GatewayConfigStatus defines the observed state of GatewayConfig. + +##### Fields + + + + + + +#### HTTPBodyField + + + +**Appears in:** +- [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1alpha1-httpbodymutation) + +HTTPBodyField represents a JSON field name and value for body mutation + +##### Fields + + + + + + +#### HTTPBodyMutation + + + +**Appears in:** +- [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulebackendref) +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) + +HTTPBodyMutation defines the mutation of HTTP request body JSON fields that will be applied to the request + +##### Fields + + + + + + +#### HTTPHeaderMutation + + + +**Appears in:** +- [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayrouterulebackendref) +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) + +HTTPHeaderMutation defines the mutation of HTTP headers that will be applied to the request + +##### Fields + + + + + + +#### JWKS + + + +**Appears in:** +- [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteoauth) + +JWKS defines how to obtain JSON Web Key Sets (JWKS) either from a remote HTTP/HTTPS endpoint or from a local source. + +##### Fields + + + + + + +#### JWTSource + + + +**Appears in:** +- [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationsource) + +JWTSource defines the MCP authorization source for JWT tokens. +At least one of scopes or claims must be provided. +Scopes and claims are AND-ed: when both are specified, both sets must match. + +##### Fields + + + + + + +#### LLMRequestCost + + + +**Appears in:** +- [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aigatewayroutespec) +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-gatewayconfigspec) + +LLMRequestCost configures each request cost. + +##### Fields + + + + + + +#### LLMRequestCostType + +**Underlying type:** string + +**Appears in:** +- [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1alpha1-llmrequestcost) + +LLMRequestCostType specifies the type of the LLMRequestCost. + + + +##### Possible Values + + +#### MCPAuthorizationSource + + + +**Appears in:** +- [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorizationrule) + +MCPAuthorizationSource defines the source of an authorization rule. + +##### Fields + + + + + + +#### MCPAuthorizationTarget + + + +**Appears in:** +- [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorizationrule) + +MCPAuthorizationTarget defines the target of an authorization rule. + +##### Fields + + + + + + +#### MCPBackendAPIKey + + + +**Appears in:** +- [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendsecuritypolicy) + +MCPBackendAPIKey defines the configuration for the API Key Authentication to a backend. +When both `header` and `queryParam` are unspecified, the API key will be injected into the "Authorization" header by default. + +##### Fields + + + + + + +#### MCPBackendSecurityPolicy + + + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) + +MCPBackendSecurityPolicy defines the security policy for a backend MCP server. + +##### Fields + + + + + + +#### MCPBackendSelector + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) + +MCPBackendSelector defines which of a MCPRoute's backends a request may fan out to. + +##### Fields + + + + + + +#### MCPBackendSelectorRule + + + +**Appears in:** +- [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpbackendselector) + +MCPBackendSelectorRule defines a single backend selection rule. + +##### Fields + + + + + + +#### MCPHeaderForward + + + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) + +MCPHeaderForward specifies a header to extract from the incoming request and forward to a backend. + +##### Fields + + + + + + +#### MCPRouteAuthorization + + + +**Appears in:** +- [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutesecuritypolicy) + +MCPRouteAuthorization defines the authorization configuration for a MCPRoute. + +##### Fields + + + + + + +#### MCPRouteAuthorizationRule + + + +**Appears in:** +- [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteauthorization) + +MCPRouteAuthorizationRule defines an authorization rule for MCPRoute based on the MCP authorization spec. +Reference: https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling + +##### Fields + + + + + + +#### MCPRouteBackendRef + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) + +MCPRouteBackendRef wraps a EG's BackendObjectReference to reference an MCP server. + +##### Fields + + + + + + +#### MCPRouteBackendTrafficPolicy + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) + +MCPRouteBackendTrafficPolicy configures how the Envoy Gateway BackendTrafficPolicy +resources auto-generated for a MCPRoute interact with other BackendTrafficPolicy +resources (e.g. an operator-defined rate limiting policy) targeting the same HTTPRoute. + +##### Fields + + + + + + +#### MCPRouteOAuth + + + +**Appears in:** +- [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutesecuritypolicy) + +MCPRouteOAuth defines a MCP spec compatible OAuth authentication configuration for a MCPRoute. +Reference: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization + +##### Fields + + + + + + +#### MCPRouteSecurityPolicy + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutespec) + +MCPRouteSecurityPolicy defines the security policy for a MCPRoute. + +##### Fields + + + + + + +#### MCPRouteSpec + + + +**Appears in:** +- [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproute) + +MCPRouteSpec details the MCPRoute configuration. + +##### Fields + + + + + + +#### MCPRouteStatus + + + +**Appears in:** +- [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproute) + +MCPRouteStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### MCPToolFilter + + + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcproutebackendref) + +MCPToolFilter filters tools using include and exclude patterns with exact matches or regular expressions. +Exclude rules take precedence over include rules (deny-wins). When both include and exclude are specified, +a tool must match an include rule AND not match any exclude rule to be allowed. + +##### Fields + + + + + + +#### PerModelQuota + + + +**Appears in:** +- [QuotaPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicyspec) + + + +##### Fields + + + + + + +#### ProtectedResourceMetadata + + + +**Appears in:** +- [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcprouteoauth) + +ProtectedResourceMetadata represents the Protected Resource Metadata of the MCP server as per RFC 9728. + +References: +* https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#authorization-server-location +* https://datatracker.ietf.org/doc/html/rfc9728#name-protected-resource-metadata + +##### Fields + + + + + + +#### QuotaBucketMode + +**Underlying type:** string + +**Appears in:** +- [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) + +QuotaBucketMode specifies whether the default and per request buckets values are exclusive or inclusive. + + + + +##### Possible Values + + +#### QuotaDefinition + + + +**Appears in:** +- [PerModelQuota](#github-com-envoyproxy-ai-gateway-api-v1alpha1-permodelquota) + +QuotaDefinition specified expression for computing request cost and rules for matching requests to quota buckets. + +##### Fields + + + + + + +#### QuotaPolicySpec + + + +**Appears in:** +- [QuotaPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicy) + +QuotaPolicySpec specifies rules for computing token based costs of requests. + +##### Fields + + + + + + +#### QuotaPolicyStatus + + + +**Appears in:** +- [QuotaPolicy](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicy) + +QuotaPolicyStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### QuotaRule + + + +**Appears in:** +- [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) + + + +##### Fields + + + + + + +#### QuotaValue + + + +**Appears in:** +- [QuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotadefinition) +- [QuotaRule](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotarule) +- [ServiceQuotaDefinition](#github-com-envoyproxy-ai-gateway-api-v1alpha1-servicequotadefinition) + +QuotaValue defines the quota limits using sliding window. + +##### Fields + + + + + + +#### ServiceQuotaDefinition + + + +**Appears in:** +- [QuotaPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-quotapolicyspec) + + + +##### Fields + + + + + + +#### ToolCall + + + +**Appears in:** +- [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1alpha1-mcpauthorizationtarget) + +ToolCall represents a tool call in the MCP authorization target. + +##### Fields + + + + + + +#### VersionedAPISchema + + + +**Appears in:** +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1alpha1-aiservicebackendspec) + +VersionedAPISchema defines the API schema of either AIGatewayRoute (the input) or AIServiceBackend (the output). + +This allows the ai-gateway to understand the input and perform the necessary transformation +depending on the API schema pair (input, output). + +Note that this is vendor specific, and the stability of the API schema is not guaranteed by +the ai-gateway, but by the vendor via proper versioning. + +##### Fields + + + + + + + +## aigateway.envoyproxy.io/v1beta1 + +Package v1beta1 contains API schema definitions for the aigateway.envoyproxy.io +API group. This is the beta version of the API, preferred over v1alpha1 for new resources. + + +## Resource Kinds + +### Available Kinds +- [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroute) +- [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutelist) +- [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackend) +- [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendlist) +- [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicy) +- [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicylist) +- [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfig) +- [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfiglist) +- [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproute) +- [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutelist) + +### Kind Definitions +#### AIGatewayRoute + + + +**Appears in:** +- [AIGatewayRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutelist) + +AIGatewayRoute combines multiple AIServiceBackends and attaching them to Gateway(s) resources. + +This serves as a way to define a "unified" AI API for a Gateway which allows downstream +clients to use a single schema API to interact with multiple AI backends. + +Envoy AI Gateway will generate the following k8s resources corresponding to the AIGatewayRoute: + + - HTTPRoute of the Gateway API as a top-level resource to bind all backends. + The name of the HTTPRoute is the same as the AIGatewayRoute. + - HTTPRouteFilter of the Envoy Gateway API per namespace for automatic hostname rewrite. + The name of the HTTPRouteFilter is `ai-eg-host-rewrite-${AIGatewayRoute.Name}`. + +All of these resources are created in the same namespace as the AIGatewayRoute. Note that this is the implementation +detail subject to change. If you want to customize the default behavior of the Envoy AI Gateway, you can use these +resources as a reference and create your own resources. Alternatively, you can use EnvoyPatchPolicy API of the Envoy +Gateway to patch the generated resources. For example, you can configure the retry fallback behavior by attaching +BackendTrafficPolicy API of Envoy Gateway to the generated HTTPRoute. + +##### Fields + + + + + + + + +#### AIGatewayRouteList + + + + +AIGatewayRouteList contains a list of AIGatewayRoute. + +##### Fields + + + + + + + + +#### AIServiceBackend + + + +**Appears in:** +- [AIServiceBackendList](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendlist) + +AIServiceBackend is a resource that represents a single backend for AIGatewayRoute. +A backend is a service that handles traffic with a concrete API specification. + +A AIServiceBackend is "attached" to a Backend which is either a k8s Service or a Backend resource of the Envoy Gateway. + +When a backend with an attached AIServiceBackend is used as a routing target in the AIGatewayRoute (more precisely, the +HTTPRouteSpec defined in the AIGatewayRoute), the ai-gateway will generate the necessary configuration to do +the backend specific logic in the final HTTPRoute. + +##### Fields + + + + + + + + +#### AIServiceBackendList + + + + +AIServiceBackendList contains a list of AIServiceBackends. + +##### Fields + + + + + + + + +#### BackendSecurityPolicy + + + +**Appears in:** +- [BackendSecurityPolicyList](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicylist) + +BackendSecurityPolicy specifies configuration for authentication and authorization rules on the traffic +exiting the gateway to the backend. + +##### Fields + + + + + + + + +#### BackendSecurityPolicyList + + + + +BackendSecurityPolicyList contains a list of BackendSecurityPolicy + +##### Fields + + + + + + + + +#### GatewayConfig + + + +**Appears in:** +- [GatewayConfigList](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfiglist) + +GatewayConfig provides configuration for the AI Gateway external processor +container that is deployed alongside the Gateway. + +A GatewayConfig is referenced by a Gateway via the annotation +"aigateway.envoyproxy.io/gateway-config". The GatewayConfig must be in the +same namespace as the Gateway that references it. + +This allows gateway-level configuration of the external processor, including +environment variables (e.g., for tracing configuration) and resource requirements. + +Multiple Gateways can reference the same GatewayConfig to share configuration. + +Environment Variable Precedence: +When merging environment variables, the following precedence applies (highest to lowest): + 1. GatewayConfig.Spec.ExtProc.Kubernetes.Env (this resource) + 2. Global controller flags (extProcExtraEnvVars) + +If the same environment variable name exists in both sources, the GatewayConfig +value takes precedence. + +##### Fields + + + + + + + + +#### GatewayConfigList + + + + +GatewayConfigList contains a list of GatewayConfig. + +##### Fields + + + + + + + + +#### MCPRoute + + + +**Appears in:** +- [MCPRouteList](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutelist) + +MCPRoute defines how to route MCP requests to the backend MCP servers. + +This serves as a way to define a "unified" AI API for a Gateway which allows downstream +clients to use a single schema API to interact with multiple MCP backends. + +##### Fields + + + + + + + + +#### MCPRouteList + + + + +MCPRouteList contains a list of MCPRoute. + +##### Fields + + + + + + + + +## Supporting Types + +### Available Types +- [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterule) +- [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulebackendref) +- [AIGatewayRouteRuleMatch](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulematch) +- [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutespec) +- [AIGatewayRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutestatus) +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) +- [AIServiceBackendStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendstatus) +- [APISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-apischema) +- [AWSCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1beta1-awscredentialsfile) +- [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-awsoidcexchangetoken) +- [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-azureoidcexchangetoken) +- [BackendSecurityPolicyAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyapikey) +- [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyawscredentials) +- [BackendSecurityPolicyAnthropicAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyanthropicapikey) +- [BackendSecurityPolicyAzureAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazureapikey) +- [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazurecredentials) +- [BackendSecurityPolicyCredentialOverride](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicycredentialoverride) +- [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicygcpcredentials) +- [BackendSecurityPolicyOIDC](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyoidc) +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) +- [BackendSecurityPolicyStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicystatus) +- [BackendSecurityPolicyType](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicytype) +- [CredentialOverrideFromDynamicMetadata](#github-com-envoyproxy-ai-gateway-api-v1beta1-credentialoverridefromdynamicmetadata) +- [CredentialOverrideFromRequestHeaders](#github-com-envoyproxy-ai-gateway-api-v1beta1-credentialoverridefromrequestheaders) +- [GCPCredentialsFile](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpcredentialsfile) +- [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpoidcexchangetoken) +- [GCPServiceAccountImpersonationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpserviceaccountimpersonationconfig) +- [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityfederationconfig) +- [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityprovider) +- [GatewayConfigExtProc](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigextproc) +- [GatewayConfigForwardProxy](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigforwardproxy) +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) +- [GatewayConfigStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigstatus) +- [HTTPBodyField](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodyfield) +- [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodymutation) +- [HTTPHeaderMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadermutation) +- [HTTPHeaderValueFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefilter) +- [HTTPHeaderValueFilterMode](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefiltermode) +- [JWKS](#github-com-envoyproxy-ai-gateway-api-v1beta1-jwks) +- [JWTSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-jwtsource) +- [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcost) +- [LLMRequestCostType](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcosttype) +- [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationsource) +- [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationtarget) +- [MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikey) +- [MCPBackendAPIKeyInjectionPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikeyinjectionpolicy) +- [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendsecuritypolicy) +- [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselector) +- [MCPBackendSelectorRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselectorrule) +- [MCPHeaderForward](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpheaderforward) +- [MCPPromptFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcppromptfilter) +- [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorization) +- [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorizationrule) +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) +- [MCPRouteBackendTrafficPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendtrafficpolicy) +- [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteoauth) +- [MCPRoutePrefixMode](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteprefixmode) +- [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutesecuritypolicy) +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) +- [MCPRouteStatus](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutestatus) +- [MCPToolFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcptoolfilter) +- [ProtectedResourceMetadata](#github-com-envoyproxy-ai-gateway-api-v1beta1-protectedresourcemetadata) +- [ToolCall](#github-com-envoyproxy-ai-gateway-api-v1beta1-toolcall) +- [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-versionedapischema) + +### Type Definitions +#### AIGatewayRouteRule + + + +**Appears in:** +- [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutespec) + +AIGatewayRouteRule is a rule that defines the routing behavior of the AIGatewayRoute. + +##### Fields + + + + + + +#### AIGatewayRouteRuleBackendRef + + + +**Appears in:** +- [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterule) + +AIGatewayRouteRuleBackendRef is a reference to a backend with a weight. +It can reference either an AIServiceBackend or an InferencePool resource. + +##### Fields + + + + + + +#### AIGatewayRouteRuleMatch + + + +**Appears in:** +- [AIGatewayRouteRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterule) + + + +##### Fields + + + + + + +#### AIGatewayRouteSpec + + + +**Appears in:** +- [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroute) + +AIGatewayRouteSpec details the AIGatewayRoute configuration. + +##### Fields + + + + + + +#### AIGatewayRouteStatus + + + +**Appears in:** +- [AIGatewayRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroute) + +AIGatewayRouteStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### AIServiceBackendSpec + + + +**Appears in:** +- [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackend) + +AIServiceBackendSpec details the AIServiceBackend configuration. + +##### Fields + + + + + + +#### AIServiceBackendStatus + + + +**Appears in:** +- [AIServiceBackend](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackend) + +AIServiceBackendStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### APISchema + +**Underlying type:** string + +**Appears in:** +- [VersionedAPISchema](#github-com-envoyproxy-ai-gateway-api-v1beta1-versionedapischema) + +APISchema defines the API schema. + + + +##### Possible Values + + +#### AWSCredentialsFile + + + +**Appears in:** +- [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyawscredentials) + +AWSCredentialsFile specifies the credentials file to use for the AWS provider. +Envoy reads the secret file, and the profile to use is specified by the Profile field. + +##### Fields + + + + + + +#### AWSOIDCExchangeToken + + + +**Appears in:** +- [BackendSecurityPolicyAWSCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyawscredentials) + +AWSOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. +For AWS, the controller will query STS to obtain AWS AccessKeyId, SecretAccessKey, and SessionToken, +and store them in a temporary credentials file. + +##### Fields + + + + + + +#### AzureOIDCExchangeToken + + + +**Appears in:** +- [BackendSecurityPolicyAzureCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyazurecredentials) + +AzureOIDCExchangeToken specifies credentials to obtain oidc token from a sso server. +For Azure, the controller will query Azure Entra ID to get an Azure Access Token, +and store them in a secret. + +##### Fields + + + + + + +#### BackendSecurityPolicyAPIKey + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyAPIKey specifies the API key. + +##### Fields + + + + + + +#### BackendSecurityPolicyAWSCredentials + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyAWSCredentials contains the supported authentication mechanisms to access AWS. + +When neither CredentialsFile nor OIDCExchangeToken is specified, the AWS SDK's default credential chain +will be used. This automatically supports various authentication methods in the following order: + 1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) + 2. EKS Pod Identity - automatically rotates credentials for pods in EKS clusters + 3. IAM Roles for Service Accounts (IRSA) - injects credentials via mounted service account tokens + 4. EC2 instance metadata (IAM instance roles) + 5. ECS task roles + +The default credential chain is recommended for Kubernetes deployments as it supports automatic +credential rotation without manual configuration. Credentials are refreshed automatically when +they approach expiration (typically hourly for IRSA and Pod Identity). + +##### Fields + + + + + + +#### BackendSecurityPolicyAnthropicAPIKey + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyAnthropicAPIKey specifies the Anthropic API key. + +##### Fields + + + + + + +#### BackendSecurityPolicyAzureAPIKey + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyAzureAPIKey specifies the Azure OpenAI API key. + +##### Fields + + + + + + +#### BackendSecurityPolicyAzureCredentials + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyAzureCredentials contains the supported authentication mechanisms to access Azure. +Only one of ClientSecretRef or OIDCExchangeToken must be specified. Credentials will not be generated if +neither are set. + +##### Fields + + + + + + +#### BackendSecurityPolicyCredentialOverride + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyCredentialOverride configures per-request credential sourcing. +A trusted upstream filter (ext_authz or a preceding ext_proc) resolves the caller's +credential and delivers it via one of the two sources below. +Exactly one source must be configured. + +##### Fields + + + + + + +#### BackendSecurityPolicyGCPCredentials + + + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyGCPCredentials contains the supported authentication mechanisms to access GCP. + +When neither CredentialsFile nor WorkloadIdentityFederationConfig is specified, Application Default +Credentials (ADC) will be used. This supports GKE Workload Identity, environment variables, and +default service account credentials when running on GCP. + +##### Fields + + + + + + +#### BackendSecurityPolicyOIDC + + + +**Appears in:** +- [AWSOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-awsoidcexchangetoken) +- [AzureOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-azureoidcexchangetoken) +- [GCPOIDCExchangeToken](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpoidcexchangetoken) +- [GCPWorkloadIdentityProvider](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityprovider) + +BackendSecurityPolicyOIDC specifies OIDC related fields. + +##### Fields + + + + + + +#### BackendSecurityPolicySpec + + + +**Appears in:** +- [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicy) + +BackendSecurityPolicySpec specifies authentication rules on access the provider from the Gateway. +Only one mechanism to access a backend(s) can be specified. + +Only one type of BackendSecurityPolicy can be defined. + +##### Fields + + + + + + +#### BackendSecurityPolicyStatus + + + +**Appears in:** +- [BackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicy) + +BackendSecurityPolicyStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### BackendSecurityPolicyType + +**Underlying type:** string + +**Appears in:** +- [BackendSecurityPolicySpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicyspec) + +BackendSecurityPolicyType specifies the type of auth mechanism used to access a backend. + + + +##### Possible Values + + +#### CredentialOverrideFromDynamicMetadata + + + +**Appears in:** +- [BackendSecurityPolicyCredentialOverride](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicycredentialoverride) + +CredentialOverrideFromDynamicMetadata sources the per-request credential from Envoy +dynamic metadata produced by a trusted filter running earlier in the filter chain. + +##### Fields + + + + + + +#### CredentialOverrideFromRequestHeaders + + + +**Appears in:** +- [BackendSecurityPolicyCredentialOverride](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicycredentialoverride) + +CredentialOverrideFromRequestHeaders sources the per-request credential from a request header. + +##### Fields + + + + + + +#### GCPCredentialsFile + + + +**Appears in:** +- [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicygcpcredentials) + +GCPCredentialsFile specifies the service account key json file to authenticate with GCP provider. + +##### Fields + + + + + + +#### GCPOIDCExchangeToken + + + +**Appears in:** +- [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityfederationconfig) + + + +##### Fields + + + + + + +#### GCPServiceAccountImpersonationConfig + + + +**Appears in:** +- [GCPWorkloadIdentityFederationConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gcpworkloadidentityfederationconfig) + + + +##### Fields + + + + + + +#### GCPWorkloadIdentityFederationConfig + + + +**Appears in:** +- [BackendSecurityPolicyGCPCredentials](#github-com-envoyproxy-ai-gateway-api-v1beta1-backendsecuritypolicygcpcredentials) + + + +##### Fields + + + + + + + +#### GatewayConfigExtProc + + + +**Appears in:** +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) + +GatewayConfigExtProc holds configuration for the external processor. + +##### Fields + + + + + + +#### GatewayConfigForwardProxy + + + +**Appears in:** +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) + +GatewayConfigForwardProxy configures an HTTP CONNECT forward proxy for upstream egress. + +##### Fields + + + + + + +#### GatewayConfigSpec + + + +**Appears in:** +- [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfig) + +GatewayConfigSpec defines the configuration for the AI Gateway. + +##### Fields + + + + + + +#### GatewayConfigStatus + + + +**Appears in:** +- [GatewayConfig](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfig) + +GatewayConfigStatus defines the observed state of GatewayConfig. + +##### Fields + + + + + + +#### HTTPBodyField + + + +**Appears in:** +- [HTTPBodyMutation](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpbodymutation) + +HTTPBodyField represents a JSON field name and value for body mutation + +##### Fields + + + + + + +#### HTTPBodyMutation + + + +**Appears in:** +- [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulebackendref) +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) + +HTTPBodyMutation defines the mutation of HTTP request body JSON fields that will be applied to the request + +##### Fields + + + + + + +#### HTTPHeaderMutation + + + +**Appears in:** +- [AIGatewayRouteRuleBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayrouterulebackendref) +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) + +HTTPHeaderMutation defines the mutation of HTTP headers that will be applied to the request + +##### Fields + + + + + + +#### HTTPHeaderValueFilter + + + +**Appears in:** +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) + +HTTPHeaderValueFilter filters individual values out of a multi-valued request header before the +request is forwarded to this backend. + +This drops (or restricts to) specific values from a comma-separated header while forwarding the +rest untouched, so one unsupported value does not fail the whole request. + +Only comma-delimited headers are supported. Values are matched exactly, after surrounding +whitespace is trimmed. To remove a header entirely, use HeaderMutation.Remove instead. + +##### Fields + + + + + + +#### HTTPHeaderValueFilterMode + +**Underlying type:** string + +**Appears in:** +- [HTTPHeaderValueFilter](#github-com-envoyproxy-ai-gateway-api-v1beta1-httpheadervaluefilter) + +HTTPHeaderValueFilterMode specifies the filtering semantics of an HTTPHeaderValueFilter. + + + +##### Possible Values + + +#### JWKS + + + +**Appears in:** +- [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteoauth) + +JWKS defines how to obtain JSON Web Key Sets (JWKS) either from a remote HTTP/HTTPS endpoint or from a local source. + +##### Fields + + + + + + +#### JWTSource + + + +**Appears in:** +- [MCPAuthorizationSource](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationsource) + +JWTSource defines the MCP authorization source for JWT tokens. +At least one of scopes or claims must be provided. +Scopes and claims are AND-ed: when both are specified, both sets must match. + +##### Fields + + + + + + +#### LLMRequestCost + + + +**Appears in:** +- [AIGatewayRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aigatewayroutespec) +- [GatewayConfigSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-gatewayconfigspec) + +LLMRequestCost configures each request cost. + +##### Fields + + + + + + +#### LLMRequestCostType + +**Underlying type:** string + +**Appears in:** +- [LLMRequestCost](#github-com-envoyproxy-ai-gateway-api-v1beta1-llmrequestcost) + +LLMRequestCostType specifies the type of the LLMRequestCost. + + + +##### Possible Values + + +#### MCPAuthorizationSource + + + +**Appears in:** +- [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorizationrule) + +MCPAuthorizationSource defines the source of an authorization rule. + +##### Fields + + + + + + +#### MCPAuthorizationTarget + + + +**Appears in:** +- [MCPRouteAuthorizationRule](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorizationrule) + +MCPAuthorizationTarget defines the target of an authorization rule. + +##### Fields + + + + + + +#### MCPBackendAPIKey + + + +**Appears in:** +- [MCPBackendSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendsecuritypolicy) + +MCPBackendAPIKey defines the configuration for the API Key Authentication to a backend. +When both `header` and `queryParam` are unspecified, the API key will be injected into the "Authorization" header by default. + +##### Fields + + + + + + +#### MCPBackendAPIKeyInjectionPolicy + +**Underlying type:** string + +**Appears in:** +- [MCPBackendAPIKey](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendapikey) + +MCPBackendAPIKeyInjectionPolicy controls when the configured API key is written onto the request. + + + +##### Possible Values + + +#### MCPBackendSecurityPolicy + + + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) + +MCPBackendSecurityPolicy defines the security policy for a backend MCP server. + +##### Fields + + + + + + +#### MCPBackendSelector + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) + +MCPBackendSelector defines which of a MCPRoute's backends a request may fan out to. + +##### Fields + + + + + + +#### MCPBackendSelectorRule + + + +**Appears in:** +- [MCPBackendSelector](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpbackendselector) + +MCPBackendSelectorRule defines a single backend selection rule. + +##### Fields + + + + + + +#### MCPHeaderForward + + + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) + +MCPHeaderForward specifies a header to extract from the incoming request and forward to a backend. + +##### Fields + + + + + + +#### MCPPromptFilter + + + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) + +MCPPromptFilter filters prompts using include and exclude patterns with exact matches or regular expressions. +Exclude rules take precedence over include rules (deny-wins). When both include and exclude are specified, +a prompt must match an include rule AND not match any exclude rule to be allowed. + +##### Fields + + + + + + +#### MCPRouteAuthorization + + + +**Appears in:** +- [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutesecuritypolicy) + +MCPRouteAuthorization defines the authorization configuration for a MCPRoute. + +##### Fields + + + + + + +#### MCPRouteAuthorizationRule + + + +**Appears in:** +- [MCPRouteAuthorization](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteauthorization) + +MCPRouteAuthorizationRule defines an authorization rule for MCPRoute based on the MCP authorization spec. +Reference: https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling + +##### Fields + + + + + + +#### MCPRouteBackendRef + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) + +MCPRouteBackendRef wraps a EG's BackendObjectReference to reference an MCP server. + +##### Fields + + + + + + +#### MCPRouteBackendTrafficPolicy + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) + +MCPRouteBackendTrafficPolicy configures how the Envoy Gateway BackendTrafficPolicy +resources auto-generated for a MCPRoute interact with other BackendTrafficPolicy +resources (e.g. an operator-defined rate limiting policy) targeting the same HTTPRoute. + +##### Fields + + + + + + +#### MCPRouteOAuth + + + +**Appears in:** +- [MCPRouteSecurityPolicy](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutesecuritypolicy) + +MCPRouteOAuth defines a MCP spec compatible OAuth authentication configuration for a MCPRoute. +Reference: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization + +##### Fields + + + + + + +#### MCPRoutePrefixMode + +**Underlying type:** string + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) + +MCPRoutePrefixMode controls how tool and prompt names are prefixed when exposed to clients. + +PrefixMode can be set at the route level (MCPRouteSpec) and overridden per backend +(MCPRouteBackendRef); the per-backend value takes precedence when set. A single route can +therefore mix Always and Never backends. This is intentional: it lets an operator keep the +route-level default (Always is recommended for most routes, since it guarantees a collision-free +catalog without any extra configuration) while opting specific, well-known backends into bare +names, e.g. to match tool names an existing client or interactive MCP App already hardcodes. +Per-backend override is meant as a targeted escape hatch, not a replacement for choosing one +naming policy for the route as a whole. + + + +##### Possible Values + + +#### MCPRouteSecurityPolicy + + + +**Appears in:** +- [MCPRouteSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutespec) + +MCPRouteSecurityPolicy defines the security policy for a MCPRoute. + +##### Fields + + + + + + +#### MCPRouteSpec + + + +**Appears in:** +- [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproute) + +MCPRouteSpec details the MCPRoute configuration. + +##### Fields + + + + + + +#### MCPRouteStatus + + + +**Appears in:** +- [MCPRoute](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproute) + +MCPRouteStatus contains the conditions by the reconciliation result. + +##### Fields + + + + + + +#### MCPToolFilter + + + +**Appears in:** +- [MCPRouteBackendRef](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcproutebackendref) + +MCPToolFilter filters tools using include and exclude patterns with exact matches or regular expressions. +Exclude rules take precedence over include rules (deny-wins). When both include and exclude are specified, +a tool must match an include rule AND not match any exclude rule to be allowed. + +##### Fields + + + + + + +#### ProtectedResourceMetadata + + + +**Appears in:** +- [MCPRouteOAuth](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcprouteoauth) + +ProtectedResourceMetadata represents the Protected Resource Metadata of the MCP server as per RFC 9728. + +References: +* https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#authorization-server-location +* https://datatracker.ietf.org/doc/html/rfc9728#name-protected-resource-metadata + +##### Fields + + + + + + +#### ToolCall + + + +**Appears in:** +- [MCPAuthorizationTarget](#github-com-envoyproxy-ai-gateway-api-v1beta1-mcpauthorizationtarget) + +ToolCall represents a tool call in the MCP authorization target. + +##### Fields + + + + + + +#### VersionedAPISchema + + + +**Appears in:** +- [AIServiceBackendSpec](#github-com-envoyproxy-ai-gateway-api-v1beta1-aiservicebackendspec) + +VersionedAPISchema defines the API schema of either AIGatewayRoute (the input) or AIServiceBackend (the output). + +This allows the ai-gateway to understand the input and perform the necessary transformation +depending on the API schema pair (input, output). + +Note that this is vendor specific, and the stability of the API schema is not guaranteed by +the ai-gateway, but by the vendor via proper versioning. + +##### Fields + + + + + + diff --git a/site/versioned_docs/version-1.2/capabilities/a2a/index.md b/site/versioned_docs/version-1.2/capabilities/a2a/index.md new file mode 100644 index 0000000000..198a15b31d --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/a2a/index.md @@ -0,0 +1,337 @@ +--- +id: a2a +title: Agent2Agent (A2A) Traffic +sidebar_position: 9 +--- + +:::caution Preview +Agent Router does not yet have a dedicated A2A route type. This page shows how to route [A2A](https://a2a-protocol.org/) agents through the gateway **today**, using only Envoy Gateway resources plus the `envoy.filters.http.a2a` filter that ships inside Envoy Proxy. The filter is alpha in Envoy and the recipe relies on `EnvoyPatchPolicy`, so treat this as a preview. A first-class `A2ARoute` API is being discussed in [issue #2070](https://github.com/theagentrouter/agent-router/issues/2070). +::: + +:::info Help shape A2A support +Early adopters of A2A can already configure Envoy through Agent Router to handle A2A traffic with the approach on this page. The community is working on first-class A2A features and CRDs. If you want to help, or just want to tell us what your agents need, join the discussion on [GitHub](https://github.com/theagentrouter/agent-router/issues/2070), the [community Discord](https://discord.gg/xuxtPq43gZ), or the [weekly community meeting](https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w) held every Monday. +::: + +## What you get + +Following this page, an A2A agent behind the gateway gets: + +- **Routing** for the agent card (`GET /.well-known/agent-card.json`) and the JSON-RPC endpoint, with hostnames, timeouts long enough for Server-Sent Events, and optional path rewriting. +- **Protocol validation** by Envoy's A2A filter: every POST on the route must be a JSON-RPC 2.0 envelope, and oversized bodies are rejected. +- **A2A fields in access logs and metrics**: the JSON-RPC `method` and `id` (and, for A2A 0.3 clients, task and context IDs) land in Envoy dynamic metadata; the filter also exposes `a2a.requests_rejected`, `a2a.invalid_json` and `a2a.body_too_large` counters. +- **Client authentication** (JWT, API key, external auth) on the RPC endpoint via `SecurityPolicy`, while the public card stays open. +- Optionally, a **gateway-served agent card** so clients discover the gateway URL rather than the agent's internal address. + +What you do **not** get yet: aggregation of several agents behind one card, rewriting of the backend's own card, method-level authorization, or support for the gRPC and HTTP+JSON bindings. See [Limitations](#limitations). + +## Prerequisites + +- Agent Router installed following [Prerequisites](../../getting-started/prerequisites.md). The Envoy Gateway values file used there already sets `extensionApis.enableEnvoyPatchPolicy: true`. +- Envoy Gateway v1.8 or newer. The A2A filter exists since Envoy Proxy v1.38.0, which Envoy Gateway v1.8 is the first release to ship; see the [compatibility matrix](https://gateway.envoyproxy.io/news/releases/matrix/) for later pairings. On an older proxy image the patched listener is rejected by Envoy, which takes every route on that listener down, so check the image before you start. +- An A2A agent speaking the JSON-RPC binding. The example below assumes it runs as `Service/research-agent` on port 8080 in namespace `default`. + +:::warning +`EnvoyPatchPolicy` lets anyone with permission to create it inject arbitrary proxy configuration. Restrict it with RBAC as described in the [Envoy Gateway security note](https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/#security-warning). The A2A filter itself is marked alpha with an unknown security posture in Envoy: use it only where both clients and agents are trusted. +::: + +## Step 1: Give A2A agents their own Gateway + +The A2A filter is applied to a whole Envoy listener. Once enabled, every JSON `POST` on that listener is buffered and parsed as JSON-RPC, so an LLM route sharing the listener would have its chat-completion bodies parsed too and could hit the body size limit. Keep A2A traffic on a dedicated Gateway (or at least a dedicated listener): + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: a2a-gateway + namespace: default +spec: + gatewayClassName: envoy-ai-gateway-basic + listeners: + - name: http + protocol: HTTP + port: 80 +``` + +If you must share a listener with other traffic, see [Advanced: enabling the filter per route](#advanced-enabling-the-filter-per-route). + +## Step 2: Route the agent card and the RPC endpoint + +One `HTTPRoute` with two **named** rules. The names matter: `SecurityPolicy` targets the `rpc` rule in Step 4 so the card stays public. + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: research-agent + namespace: default +spec: + parentRefs: + - name: a2a-gateway + hostnames: + - research.agents.example.com + rules: + # A2A discovery: the card is host-rooted, so one agent per hostname. + - name: agent-card + matches: + - method: GET + path: + type: Exact + value: /.well-known/agent-card.json + backendRefs: + - name: research-agent + port: 8080 + # JSON-RPC endpoint. Streaming methods return text/event-stream and can run + # for minutes, so raise the request timeouts (Envoy's default is 15s). + - name: rpc + matches: + - method: POST + path: + type: Exact + value: /a2a + timeouts: + request: 30m + backendRequest: 30m + backendRefs: + - name: research-agent + port: 8080 +``` + +If the agent serves its RPC endpoint on a different path, add a `URLRewrite` filter to the `rpc` rule. Do not attach request buffering (`BackendTrafficPolicy.requestBuffer`) to these routes; it stalls streaming responses. + +## Step 3: Enable Envoy's A2A filter + +`EnvoyPatchPolicy` inserts the filter into the listener's HTTP filter chain immediately before the router, so it runs after any authentication filters added by `SecurityPolicy`: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: EnvoyPatchPolicy +metadata: + name: a2a-filter + namespace: default +spec: + targetRef: + group: gateway.networking.k8s.io + kind: Gateway + name: a2a-gateway + type: JSONPatch + jsonPatches: + - type: type.googleapis.com/envoy.config.listener.v3.Listener + # //. With the XDSNameSchemeV2 + # runtime flag enabled in Envoy Gateway this becomes "tcp-80". + name: default/a2a-gateway/http + operation: + op: add + # Selects the router filter; "add" at an array index inserts before it. + jsonPath: "$..http_filters[?(@.name=='envoy.filters.http.router')]" + value: + name: envoy.filters.http.a2a + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.a2a.v3.A2a + # PASS_THROUGH forwards non-A2A requests untouched. + # REJECT answers them with 400; use it only on listeners that carry + # nothing but JSON-RPC A2A traffic (see Limitations). + traffic_mode: PASS_THROUGH + # Bytes buffered while parsing the JSON-RPC envelope. Envoy's default + # is 8192, which is too small for real messages; the maximum is 10485760. + # Requests that exceed it get a 413 in both traffic modes. + max_request_body_size: 1048576 +``` + +Check that the patch was applied: + +```shell +kubectl get envoypatchpolicy a2a-filter -o yaml | grep -A3 'type: Programmed' +``` + +Both `Accepted` and `Programmed` must be `True`. If the Gateway was created with `mergeGateways`, target the `GatewayClass` instead of the `Gateway`, as in the [Envoy Gateway task](https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/). + +## Step 4: Protect the RPC endpoint, keep the card public + +The card must be fetchable without credentials; the RPC endpoint usually must not. Target the `rpc` rule by `sectionName`: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: SecurityPolicy +metadata: + name: research-agent-rpc + namespace: default +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: HTTPRoute + name: research-agent + sectionName: rpc + jwt: + providers: + - name: corp-idp + issuer: https://idp.example.com + audiences: + - research-agent + remoteJWKS: + uri: https://idp.example.com/.well-known/jwks.json +``` + +`apiKeyAuth` and `extAuth` work the same way. Advertise the matching scheme in the agent card's `securitySchemes` so clients know what to send. + +## Step 5: Log A2A fields + +The filter writes its findings as dynamic metadata under the filter name. Add them to the access log in your `EnvoyProxy` resource, alongside the MCP fields from [Access Logs](../observability/accesslogs.md): + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: EnvoyProxy +metadata: + name: a2a-access-logs + namespace: default +spec: + telemetry: + accessLog: + settings: + - sinks: + - type: File + file: + path: /dev/stdout + format: + type: JSON + json: + a2a.method: "%DYNAMIC_METADATA(envoy.filters.http.a2a:method)%" + jsonrpc.request.id: "%DYNAMIC_METADATA(envoy.filters.http.a2a:id)%" + a2a.version: "%REQ(A2A-VERSION)%" + # Populated for A2A 0.3 clients only; see Limitations. + a2a.task.id: "%DYNAMIC_METADATA(envoy.filters.http.a2a:params:taskId)%" + a2a.context.id: "%DYNAMIC_METADATA(envoy.filters.http.a2a:params:message:contextId)%" + # Common fields + start_time: "%START_TIME%" + method: "%REQ(:METHOD)%" + path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" + response_code: "%RESPONSE_CODE%" + response_code_details: "%RESPONSE_CODE_DETAILS%" + duration: "%DURATION%" + upstream_cluster: "%UPSTREAM_CLUSTER%" + route_name: "%ROUTE_NAME%" +``` + +Attach it to the Gateway through `spec.infrastructure.parametersRef`. Do not log the whole `envoy.filters.http.a2a` namespace: for 0.3 clients it contains the full message content. + +Rejections show up in `response_code_details` as `a2a_filter_reject`, `a2a_filter_not_valid_jsonrpc`, `a2a_filter_parse_error` or `a2a_filter_body_too_large`, and in the proxy stats as `http.http-.a2a.requests_rejected`, `.invalid_json` and `.body_too_large` (for example `http.http-10080.a2a.body_too_large`). + +## Step 6: Verify + +```shell +export GATEWAY_HOST=$(kubectl get gateway/a2a-gateway -o jsonpath='{.status.addresses[0].value}') + +# Discovery +curl -s -H "Host: research.agents.example.com" \ + http://$GATEWAY_HOST/.well-known/agent-card.json | jq .name + +# A2A 1.0 SendMessage (add the Authorization header if you applied Step 4) +curl -s -H "Host: research.agents.example.com" \ + -H "Content-Type: application/a2a+json" -H "A2A-Version: 1.0" \ + -d '{"jsonrpc":"2.0","id":"1","method":"SendMessage","params":{"message":{"messageId":"m1","role":"ROLE_USER","parts":[{"text":"hello"}]}}}' \ + http://$GATEWAY_HOST/a2a | jq . + +# Protocol validation: a JSON body that is not JSON-RPC is forwarded in PASS_THROUGH +# mode and rejected with 400 in REJECT mode. +curl -s -o /dev/null -w '%{http_code}\n' -H "Host: research.agents.example.com" \ + -H "Content-Type: application/json" -d '{"hello":"world"}' http://$GATEWAY_HOST/a2a +``` + +The access log line for the second request carries `"a2a.method": "SendMessage"`. + +`kubectl port-forward` is fine for these one-shot calls, but it drops long-lived streams and any connection Envoy resets (for example after a 413). Test `SendStreamingMessage` from a pod inside the cluster or through a real load balancer; through the gateway a stream that the agent keeps open for 18 seconds arrives intact. + +## Optional: serve the agent card from the gateway + +A2A clients connect to whatever URL the card advertises. If the agent only knows its in-cluster address, its card sends clients around the gateway. Rather than rewriting the backend's card, serve a static one from the gateway with Envoy Gateway's direct response filter and point the `agent-card` rule at it: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: HTTPRouteFilter +metadata: + name: research-agent-card + namespace: default +spec: + directResponse: + statusCode: 200 + contentType: application/json + body: + type: Inline + inline: | + { + "name": "Research Agent", + "description": "Summarises research papers", + "supportedInterfaces": [ + {"url": "https://research.agents.example.com/a2a", "protocolBinding": "JSONRPC", "protocolVersion": "1.0"} + ], + "capabilities": {"streaming": true}, + "defaultInputModes": ["text/plain"], + "defaultOutputModes": ["text/plain"], + "securitySchemes": {"bearer": {"type": "http", "scheme": "bearer"}}, + "skills": [] + } +``` + +```yaml +- name: agent-card + matches: + - method: GET + path: + type: Exact + value: /.well-known/agent-card.json + filters: + - type: ExtensionRef + extensionRef: + group: gateway.envoyproxy.io + kind: HTTPRouteFilter + name: research-agent-card +``` + +## Advanced: enabling the filter per route + +If the A2A route must share a listener with other traffic, insert the filter **disabled** and enable it only on the A2A routes. This is the same mechanism Agent Router uses for its own filters. + +1. In the listener patch above, add `disabled: true` next to `typed_config`. +2. Add a second patch against the route configuration. Envoy Gateway names xDS routes `httproute///rule//match//`, so a regular expression on the route name selects the rules you want. Patch the filter's own key under `typed_per_filter_config`: Envoy Gateway creates the map when the route has none and leaves other entries (for example the `SecurityPolicy` API-key config) untouched. + +```yaml +- type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration + name: default/a2a-gateway/http + operation: + op: add + # Only the rpc rule needs the filter; the card is a bodyless GET anyway. + jsonPath: "$..routes[?(@.name =~ 'httproute/default/research-agent/rule/1/')]" + path: /typed_per_filter_config/envoy.filters.http.a2a + value: + "@type": type.googleapis.com/envoy.config.route.v3.FilterConfig + # An empty config enables a filter that is disabled at the listener. + config: {} +``` + +With this in place the filter parses bodies on the `rpc` route only: a 1.5 MiB non-A2A body sent to another `HTTPRoute` on the same listener is forwarded, while the same body on `/a2a` gets a 413. Rule indexes follow the order of `rules` in the `HTTPRoute`, so keep the patch and the route in the same file. Confirm the result with `kubectl get envoypatchpolicy` (both conditions `True`) and the Envoy admin `config_dump`. This bookkeeping is exactly what a native `A2ARoute` will remove. + +## Keeping up with Envoy + +The `EnvoyPatchPolicy` above is plain Envoy configuration, so as the A2A filter grows you can amend the `typed_config` yourself without waiting for an Agent Router release. Check these sources against the Envoy version your gateway actually runs: + +- [A2A filter configuration overview](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/a2a_filter) and the [A2A proto API reference](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/a2a/v3/a2a.proto), which lists every field the filter accepts. Replace `latest` in the URL with your proxy's version tag, for example `v1.38.0`, to see exactly what your proxy supports. +- [Envoy release notes](https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history) for `a2a` entries in new releases, and the [filter source](https://github.com/envoyproxy/envoy/tree/main/source/extensions/filters/http/a2a) for behaviour that has not been documented yet. +- [Envoy Gateway compatibility matrix](https://gateway.envoyproxy.io/news/releases/matrix/) to see which Envoy Proxy version each Envoy Gateway release ships, and the [EnvoyPatchPolicy task](https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/) for the xDS resource names the patch must target, which change when the `XDSNameSchemeV2` runtime flag is on. + +To find the running version, read the image tag of the proxy pod: + +```shell +kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=a2a-gateway \ + -o jsonpath='{.items[0].spec.containers[?(@.name=="envoy")].image}{"\n"}' +``` + +To try a newer Envoy Proxy before Envoy Gateway ships it, override the proxy image in the `EnvoyProxy` resource as described in [Customize EnvoyProxy](https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/). Keep in mind that the A2A filter is alpha, so its config and behaviour can change between releases; re-read the release notes before bumping. + +## Limitations + +These reflect the filter as it ships today and will shrink as Envoy and Agent Router evolve. Check [Keeping up with Envoy](#keeping-up-with-envoy) for what your proxy version supports. + +- **A2A 1.0 method names are only partly modelled.** The filter validates every request and logs `method` and `id`, but `taskId` and `contextId` extraction currently keys on the A2A 0.3 names (`message/send`, `tasks/get`, ...). +- **`parser_config` and `storage_mode` are accepted but not yet wired up.** Leave them unset until a release note says otherwise. +- **`max_request_body_size` applies in both modes.** A JSON-RPC envelope that is not complete within the limit gets a 413 even in `PASS_THROUGH`. +- **`REJECT` suits JSON-RPC-only listeners.** Route the gRPC and HTTP+JSON bindings on a listener without the filter. +- **Agent cards are served as-is, one per hostname.** URL rewriting, aggregation and `aigw run` support are part of the first-class A2A work in [issue #2070](https://github.com/theagentrouter/agent-router/issues/2070). diff --git a/site/versioned_docs/version-1.2/capabilities/gateway-config.md b/site/versioned_docs/version-1.2/capabilities/gateway-config.md new file mode 100644 index 0000000000..9f2a39fd7d --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/gateway-config.md @@ -0,0 +1,293 @@ +--- +id: gateway-config +title: Gateway Configuration +sidebar_position: 2 +--- + +# Gateway Configuration + +The `GatewayConfig` CRD provides gateway-scoped configuration for the AI Gateway external processor container. This allows you to configure environment variables and resource requirements at the Gateway level, rather than at the route level. + +## Overview + +Use `GatewayConfig` when you need to: + +- Configure per-gateway OpenTelemetry tracing settings +- Set resource requirements (CPU/memory) for the external processor +- Share configuration across multiple Gateways +- Configure environment variables for different gateway instances without affecting others + +## Usage + +### Creating a GatewayConfig + +Create a `GatewayConfig` resource with your desired configuration: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: my-gateway-config + namespace: default +spec: + extProc: + kubernetes: + env: + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: "http://otel-collector:4317" + - name: OTEL_SERVICE_NAME + value: "my-ai-gateway" + resources: + requests: + cpu: "100m" + memory: "128Mi" + limits: + cpu: "500m" + memory: "512Mi" +``` + +### Referencing from a Gateway + +Reference the `GatewayConfig` from your Gateway using an annotation: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: my-gateway + namespace: default + annotations: + aigateway.envoyproxy.io/gateway-config: my-gateway-config +spec: + gatewayClassName: envoy-gateway + listeners: + - name: http + protocol: HTTP + port: 8080 +``` + +:::note +The `GatewayConfig` must be in the same namespace as the Gateway that references it. +::: + +## Configuration Options + +### Environment Variables + +The `spec.extProc.kubernetes.env` field accepts a list of Kubernetes `EnvVar` objects: + +```yaml +spec: + extProc: + kubernetes: + env: + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: "http://otel-collector:4317" + - name: OTEL_EXPORTER_OTLP_HEADERS + value: "api-key=your-secret" + - name: LOG_LEVEL + value: "debug" +``` + +### Resource Requirements + +The `spec.extProc.kubernetes.resources` field configures compute resources for the external processor container: + +```yaml +spec: + extProc: + kubernetes: + resources: + requests: + cpu: "100m" + memory: "128Mi" + limits: + cpu: "500m" + memory: "512Mi" +``` + +If not specified, Kubernetes default resource allocations are used. + +### Metadata Forwarding Namespaces + +The `spec.extProc.metadataForwardingNamespaces` field lists the untyped dynamic metadata namespaces Envoy forwards to the external processor on this Gateway's AI traffic. It is required for [per-request credential override from dynamic metadata](./security/upstream-auth.mdx#from-dynamic-metadata-recommended): a `BackendSecurityPolicy` using `credentialOverride.fromDynamicMetadata` only receives its metadata when the namespace it reads from is declared here. + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: my-gateway-config + namespace: default +spec: + extProc: + metadataForwardingNamespaces: + - envoy.filters.http.ext_authz +``` + +These namespaces travel into the processor. The processor writes `io.envoy.ai_gateway` back to Envoy, which receives that namespace for token-based rate limiting and access logs. + +The list belongs to the Gateway that declares it and governs that Gateway's AI traffic, so attaching a route to a Gateway cannot widen what a different Gateway forwards. Under Envoy Gateway's `mergeGateways`, every Gateway in the class shares one Envoy, so a namespace any one of them declares is forwarded for all of their AI traffic. The per-Gateway isolation described above does not hold in that setup. + +### Forward Proxy (Egress) + +The `spec.forwardProxy` field routes all upstream AI/LLM traffic from Gateways referencing this `GatewayConfig` through an HTTP CONNECT forward proxy. This is intended for data planes deployed in locked-down networks that have no direct outbound access to providers and require all egress to traverse a proxy. + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: my-gateway-config + namespace: default +spec: + forwardProxy: + # host:port of the HTTP CONNECT proxy (host may be a hostname or IP; port is required). + address: proxy.corp:3128 +``` + +Under the hood, each upstream cluster's transport socket is wrapped in Envoy's `http_11_proxy` transport socket, which opens an HTTP/1.1 `CONNECT` tunnel to the proxy and then establishes the upstream connection through it. The upstream TLS session is preserved end-to-end inside the tunnel, so the proxy sees only the `CONNECT` and the encrypted bytes — not the request contents. + +:::note +The forward proxy applies to all AI Gateway upstream (LLM) clusters for Gateways that reference this `GatewayConfig`. InferencePool backends, which use in-cluster endpoints, are not proxied. +::: + +:::warning +Proxy authentication (for example, `Proxy-Authorization`) and a TLS connection to the proxy itself are not currently supported. The proxy is expected to be reachable over plaintext HTTP and to authorize the gateway by network policy (for example, an IP allowlist). +::: + +## Environment Variable Precedence + +Environment variables can be configured at multiple levels. The precedence order is (highest to lowest): + +1. **GatewayConfig.spec.extProc.kubernetes.env** - Highest priority +2. **Global controller flags** (`--extproc-extra-env-vars`) - Lower priority + +When the same environment variable is defined at multiple levels, the higher precedence value is used. + +### Example + +If the controller is started with: + +``` +--extproc-extra-env-vars="LOG_LEVEL=info;GLOBAL_VAR=global" +``` + +And a GatewayConfig defines: + +```yaml +env: + - name: LOG_LEVEL + value: "debug" + - name: CONFIG_VAR + value: "config" +``` + +The resulting environment variables will be: + +- `LOG_LEVEL=debug` (GatewayConfig overrides global) +- `GLOBAL_VAR=global` (from global) +- `CONFIG_VAR=config` (from GatewayConfig) + +## Shared Configuration + +Multiple Gateways can reference the same `GatewayConfig`: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: shared-config +spec: + extProc: + kubernetes: + env: + - name: OTEL_SERVICE_NAME + value: "ai-gateway-cluster" +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: gateway-1 + annotations: + aigateway.envoyproxy.io/gateway-config: shared-config +spec: + # ... +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: gateway-2 + annotations: + aigateway.envoyproxy.io/gateway-config: shared-config +spec: + # ... +``` + +## Migration from Route-Level Configuration + +The route-level resource configuration (`AIGatewayRoute.spec.filterConfig.externalProcessor.resources`) is deprecated. Migrate to `GatewayConfig`: + +### Before (deprecated) + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: my-route +spec: + filterConfig: + type: ExternalProcessor + externalProcessor: + resources: + requests: + cpu: "100m" + memory: "128Mi" +``` + +### After (recommended) + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: my-config +spec: + extProc: + kubernetes: + resources: + requests: + cpu: "100m" + memory: "128Mi" +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: my-gateway + annotations: + aigateway.envoyproxy.io/gateway-config: my-config +spec: + # ... +``` + +## Status + +The `GatewayConfig` status reports the validity of the configuration: + +```yaml +status: + conditions: + - type: Accepted + status: "True" + reason: Accepted + message: "GatewayConfig reconciled successfully" +``` + +Possible condition types: + +- `Accepted`: The configuration is valid and applied +- `NotAccepted`: The configuration has validation errors + +## See Also + +- [Tracing](./observability/tracing.md) - Configure distributed tracing for AI Gateway +- [Metrics](./observability/metrics.md) - Configure metrics collection +- [Examples](https://github.com/theagentrouter/agent-router/tree/main/examples/gateway-config) - Example YAML files diff --git a/site/versioned_docs/version-1.2/capabilities/index.md b/site/versioned_docs/version-1.2/capabilities/index.md new file mode 100644 index 0000000000..ef5f586db3 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/index.md @@ -0,0 +1,66 @@ +--- +id: capabilities +title: Capabilities +sidebar_position: 5 +--- + +# Agent Router Capabilities + +Welcome to the Agent Router capabilities documentation! This section provides detailed information about the various features and capabilities that Agent Router offers to help you manage and optimize your AI/LLM traffic. + +## LLM Providers Integrations + +Support for various Large Language Model providers: + +- **[Connecting to AI Providers](./llm-integrations/connect-providers.md)**: Learn how to establish connectivity with any supported AI provider +- **[Supported Providers](./llm-integrations/supported-providers.md)**: Compatible AI/LLM service providers +- **[Supported Endpoints](./llm-integrations/supported-endpoints.md)**: Available API endpoints and operations +- **[Vendor-Specific Fields](./llm-integrations/vendor-specific-fields.md)**: Use backend-specific parameters and access provider-unique capabilities in your OpenAI-compatible requests +- **[Prompt Caching](./llm-integrations/prompt-caching.md)**: Provider-agnostic prompt caching using unified cache_control API + +## Inference Optimization + +Advanced inference optimization capabilities for AI/LLM workloads: + +- **[InferencePool Support](./inference/inferencepool-support.md)**: Intelligent routing and load balancing for inference endpoints +- **[HTTPRoute + InferencePool](./inference/httproute-inferencepool.md)**: Basic inference routing with standard Gateway API +- **[AIGatewayRoute + InferencePool](./inference/aigatewayroute-inferencepool.md)**: Advanced AI-specific routing with enhanced features + +## Gateway Configuration + +- **[GatewayConfig](./gateway-config.md)**: Gateway-scoped configuration for the external processor (env vars, resources, shared settings) +- **[Scaling](./scaling.md)**: Configure multiple controller replicas and horizontal pod autoscaling for production deployments + +## Traffic Management + +Comprehensive traffic handling and routing capabilities: + +- **[Model Virtualization](./traffic/model-virtualization.md)**: Abstract and virtualize AI models +- **[Provider Fallback](./traffic/provider-fallback.md)**: Automatic failover between AI providers +- **[Quota Policy](./traffic/quota-policy.md)**: Token-based quota management for controlling total consumption budgets +- **[Usage-based Rate Limiting](./traffic/usage-based-ratelimiting.md)**: Token-aware rate limiting for AI workloads +- **[Header and Body Mutations](./traffic/header-body-mutations.md)**: Customize HTTP headers and JSON body fields per backend or route + +## Security + +Robust security features for AI gateway deployments: + +- **[Upstream Authentication](./security/upstream-auth.mdx)**: Secure authentication to upstream AI services + +## Model Context Protocol (MCP) + +Connect AI agents to external tools and data sources: + +- **[MCP Gateway](./mcp/)**: Server multiplexing, tool routing, OAuth authentication, and observability for MCP workloads + +## Agent2Agent (A2A) + +Route traffic between agents: + +- **[A2A Traffic (preview)](./a2a/)**: Route A2A agents through the gateway today using Envoy's native A2A filter, ahead of a first-class A2A route type + +## Observability + +Monitoring and observability tools for AI workloads: + +- **[Metrics](./observability/metrics.md)**: Comprehensive metrics collection and monitoring diff --git a/site/versioned_docs/version-1.2/capabilities/inference/_setup.mdx b/site/versioned_docs/version-1.2/capabilities/inference/_setup.mdx new file mode 100644 index 0000000000..a2b4985119 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/inference/_setup.mdx @@ -0,0 +1,48 @@ +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +## Prerequisites + +Before starting, ensure you have: + +1. **Kubernetes cluster** with Gateway API support +2. **Envoy Gateway** installed and configured + +## Step 1: Install Gateway API Inference Extension CRDs + +Install the Gateway API Inference Extension CRDs and controller: + + +{`kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${vars.gaieVersion}/manifests.yaml`} + + +## Step 2: Configure Envoy Gateway for InferencePool + +Install (or upgrade) Envoy Gateway with the InferencePool addon values file, and wait for it to be ready: + + +{`helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \\ + --version v${vars.egVersion} \\ + --namespace envoy-gateway-system \\ + --create-namespace \\ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/manifests/envoy-gateway-values.yaml \\ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/inference-pool/envoy-gateway-values-addon.yaml + +kubectl wait --timeout=2m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available`} + + +See the [Envoy Gateway Installation Guide](../../getting-started/prerequisites.md#additional-features-rate-limiting-inferencepool-etc) if you're combining this with other addons (like rate limiting), or already have Envoy Gateway installed and just need to add this values file. + +## Step 3: Deploy InferencePool Base Resources + +Deploy the sample inference backends, InferencePools, and Endpoint Picker Providers (EPP) that the rest of this guide builds on: + + +{`kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/inference-pool/base.yaml`} + + +This creates: + +- The `vllm-llama3-8b-instruct` `InferencePool` (a simulated vLLM deployment) with its Endpoint Picker Provider (EPP) +- The `mistral` `InferencePool` with its own Endpoint Picker Provider (EPP) +- A standard `envoy-ai-gateway-basic-testupstream` `AIServiceBackend`/`Backend` used later for non-InferencePool (fallback) routing diff --git a/site/versioned_docs/version-1.2/capabilities/inference/aigatewayroute-inferencepool.md b/site/versioned_docs/version-1.2/capabilities/inference/aigatewayroute-inferencepool.md new file mode 100644 index 0000000000..78ce39db5d --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/inference/aigatewayroute-inferencepool.md @@ -0,0 +1,264 @@ +--- +id: aigatewayroute-inferencepool +title: AIGatewayRoute + InferencePool Guide +sidebar_position: 3 +--- + +import Setup from './\_setup.mdx'; + +# AIGatewayRoute + InferencePool Guide + +This guide demonstrates how to use InferencePool with AIGatewayRoute for advanced AI-specific inference routing. This approach provides enhanced features like model-based routing, token rate limiting, and advanced observability. + + + +## Step 4: Configure Gateway and AIGatewayRoute + +Create a Gateway and AIGatewayRoute with multiple InferencePool backends: + +```yaml +cat < + +## Step 4: Configure Gateway and HTTPRoute + +Create a Gateway and HTTPRoute that uses the InferencePool: + +```yaml +cat <>Envoy: Request with model header + Envoy->>RLS: Check Rate Limit + RLS-->>Envoy: Rate limit OK + Envoy->>Processor: Router-level ExtProc Request + Note over Processor: Extract Model Name + Processor-->>Envoy: ClearRouteCache + Envoy->>EPP: Router-level ExtProc Request + Note over EPP: Pick Optimal Endpoint + EPP-->>Envoy: Add x-gateway-destination-endpoint header + loop Retry/Fallback loop + Note over Envoy: Forward to Selected Endpoint + Envoy->>Processor: Upstream-level ExtProc Request + Note over Processor: Request Transform + Processor-->>Envoy: Transformed Request + Envoy->>Backend: Forward Request + Backend-->>Envoy: Response + end + Envoy->>Processor: Process Response + Note over Processor: Extract Token Usage & Metrics + Processor-->>Envoy: Add Usage Metadata + Envoy->>RLS: Update Rate Limit Budget + RLS-->>Envoy: Budget Updated + Envoy->>Client: Response +``` + +### HTTPRoute + InferencePool Flow + +```mermaid +sequenceDiagram + participant Client as Client + participant Envoy as Envoy Proxy + participant EPP as Endpoint Picker Provider + participant Backend as Inference Backend + + Client->>Envoy: Request + Envoy->>EPP: Router-level ExtProc Request + Note over EPP: Pick Optimal Endpoint + EPP-->>Envoy: Add x-gateway-destination-endpoint header + loop Retry/Fallback loop + Note over Envoy: Forward to Selected Endpoint + Envoy->>Backend: Forward Request + Backend-->>Envoy: Response + end + Envoy->>Client: Response +``` + +## Integration Approaches + +Agent Router supports two ways to use InferencePool: + +### 1. HTTPRoute + InferencePool + +Direct integration with standard Gateway API HTTPRoute for simple inference routing scenarios. + +**Use Cases:** + +- Simple inference workloads without complex AI-specific requirements +- Direct OpenAI-compatible API forwarding + +### 2. AIGatewayRoute + InferencePool + +Enhanced integration with AI Gateway's custom route type for advanced AI-specific features. + +**Use Cases:** + +- Multi-model routing based on request content +- Token-based rate limiting +- Advanced observability and metrics +- Request/response transformation + +## AIGatewayRoute Advantages + +AIGatewayRoute provides several advantages over standard HTTPRoute when used with InferencePool: + +### Advanced OpenAI Routing + +- Built-in OpenAI API schema validation +- Seamless integration with OpenAI SDKs +- Route multiple models in a single listener +- Mix InferencePool and traditional backends +- Automatic model extraction from request body + +### AI-Specific Features + +- Token-based rate limiting +- Model performance metrics +- Cost tracking and management +- Request/response transformation + +## Getting Started + +To use InferencePool with Agent Router, you'll need to: + +1. **Install Prerequisites**: + - Deploy Gateway API Inference Extension CRD. + - Configure Envoy Gateway with the InferencePool support. See [Envoy Gateway Installation Guide](../../getting-started/prerequisites.md#additional-features-rate-limiting-inferencepool-etc) for details. +2. **Configure InferencePool**: Define your inference endpoints and selection criteria +3. **Set up Routes**: Configure either HTTPRoute or AIGatewayRoute to use the InferencePool +4. **Deploy Endpoint Picker**: Install and configure the endpoint picker provider + +The following sections provide detailed guides for both integration approaches with complete examples and step-by-step instructions. diff --git a/site/versioned_docs/version-1.2/capabilities/llm-integrations/connect-providers.md b/site/versioned_docs/version-1.2/capabilities/llm-integrations/connect-providers.md new file mode 100644 index 0000000000..eb2ab1d020 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/llm-integrations/connect-providers.md @@ -0,0 +1,572 @@ +--- +id: connect-providers +title: Connecting to AI Providers +sidebar_position: 10 +--- + +# Connecting to AI Providers + +Agent Router provides a unified interface for connecting to multiple AI providers through a standardized configuration approach. This page explains the fundamental concepts, resources, and relationships required to establish connectivity with any supported AI provider. + +## Overview + +Establishing connectivity with an AI provider involves configuring three key Kubernetes resources that work together to enable secure, scalable access to AI services: + +1. **AIServiceBackend** - Defines the backend service and its API schema +2. **BackendSecurityPolicy** - Configures authentication credentials +3. **AIGatewayRoute** - Routes client requests to the appropriate backends + +These resources provide a consistent configuration model regardless of which AI provider you're connecting to, whether it's OpenAI, AWS Bedrock, Azure OpenAI, or any other [supported provider](./supported-providers). + +## Core Resources for Provider Connectivity + +### AIServiceBackend + +The `AIServiceBackend` resource represents an individual AI service backend and serves as the bridge between your gateway and the AI provider's API. + +#### Purpose and Configuration + +- **API Schema Definition**: Specifies which API format the backend expects (OpenAI v1, AWS Bedrock, Azure OpenAI, etc.) +- **Backend Reference**: Points to the Envoy Gateway Backend resource +- **Security Integration**: Links to authentication policies for upstream services + +#### Key Fields + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIServiceBackend +metadata: + name: my-provider-backend +spec: + # API schema the backend expects + schema: + name: OpenAI # Provider API format + version: "v1" # API version (optional for OpenAI, defaults to "v1") + + # Reference to the Envoy Gateway Backend resource + backendRef: + name: my-provider-backend + kind: Backend + group: gateway.envoyproxy.io +``` + +#### Schema Configuration Examples + +Different providers require different schema configurations: + +| Provider | Schema Configuration | +| -------------------------- | --------------------------------------------------------- | +| OpenAI | `{"name":"OpenAI","version":"v1"}` | +| AWS Bedrock | `{"name":"AWSBedrock"}` | +| Azure OpenAI | `{"name":"AzureOpenAI","version":"2025-01-01-preview"}` | +| GCP Vertex AI | `{"name":"GCPVertexAI"}` | +| GCP Anthropic on Vertex AI | `{"name":"GCPAnthropic", "version": "vertex-2023-10-16"}` | + +:::tip +Many providers offer OpenAI-compatible APIs, which allows them to use the OpenAI schema configuration with provider-specific version paths. +::: + +### BackendSecurityPolicy + +The `BackendSecurityPolicy` resource configures authentication credentials needed to access upstream AI services securely. + +#### Purpose and Configuration + +- **Credential Management**: Stores API keys, cloud credentials, or other authentication mechanisms +- **Security Isolation**: Keeps sensitive credentials separate from routing configuration +- **Provider Flexibility**: Supports multiple authentication types for different providers + +#### Authentication Types + +##### API Key Authentication + +Commonly used across many providers + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: openai-auth +spec: + type: APIKey + apiKey: + secretRef: + name: openai-secret +``` + +:::note +The secret must contain the API key with the key name `"apiKey"`. +::: + +##### AWS Credentials + +Used when connecting to AWS Bedrock. Supports three authentication methods: + +**Option 1: EKS Pod Identity or IRSA (Recommended for production)** + +When running on EKS, the AWS SDK automatically uses the default credential chain, which includes EKS Pod Identity and IRSA. Simply configure the region: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: bedrock-auth +spec: + type: AWSCredentials + awsCredentials: + region: us-east-1 + # No credentialsFile needed - automatically uses: + # - EKS Pod Identity (if Pod Identity association exists) + # - IRSA (if ServiceAccount has eks.amazonaws.com/role-arn annotation) +``` + +See the [Connect AWS Bedrock guide](../../getting-started/connect-providers/aws-bedrock.md) for detailed setup instructions. + +**Option 2: Static Credentials (Development/Testing)** + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: bedrock-auth +spec: + type: AWSCredentials + awsCredentials: + region: us-east-1 + credentialsFile: + secretRef: + name: aws-secret + profile: default # Optional, defaults to "default" +``` + +:::note +When using static credentials, the secret must contain the AWS credentials file with the key name `"credentials"`. +::: + +##### Azure Credentials + +Used for connecting to Azure OpenAI + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: azure-auth +spec: + type: AzureCredentials + azureCredentials: + clientID: "your-azure-client-id" + tenantID: "your-azure-tenant-id" + clientSecretRef: + name: azure-secret +``` + +:::note +The secret must contain the Azure client secret with the key name `"client-secret"`. +::: + +##### GCP Credentials + +Used for connecting to GCP Vertex AI and Anthropic on GCP. Supports three authentication methods: + +1. Application Default Credentials (Recommended for GKE): + When running on GKE with Workload Identity configured, you can use Application Default Credentials (ADC) without managing service account keys. Simply specify the project and region: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: gcp-auth-adc + namespace: default +spec: + type: GCPCredentials + gcpCredentials: + projectName: "your-gcp-project" + region: "us-central1" +``` + +ADC automatically supports GKE Workload Identity, the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, and default service account credentials when running on GCP. + +2. Service Account Key Files: + A service account key file is a JSON file containing a private key that authenticates as a service account. + You create a service account in GCP, generate a key file, download it, and then store it in the k8s secret referenced by BackendSecurityPolicy. + Agent Router uses this key file to generate an access token and authenticate with GCP Vertex AI. + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: gcp-auth-service-account-key + namespace: default +spec: + type: GCPCredentials + gcpCredentials: + projectName: GCP_PROJECT_NAME # Replace with your GCP project name + region: GCP_REGION # Replace with your GCP region + credentialsFile: + secretRef: + name: envoy-ai-gateway-basic-gcp-service-account-key-file +``` + +3. Workload Identity Federation: + Workload Identity Federation is a modern, keyless authentication method that allows workloads running outside of GCP to impersonate a service account using their own native identity. + It leverages a trust relationship between GCP and an external identity provider such as OIDC. + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: gcp-auth +spec: + type: GCPCredentials + gcpCredentials: + projectName: "your-gcp-project" + region: "us-central1" + workloadIdentityFederationConfig: + projectID: "your-gcp-project-id" + workloadIdentityPoolName: "your-workload-identity-pool" + workloadIdentityProviderName: "your-identity-provider" + serviceAccountImpersonation: + serviceAccountName: "your-service-account" + oidcExchangeToken: + oidc: + provider: + issuer: "https://your-oidc-provider.com" + clientID: "your-oidc-client-id" + clientSecret: + name: "gcp-client-secret" + namespace: default +``` + +##### Secrets in another namespace + +A `secretRef` or `clientSecretRef` without `namespace` refers to a Secret in the BackendSecurityPolicy's own namespace. A Secret in another namespace also needs a [ReferenceGrant](https://gateway-api.sigs.k8s.io/api-types/referencegrant/) in the Secret's namespace that allows the reference. Without it, the BackendSecurityPolicy is marked `NotAccepted`. + +```yaml +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: allow-backend-security-policy-secrets + namespace: credentials # the Secret's namespace +spec: + from: + - group: aigateway.envoyproxy.io + kind: BackendSecurityPolicy + namespace: ai-backends # the BackendSecurityPolicy's namespace + to: + - group: "" + kind: Secret +``` + +With `aigw run`, put the ReferenceGrant in the configuration file. + +#### Security Best Practices + +- **Store credentials in Kubernetes Secrets**: Never expose sensitive data in plain text +- **Use principle of least privilege**: Grant only necessary permissions +- **Rotate credentials regularly**: Implement credential rotation policies +- **Separate environments**: Use different credentials for development, staging, and production + +### AIGatewayRoute + +The `AIGatewayRoute` resource defines how client requests are routed to appropriate AI backends and manages the unified API interface. + +#### Purpose and Configuration + +- **Request Routing**: Directs traffic to specific backends based on model names or other criteria +- **API Unification**: Provides a consistent interface regardless of backend provider +- **Request Transformation**: Automatically converts between different API schemas +- **Load Balancing**: Distributes traffic across multiple backends + +#### Basic Configuration + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: multi-provider-route +spec: + # Gateway to attach to + parentRefs: + - name: my-gateway + kind: Gateway + group: gateway.networking.k8s.io + + # Routing rules + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gpt-4o-mini + backendRefs: + - name: openai-backend + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: claude-3-sonnet + backendRefs: + - name: bedrock-backend +``` + +## Resource Relationships and Data Flow + +Understanding how these resources work together is crucial for successful provider connectivity: + +```mermaid +graph TD + A[Client Request] --> B[AIGatewayRoute] + B --> C{Model Header} + C -->|gpt-4o-mini| D[OpenAI AIServiceBackend] + C -->|claude-3-sonnet| E[Bedrock AIServiceBackend] + D --> F[OpenAI BackendSecurityPolicy] + E --> G[AWS BackendSecurityPolicy] + F --> H[OpenAI API] + G --> I[AWS Bedrock API] + + style A fill:#e1f5fe + style B fill:#f3e5f5 + style D fill:#e8f5e8 + style E fill:#e8f5e8 + style F fill:#fff3e0 + style G fill:#fff3e0 +``` + +### Data Flow Process + +1. **Request Reception**: Client sends a request to the AI Gateway with the OpenAI-compatible format +2. **Route Matching**: AIGatewayRoute examines request headers (like `x-ai-eg-model`) to determine the target backend +3. **Backend Resolution**: The matching rule identifies the appropriate AIServiceBackend +4. **Authentication**: The AIServiceBackend's security policy provides credentials for upstream authentication +5. **Schema Transformation**: If needed, the request is transformed from the input schema to the backend's expected schema +6. **Provider Communication**: The request is forwarded to the actual AI provider with proper authentication +7. **Response Processing**: The provider's response is transformed back to the unified schema format +8. **Client Response**: The standardized response is returned to the client + +## Common Configuration Patterns + +### Single Provider Setup + +For a simple single-provider setup: + +```yaml +# Backend configuration +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIServiceBackend +metadata: + name: openai-backend +spec: + schema: + name: OpenAI + version: v1 + backendRef: + name: openai-backend + kind: Backend + group: gateway.envoyproxy.io + +--- +# Security configuration +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: openai-auth +spec: + targetRefs: + - group: aigateway.envoyproxy.io + kind: AIServiceBackend + name: openai-backend + type: APIKey + apiKey: + secretRef: + name: openai-secret + +--- +# Routing configuration +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: openai-route +spec: + parentRefs: + - name: my-gateway + kind: Gateway + group: gateway.networking.k8s.io + rules: + - backendRefs: + - name: openai-backend +``` + +### Multi-Provider Setup with Fallback + +For high availability with multiple providers: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: multi-provider-fallback +spec: + parentRefs: + - name: my-gateway + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gpt-4o-mini + backendRefs: + - name: openai-backend + - name: groq-backend # Fallback provider +``` + +### Model-Specific Routing + +For routing different models to specialized providers: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: model-specific-routing +spec: + parentRefs: + - name: my-gateway + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gpt-4o-mini + backendRefs: + - name: openai-backend + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: claude-3-sonnet + backendRefs: + - name: bedrock-backend + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: text-embedding-ada-002 + backendRefs: + - name: openai-embeddings-backend +``` + +Configure model ownership and creation information for the `/models` endpoint: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: model-metadata +spec: + parentRefs: + - name: my-gateway + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gpt-4o-mini + modelsOwnedBy: "OpenAI" + modelsCreatedAt: "2024-01-01T00:00:00Z" + backendRefs: + - name: openai-backend +``` + +## Provider-Specific Considerations + +### OpenAI-Compatible Providers + +Many providers offer OpenAI-compatible APIs: + +- Use OpenAI schema configuration +- Adjust the version field if the provider uses custom paths +- Standard API key authentication typically applies + +### Cloud Provider Integration + +Cloud providers like AWS Bedrock and Azure OpenAI require: + +- Cloud-specific credential types (AWS IAM, Azure Service Principal, GCP Workload Identity) +- Region specification for multi-region services +- Custom schema configurations for native APIs + +### Self-Hosted Models + +Self-hosted models using frameworks like vLLM: + +- Often compatible with OpenAI schema +- May not require authentication (internal networks) +- Custom endpoints through Envoy Gateway Backend resources + +## Validation and Troubleshooting + +### Configuration Validation + +The AI Gateway validates configurations at deployment time: + +- **Schema Compatibility**: Ensures input and output schemas are compatible +- **Resource References**: Validates that referenced resources exist +- **Credential Access**: Verifies that secrets are accessible + +### Status Conditions + +All resources provide status conditions to monitor their health: + +- **Accepted**: Resource is valid and has been accepted by the controller +- **NotAccepted**: Resource has validation errors or configuration issues + +### Common Issues and Solutions + +**Authentication Failures (401/403)** + +- Verify API keys and credentials are correct +- Check secret references and key names +- Ensure credentials have necessary permissions + +**Schema Mismatch Errors** + +- Confirm the backend schema matches the provider's API +- Check version specifications for provider-specific paths +- Review API documentation for schema requirements + +**Routing Issues** + +- Verify header matching rules in AIGatewayRoute +- Check that model names match expected values +- Ensure backend references point to existing AIServiceBackends + +**Backend Reference Errors** + +- Ensure backendRef points to Envoy Gateway Backend resources +- Verify the Backend resource exists and is properly configured +- Check that the group field is set to `gateway.envoyproxy.io` + +## Next Steps + +Now that you understand the connectivity fundamentals: + +- **[Supported Providers](/docs/capabilities/llm-integrations/supported-providers)** - View the complete list of supported providers and their configurations +- **[Supported Endpoints](/docs/capabilities/llm-integrations/supported-endpoints)** - Learn about available API endpoints and their capabilities +- **[Getting Started Guide](/docs/getting-started/connect-providers)** - Follow hands-on tutorials for specific providers +- **[Traffic Management](/docs/capabilities/traffic)** - Configure advanced routing, rate limiting, and fallback strategies +- **[Security](/docs/capabilities/security)** - Implement comprehensive security policies for your AI traffic + +## API Reference + +For detailed information about resource fields and configuration options: + +- [AIServiceBackend API Reference](../../api/api.mdx#aiservicebackend) +- [BackendSecurityPolicy API Reference](../../api/api.mdx#backendsecuritypolicy) +- [AIGatewayRoute API Reference](../../api/api.mdx#aigatewayroute) diff --git a/site/versioned_docs/version-1.2/capabilities/llm-integrations/index.md b/site/versioned_docs/version-1.2/capabilities/llm-integrations/index.md new file mode 100644 index 0000000000..65f75554a0 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/llm-integrations/index.md @@ -0,0 +1,22 @@ +--- +id: llm-integrations +title: LLM Provider Integrations +--- + +# LLM Provider Integrations + +Agent Router provides integration to multiple LLM Providers. This section provides insight into the providers and endpoints available to integrate with. + +## Provider Support + +- **[Supported Providers](./supported-providers.md)** - Complete list of supported AI/LLM service providers and their configurations +- **[Supported Endpoints](./supported-endpoints.md)** - Available API endpoints and operations across different providers + +## Advanced Features + +- **[Vendor-Specific Fields](./vendor-specific-fields.md)** - Use backend-specific parameters and access provider-unique capabilities in your OpenAI-compatible requests +- **[Prompt Caching](./prompt-caching.md)** - Provider-agnostic prompt caching using unified cache_control API + +## Getting Started with Provider Connectivity + +- **[Connecting to AI Providers](./connect-providers.md)** - Learn the fundamental concepts, resources, and relationships required to establish connectivity with any supported AI provider diff --git a/site/versioned_docs/version-1.2/capabilities/llm-integrations/prompt-caching.md b/site/versioned_docs/version-1.2/capabilities/llm-integrations/prompt-caching.md new file mode 100644 index 0000000000..45d6f7a609 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/llm-integrations/prompt-caching.md @@ -0,0 +1,174 @@ +--- +id: prompt-caching +title: Prompt Caching +sidebar_position: 6 +--- + +# Prompt Caching + +Agent Router provides provider-agnostic prompt caching through a unified `cache_control` API. The same cache syntax works across multiple providers: Direct Anthropic, GCP Vertex AI (Claude models), and AWS Bedrock (Claude models). This reduces costs and improves response times by caching frequently-used content like system prompts, tool definitions, and reference documents. + +## Supported Providers + +| Provider | API Schema | Cache Support | +| ---------------------- | --------------------- | ------------- | +| Anthropic (Direct) | `Anthropic` | Native | +| GCP Vertex AI (Claude) | `GCPAnthropic` | Translated | +| AWS Bedrock (Claude) | `AWSBedrockAnthropic` | Translated | + +## How It Works + +- Add `cache_control: {"type": "ephemeral"}` to content blocks in your request. +- AI Gateway translates this to the provider-specific format automatically. +- Cache is maintained per-provider; all providers require a minimum of 1,024 tokens for caching. +- A maximum of 4 cache breakpoints are allowed per request across all providers. +- On cache hit, the provider charges reduced input token costs. + +## Usage + +No gateway-side configuration is needed. Caching is controlled entirely at the request level by adding `cache_control` to the content blocks you want cached. + +### Basic Example: System Prompt Caching + +Cache a system prompt so that subsequent requests reuse the cached content: + +```json +{ + "model": "claude-sonnet-4-5", + "messages": [ + { + "role": "system", + "content": [ + { + "type": "text", + "text": "You are a helpful assistant with extensive knowledge...(long system prompt)...", + "cache_control": { "type": "ephemeral" } + } + ] + }, + { + "role": "user", + "content": "What is the capital of France?" + } + ] +} +``` + +### Multiple Cache Points + +You can place up to 4 cache breakpoints in a single request to cache different parts of the conversation: + +```json +{ + "model": "claude-sonnet-4-5", + "messages": [ + { + "role": "system", + "content": [ + { + "type": "text", + "text": "System instructions...", + "cache_control": { "type": "ephemeral" } + } + ] + }, + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Reference document content...", + "cache_control": { "type": "ephemeral" } + }, + { + "type": "text", + "text": "Question about the document" + } + ] + } + ] +} +``` + +### Tool Definition Caching + +Cache complex tool schemas that remain the same across requests: + +```json +{ + "model": "claude-sonnet-4-5", + "messages": [ + { + "role": "user", + "content": "Help me search for information about cloud computing trends." + } + ], + "tools": [ + { + "type": "function", + "function": { + "name": "search_knowledge_base", + "description": "Search through a comprehensive knowledge base...", + "parameters": { + "type": "object", + "properties": { + "query": { + "type": "string", + "description": "Natural language search query" + } + }, + "required": ["query"] + }, + "cache_control": { "type": "ephemeral" } + } + } + ] +} +``` + +## Response Format + +When caching is active, the response includes cache information in the `usage` field: + +```json +{ + "usage": { + "prompt_tokens": 2000, + "completion_tokens": 150, + "prompt_tokens_details": { + "cached_tokens": 1800 + } + } +} +``` + +- `cached_tokens` indicates the number of tokens served from cache at a reduced cost. +- Cache write tokens are tracked internally for billing purposes. + +## Best Practices + +:::tip + +- Place `cache_control` on content that exceeds 1,024 tokens. Content below this threshold will not be cached. +- Cache system prompts, tool definitions, and reference documents that do not change between requests. +- Position cache breakpoints strategically -- cached content must appear at the beginning of the message. +- Monitor `cached_tokens` in responses to verify caching effectiveness and measure cost savings. + ::: + +## Provider-Specific Notes + +:::note + +- **All providers**: Minimum 1,024 tokens per cached block, maximum 4 cache breakpoints per request. +- **Anthropic Direct**: Uses the native `cache_control` field directly with no translation. +- **GCP Vertex AI**: AI Gateway translates `cache_control` to Vertex AI's caching format automatically. +- **AWS Bedrock**: AI Gateway translates `cache_control` to Bedrock's cachePoint format automatically. +- All providers support the `"ephemeral"` cache type. +- Existing requests without `cache_control` continue to work with no changes. + ::: + +## Further Reading + +- [Prompt Caching Examples](https://github.com/theagentrouter/agent-router/tree/main/examples/cache) -- Detailed examples with curl commands for each provider. +- [Connecting to GCP Vertex AI](../../getting-started/connect-providers/gcp-vertexai.md) -- Set up GCP Vertex AI as a provider. +- [Connecting to AWS Bedrock](../../getting-started/connect-providers/aws-bedrock.md) -- Set up AWS Bedrock as a provider. diff --git a/site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-endpoints.md b/site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-endpoints.md new file mode 100644 index 0000000000..ed83453c1c --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-endpoints.md @@ -0,0 +1,672 @@ +--- +id: supported-endpoints +title: Supported API Endpoints +sidebar_position: 9 +--- + +The Agent Router provides OpenAI-compatible API endpoints as well as the Anthropic-compatible API for routing and managing LLM/AI traffic. This page documents which OpenAI API endpoints and Anthropic-compatible API endpoints are currently supported and their capabilities. + +## Overview + +The Agent Router acts as a proxy that accepts OpenAI-compatible and Anthropic-compatible requests and routes them to various AI providers. While it maintains compatibility with the OpenAI API specification, it currently supports a subset of the full OpenAI API. + +## Supported Endpoints + +### Chat Completions + +**Endpoint:** `POST /v1/chat/completions` + +**Status:** ✅ Fully Supported + +**Description:** Create a chat completion response for the given conversation. + +**Features:** + +- ✅ Streaming and non-streaming responses +- ✅ Function calling +- ✅ Response format specification (including JSON schema) +- ✅ Temperature, top_p, and other sampling parameters +- ✅ System and user messages +- ✅ Audio and video inputs +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Token usage tracking and cost calculation +- ✅ Provider fallback and load balancing + +**Supported Providers:** + +- OpenAI +- AWS Bedrock (with automatic translation) +- Azure OpenAI (with automatic translation) +- GCP VertexAI (with automatic translation) +- GCP Anthropic (with automatic translation) +- Any OpenAI-compatible provider (Groq, Together AI, Mistral, Tetrate Agent Router Service, etc.) + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4o-mini", + "messages": [ + { + "role": "user", + "content": "Hello, how are you?" + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +### Anthropic Messages + +**Endpoint:** `POST /anthropic/v1/messages` + +**Status:** ✅ Fully Supported + +**Description:** Send a structured list of input messages with text and/or image content, and the model will generate the next message in the conversation. + +**Features:** + +- ✅ Streaming and non-streaming responses +- ✅ Function calling +- ✅ Extended thinking +- ✅ Response format specification (including JSON schema) +- ✅ Temperature, top_p, and other sampling parameters +- ✅ System and user messages +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Token usage tracking and cost calculation +- ✅ Provider fallback and load balancing + +**Supported Providers:** + +- Anthropic +- GCP Anthropic +- AWS Anthropic +- AWS Bedrock + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "claude-sonnet-4", + "messages": [ + { + "role": "user", + "content": "Hello, how are you?" + } + ], + "max_tokens": 100 + }' \ + $GATEWAY_URL/anthropic/v1/messages +``` + +### Anthropic Count Tokens + +**Endpoint:** `POST /anthropic/v1/messages/count_tokens` + +**Status:** ✅ Fully Supported + +**Description:** Count the number of input tokens for a Messages API request without actually creating a message. Useful for estimating costs and validating request sizes before sending. + +**Features:** + +- ✅ Token counting for messages, system prompts, and tools +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Provider fallback and load balancing + +**Supported Providers:** + +- Anthropic +- GCP Anthropic +- AWS Anthropic + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -H "anthropic-version: 2023-06-01" \ + -d '{ + "model": "claude-sonnet-4", + "messages": [ + { + "role": "user", + "content": "Hello, how are you?" + } + ] + }' \ + $GATEWAY_URL/anthropic/v1/messages/count_tokens +``` + +**Response:** + +```json +{ + "input_tokens": 14 +} +``` + +### Completions + +**Endpoint:** `POST /v1/completions` + +**Status:** ✅ Fully Supported + +**Description:** Create a text completion for the given prompt (legacy endpoint). + +**Features:** + +- ✅ Non-streaming responses +- ✅ Streaming responses +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Temperature, top_p, and other sampling parameters +- ✅ Single and batch prompt processing +- ✅ Token usage tracking and cost calculation +- ✅ Provider fallback and load balancing +- ✅ Full metrics support (token usage, request duration, time to first token, inter-token latency) + +**Supported Providers:** + +- OpenAI +- Any OpenAI-compatible provider that supports completions + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "babbage-002", + "prompt": "def fib(n):\n if n <= 1:\n return n\n else:\n return fib(n-1) + fib(n-2)", + "max_tokens": 25, + "temperature": 0.4, + "top_p": 0.9 + }' \ + $GATEWAY_URL/v1/completions +``` + +### Embeddings + +**Endpoint:** `POST /v1/embeddings` + +**Status:** ✅ Fully Supported + +**Description:** Create embeddings for the given input text. + +**Features:** + +- ✅ Single and batch text embedding +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Token usage tracking and cost calculation +- ✅ Provider fallback and load balancing + +**Supported Providers:** + +- OpenAI +- AWS Bedrock (Titan models, with automatic translation) +- GCP VertexAI (with automatic translation) +- Any OpenAI-compatible provider that supports embeddings, including Azure OpenAI. + +### Image Generation + +**Endpoint:** `POST /v1/images/generations` + +**Status:** ✅ Supported + +**Description:** Generate one or more images from a text prompt using OpenAI-compatible models. + +**Features:** + +- **Non-streaming responses**: Returns JSON payload with image URLs or base64 content +- **Model selection**: Via request body `model` or `x-ai-eg-model` header +- **Parameters**: `prompt`, `size`, `n`, `quality`, `response_format` +- **Metrics**: Records image count, model, and size; token usage when provided +- **Provider fallback and load balancing** + +**Supported Providers:** + +- OpenAI +- Any OpenAI-compatible provider that supports image generations + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-image-1", + "prompt": "a serene mountain landscape at sunrise in watercolor", + "size": "1024x1024", + "n": 1 + }' \ + $GATEWAY_URL/v1/images/generations +``` + +### Audio Transcriptions + +**Endpoint:** `POST /v1/audio/transcriptions` + +**Status:** ✅ Supported + +**Description:** Transcribe audio into text in the language of the audio. + +**Features:** + +- ✅ Multipart/form-data file upload (OpenAI-compatible) +- ✅ Model selection via form field `model` or `x-ai-eg-model` header +- ✅ Optional parameters: `language`, `prompt`, `response_format`, `temperature`, `timestamp_granularities[]` +- ✅ JSON and verbose JSON response formats +- ✅ Provider fallback and load balancing +- ✅ Model name virtualization (override model names for backends) + +**Supported Providers:** + +- OpenAI +- Any OpenAI-compatible provider that supports audio transcriptions + +**Example:** + +```bash +curl -F "model=whisper-1" \ + -F "file=@audio.mp3" \ + -F "language=en" \ + $GATEWAY_URL/v1/audio/transcriptions +``` + +### Audio Translations + +**Endpoint:** `POST /v1/audio/translations` + +**Status:** ✅ Supported + +**Description:** Translate audio into English text. + +**Features:** + +- ✅ Multipart/form-data file upload (OpenAI-compatible) +- ✅ Model selection via form field `model` or `x-ai-eg-model` header +- ✅ Optional parameters: `prompt`, `response_format`, `temperature` +- ✅ Provider fallback and load balancing +- ✅ Model name virtualization (override model names for backends) + +**Supported Providers:** + +- OpenAI +- Any OpenAI-compatible provider that supports audio translations + +**Example:** + +```bash +curl -F "model=whisper-1" \ + -F "file=@audio.mp3" \ + $GATEWAY_URL/v1/audio/translations +``` + +### Responses + +**Endpoint:** `POST /v1/responses` + +**Status:** ✅ Fully Supported + +**Description:** Creates a model response. Provide text or image inputs to generate text or JSON outputs. Have the model call your own custom code or use built-in tools. + +**Features:** + +- ✅ Streaming and non-streaming responses +- ✅ Function calling +- ✅ MCP Tools support +- ✅ Reasoning +- ✅ Multi-turn conversations +- ✅ Native multimodal support for text and images +- ✅ Response format specification (including JSON schema) +- ✅ Temperature, top_p, and other sampling parameters +- ✅ System and user messages +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Token usage tracking and cost calculation +- ✅ Provider fallback and load balancing + +**Supported Providers:** + +- OpenAI +- Azure OpenAI with an API version that supports Responses, such as `2025-04-01-preview` +- Any OpenAI-compatible provider (Groq, Together AI, Mistral, Tetrate Agent Router Service, etc.) + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4.1", + "input": [ + { + "role": "user", + "content": [ + {"type": "input_text", "text": "what is in this image?"}, + { + "type": "input_image", + "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" + } + ] + } + ] + }' \ + $GATEWAY_URL/v1/responses +``` + +### Responses Input Tokens + +**Endpoint:** `POST /v1/responses/input_tokens` + +**Status:** ✅ Supported + +**Description:** Count the number of input tokens for a Responses API request without generating a response. Accepts the same request body as `/v1/responses` and returns the input token count. This is useful for validating context window fit and estimating cost before making an inference call. + +**Features:** + +- ✅ Same request body format as `/v1/responses` +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Token usage tracking +- ✅ Provider fallback and load balancing + +**Supported Providers:** + +- OpenAI +- Azure OpenAI (with automatic `api-version` injection) + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4.1", + "input": "Hello, how are you?", + "instructions": "You are a helpful assistant." + }' \ + $GATEWAY_URL/v1/responses/input_tokens +``` + +**Response Format:** + +```json +{ + "input_tokens": 15 +} +``` + +### Rerank + +**Endpoint:** `POST /cohere/v2/rerank` + +**Status:** ✅ Fully Supported + +**Description:** Rerank a list of documents for a given query to return relevance scores and an ordered list. Cohere-compatible API. + +**Features:** + +- ✅ Single-query document reranking +- ✅ Model selection via request body or `x-ai-eg-model` header +- ✅ Token usage tracking and cost calculation +- ✅ Provider fallback and load balancing + +**Supported Providers:** + +- Cohere +- Any Cohere-compatible provider that supports rerank, including vLLM. + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "rerank-english-v3.0", + "query": "What is the capital of France?", + "documents": [ + "Paris is the capital of France.", + "Berlin is the capital of Germany." + ] + }' \ + $GATEWAY_URL/cohere/v2/rerank +``` + +### System One (TypeSafe Jev) + +**Endpoint:** `POST /typesafe/v1/systemone` + +**Status:** ✅ Fully Supported + +**Description:** Evaluate application state against typed questions and return structured decisions with probabilities and confidence. Native [TypeSafe AI](https://docs.typesafe.ai/api.md) API, passed through unchanged, so the official TypeSafe SDKs work by changing only the base URL. + +**Features:** + +- ✅ `noul` (yes/no probability), `choice` (single selection) and `score` (rubric) questions, batched in one call +- ✅ Model selection via request body or `x-ai-eg-model` header, including the `jev-latest` and `jev-preview` aliases +- ✅ Token usage tracking and cost calculation (TypeSafe bills input tokens only) +- ✅ Provider fallback and load balancing +- ❌ Streaming (not offered by the TypeSafe API) + +**Supported Providers:** + +- TypeSafe AI + +**Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "jev-latest", + "state": {"ticket": "I was charged twice for order #4471."}, + "questions": { + "is_billing": {"type": "noul", "instructions": "Is this ticket about billing?"}, + "team": { + "type": "choice", + "instructions": "Which team should handle this?", + "criteria": {"billing": null, "shipping": null, "other": null} + } + } + }' \ + $GATEWAY_URL/typesafe/v1/systemone +``` + +### Tokenize + +**Endpoint:** `POST /tokenize` + +**Status:** ✅ Supported + +**Description:** Count tokens for text input without generating a response. Useful for cost estimation, prompt optimization, and understanding model input limits. The request format is compatible with the [vLLM tokenize API](https://docs.vllm.ai/en/latest/api/tokenization.html). + +**Features:** + +- ✅ Chat message tokenization (OpenAI messages format) +- ✅ Completion prompt tokenization (single string prompt) +- ✅ Model selection via `model` field in request body +- ✅ Tool/function call tokenization support +- ✅ Provider fallback and load balancing +- ✅ Metrics support (request duration) + +**Supported Providers:** + +| Provider | API Schema | Translation Target | Notes | +| ----------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| OpenAI-compatible (e.g., vLLM) | `OpenAI` | Passthrough | vLLM natively supports `/tokenize`. OpenAI itself does not offer a tokenize REST API. | +| GCP Vertex AI (Gemini) | `GCPVertexAI` | [Gemini CountTokens API](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/count-tokens) | Supports `media_resolution` parameter. | +| GCP Anthropic (Claude on Vertex AI) | `GCPAnthropic` | [Anthropic MessageCountTokens API](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude/count-tokens) | Uses `rawPredict` method with `count-tokens` virtual model. | +| AWS Bedrock | `AWSBedrock` | [AWS Bedrock CountTokens API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_CountTokens.html) | Supports models that implement the Converse API. | +| AWS Bedrock (Anthropic) | `AWSAnthropic` | [AWS Bedrock CountTokens API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_CountTokens.html) | Uses the InvokeModel-style CountTokens API with the Anthropic Messages body. | + +**Chat Message Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "meta-llama/Llama-3.1-8B-Instruct", + "messages": [ + { + "role": "system", + "content": "You are a helpful assistant." + }, + { + "role": "user", + "content": "How many tokens is this message?" + } + ] + }' \ + $GATEWAY_URL/tokenize +``` + +**Completion Prompt Example:** + +```bash +curl -H "Content-Type: application/json" \ + -d '{ + "model": "meta-llama/Llama-3.1-8B-Instruct", + "prompt": "Once upon a time, in a land far away" + }' \ + $GATEWAY_URL/tokenize +``` + +**Response Format:** + +For translated backends (GCP Vertex AI, GCP Anthropic, AWS Bedrock, AWS Bedrock Anthropic), the response contains only the token count: + +```json +{ + "count": 15 +} +``` + +For OpenAI-compatible backends that natively support tokenization (e.g., vLLM), the response contains the token count and additional fields may be present depending on request parameters: + +```json +{ + "count": 15, + "max_model_len": 131072, + "tokens": [1234, 5678, 9012], + "token_strs": ["Hello", " world", "!"] +} +``` + +**Configuration Notes:** + +- For **vLLM backends**: Configure with `OpenAI` schema. vLLM natively provides `/tokenize` and the gateway passes the request through. +- For **GCP Vertex AI**: Configure with `GCPVertexAI` schema. Requests are automatically translated to the Gemini CountTokens API. Completion prompts are automatically converted to chat messages. +- For **GCP Anthropic**: Configure with `GCPAnthropic` schema. Requests are translated to the Anthropic MessageCountTokens API via `rawPredict`. Completion prompts are automatically converted to chat messages. Model version suffixes (`@default`, `@latest`) are automatically stripped. +- For **AWS Bedrock**: Configure with `AWSBedrock` schema. Requests are translated to the AWS Bedrock CountTokens API using the Converse-style input. Completion prompts are automatically converted to chat messages. Cross-region inference (CRIS) model ID prefixes are automatically stripped. +- For **AWS Bedrock (Anthropic)**: Configure with `AWSAnthropic` schema. Requests are translated to the AWS Bedrock CountTokens API using the InvokeModel-style Anthropic Messages body. Completion prompts are automatically converted to chat messages. Cross-region inference (CRIS) model ID prefixes are automatically stripped. + +### Models + +**Endpoint:** `GET /v1/models` + +**Description:** List available models configured in the AI Gateway. + +**Features:** + +- ✅ Returns models declared in AIGatewayRoute configurations +- ✅ OpenAI-compatible response format +- ✅ Model metadata (ID, owned_by, created timestamp) + +**Example:** + +```bash +curl $GATEWAY_URL/v1/models +``` + +**Response Format:** + +```json +{ + "object": "list", + "data": [ + { + "id": "gpt-4o-mini", + "object": "model", + "created": 1677610602, + "owned_by": "openai" + } + ] +} +``` + +## Provider-Endpoint Compatibility Table + +The following table summarizes which providers support which endpoints: + +| Provider | Chat Completions | Completions | Embeddings | Image Generation | Anthropic Messages | Count Tokens | Rerank | System One | Tokenize | Notes | +| ----------------------------------------------------------------------------------------------------- | :--------------: | :---------: | :--------: | :--------------: | :----------------: | :----------: | :----: | :--------: | :------: | -------------------------------------------------------------------------------------------------------------------- | +| [OpenAI](https://platform.openai.com/docs/api-reference) | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | OpenAI does not offer a tokenize REST API | +| [AWS Bedrock](https://docs.aws.amazon.com/bedrock/latest/APIReference/) | ✅ | 🚧 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via API translation (embeddings: Titan models only) | +| [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference) | ✅ | 🚧 | ✅ | ❌ | ⚠️ | ❌ | ❌ | ❌ | ❌ | Via API translation or via [OpenAI-compatible API](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/latest) | +| [Google Gemini](https://ai.google.dev/gemini-api/docs/openai) | ✅ | ⚠️ | ✅ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Groq](https://console.groq.com/docs/openai) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Grok](https://docs.x.ai/docs/api-reference) | ✅ | ⚠️ | ❌ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Together AI](https://docs.together.ai/docs/openai-api-compatibility) | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Cohere](https://docs.cohere.com/v2/docs/compatibility-api) | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | Via OpenAI-compatible API and Cohere V2 API for rerank | +| [Mistral](https://docs.mistral.ai/api/) | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [DeepInfra](https://deepinfra.com/docs/inference) | ✅ | ⚠️ | ✅ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [DeepSeek](https://api-docs.deepseek.com/) | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Hunyuan](https://cloud.tencent.com/document/product/1729/111007) | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Tencent LLM Knowledge Engine](https://www.tencentcloud.com/document/product/1255/70381) | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Tetrate Agent Router Service (TARS)](https://router.tetrate.ai/) | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Google Vertex AI](https://cloud.google.com/vertex-ai/docs/reference/rest) | ✅ | 🚧 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | Via API translation | +| [Anthropic on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude) | ✅ | ❌ | 🚧 | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | Via API translation | +| [Anthropic on AWS Bedrock](https://aws.amazon.com/bedrock/anthropic/) | 🚧 | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | Native Anthropic API | +| [SambaNova](https://docs.sambanova.ai/sambastudio/latest/open-ai-api.html) | ✅ | ⚠️ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API | +| [Anthropic](https://docs.claude.com/en/home) | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | Via OpenAI-compatible API and Native Anthropic API | +| [vLLM](https://docs.vllm.ai/en/latest/) | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | Via OpenAI-compatible API; native `/tokenize` support | +| [TypeSafe AI](https://docs.typesafe.ai/api.md) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | Native System One API for the Jev decision model | + +- ✅ - Supported and Tested on Agent Router CI +- ⚠️️ - Expected to work based on provider documentation, but not tested on the CI. +- ❌ - Not supported according to provider documentation. +- 🚧 - Unimplemented, or under active development but planned for future releases + +## Custom endpoint prefixes + +By default, the gateway registers provider endpoints under these prefixes: + +- OpenAI: `/` +- Cohere: `/cohere` +- Anthropic: `/anthropic` +- TypeSafe: `/typesafe` + +You can override them via Helm using values under `endpointConfig`: + +```yaml +# values.yaml +endpointConfig: + # Explicit provider roots + openai: "" + cohere: "/cohere" + anthropic: "/anthropic" + typesafe: "/typesafe" + # rootPrefix applies to all routes; final paths are /... + # endpointConfig: + # rootPrefix: "/" +``` + +Or with helm CLI: + +```bash +helm upgrade --install ai-gateway envoyproxy/ai-gateway-helm \ + -n envoy-ai-gateway-system --create-namespace \ + --set 'endpointConfig.openai=/' \ + --set 'endpointConfig.cohere=/cohere' \ + --set 'endpointConfig.anthropic=/anthropic' \ + --set 'endpointConfig.typesafe=/typesafe' +``` + +Notes: + +- `endpointConfig.rootPrefix` (default `/`) is prepended to all provider prefixes. +- Only these keys are accepted: `openai`, `cohere`, `anthropic`, `typesafe`. +- If any key is omitted or empty, defaults are applied as listed above. + +## What's Next + +To learn more about configuring and using the Agent Router with these endpoints: + +- **[Supported Providers](./supported-providers.md)** - Complete list of supported AI providers and their configurations +- **[Usage-Based Rate Limiting](../traffic/usage-based-ratelimiting.md)** - Configure token-based rate limiting and cost controls +- **[Provider Fallback](../traffic/provider-fallback.md)** - Set up automatic failover between providers for high availability +- **[Metrics and Monitoring](../observability/metrics.md)** - Monitor usage, costs, and performance metrics + +[issue#609]: https://github.com/theagentrouter/agent-router/issues/609 diff --git a/site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-providers.md b/site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-providers.md new file mode 100644 index 0000000000..a01b18e378 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/llm-integrations/supported-providers.md @@ -0,0 +1,45 @@ +--- +id: supported-providers +title: Supported AI Providers +sidebar_position: 8 +--- + +Since the Agent Router is designed to provide a Unified API for routing and managing LLM/AI traffic, it supports various AI providers out of the box. +A "support of provider" means two things: the API schema support and the Authentication support. \ +The former can be configured in the `AIServiceBackend` resource's `schema` field, while the latter is configured in the `BackendSecurityPolicy`. + +Below is a table of currently supported providers and their respective configurations. + +| Provider Name | API Schema Config on [AIServiceBackend] | Upstream Authentication Config on [BackendSecurityPolicy] | Status | Note | +| --------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------: | :-------------------------------------------------------: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [OpenAI](https://platform.openai.com/docs/api-reference) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [AWS Bedrock](https://docs.aws.amazon.com/bedrock/latest/APIReference/) | `{"name":"AWSBedrock"}` | [AWS Bedrock Credentials] | ✅ | | +| [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference) | `{"name":"AzureOpenAI","version":"2025-01-01-preview"}` or `{"name":"OpenAI", "prefix": "/openai/v1"}` | [Azure Credentials] or [Azure API Key] | ✅ | | +| [Google Gemini on AI Studio](https://ai.google.dev/gemini-api/docs/openai) | `{"name":"OpenAI","prefix":"/v1beta/openai"}` | [API Key] | ✅ | Only the OpenAI compatible endpoint | +| [Google Vertex AI](https://cloud.google.com/vertex-ai/docs/reference/rest) | `{"name":"GCPVertexAI"}` | [GCP Credentials] | ✅ | Supports ADC, Service Account Keys, and Workload Identity Federation | +| [Anthropic on GCP Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude) | `{"name":"GCPAnthropic", "version":"vertex-2023-10-16"}` | [GCP Credentials] | ✅ | Native Anthropic and OpenAI endpoints. Supports ADC, Service Account Keys, and Workload Identity Federation | +| [Groq](https://console.groq.com/docs/openai) | `{"name":"OpenAI","prefix":"/openai/v1"}` | [API Key] | ✅ | | +| [Grok](https://docs.x.ai/docs/api-reference?utm_source=chatgpt.com#chat-completions) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [Together AI](https://docs.together.ai/docs/openai-api-compatibility) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [Bitdeer AI](https://developers.bitdeer.ai/docs/api) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | Only the OpenAI-compatible endpoints | +| [Cohere](https://docs.cohere.com/v2/docs/compatibility-api) | `{"name":"Cohere","version":"v2"}` or `{"name":"OpenAI","prefix":"/compatibility/v1"}` | [API Key] | ✅ | Supports native Cohere v2 (e.g., /cohere/v2/rerank) and OpenAI-compatible endpoints. | +| [TypeSafe AI](https://docs.typesafe.ai/api.md) | `{"name":"TypeSafe","version":"v1"}` | [API Key] | ✅ | Native System One API for the Jev decision model (e.g., /typesafe/v1/systemone). `version` defaults to `v1`. | +| [Mistral](https://docs.mistral.ai/api/#tag/chat/operation/chat_completion_v1_chat_completions_post) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [DeepInfra](https://deepinfra.com/docs/inference) | `{"name":"OpenAI","prefix":"/v1/openai"}` | [API Key] | ✅ | Only the OpenAI compatible endpoint | +| [DeepSeek](https://api-docs.deepseek.com/) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [Hunyuan](https://cloud.tencent.com/document/product/1729/111007) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [Tencent LLM Knowledge Engine](https://www.tencentcloud.com/document/product/1255/70381?lang=en) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [Tetrate Agent Router Service (TARS)](https://router.tetrate.ai/) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| [SambaNova](https://docs.sambanova.ai/sambastudio/latest/open-ai-api.html) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | | +| Self-hosted-models | `{"name":"OpenAI","prefix":"/v1"}` | N/A | ⚠️ | Depending on the API schema spoken by self-hosted servers. For example, [vLLM] speaks the OpenAI format. Also, API Key auth can be configured as well. | +| [Anthropic](https://docs.claude.com/en/home) | `{"name":"Anthropic"}` | [Anthropic API Key] | ✅ | Support only Native Anthropic messages endpoint | + +[AIServiceBackend]: api/api.mdx#aiservicebackendspec +[BackendSecurityPolicy]: api/api.mdx#backendsecuritypolicyspec +[API Key]: api/api.mdx#backendsecuritypolicyapikey +[AWS Bedrock Credentials]: api/api.mdx#backendsecuritypolicyawscredentials +[GCP Credentials]: api/api.mdx#backendsecuritypolicygcpcredentials +[Azure Credentials]: api/api.mdx#backendsecuritypolicyazurecredentials +[Azure API Key]: api/api.mdx#backendsecuritypolicyazureapikey +[Anthropic API Key]: api/api.mdx#backendsecuritypolicyanthropicapikey +[vLLM]: https://docs.vllm.ai/en/v0.8.3/serving/openai_compatible_server.html diff --git a/site/versioned_docs/version-1.2/capabilities/llm-integrations/vendor-specific-fields.md b/site/versioned_docs/version-1.2/capabilities/llm-integrations/vendor-specific-fields.md new file mode 100644 index 0000000000..3fc764f2d8 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/llm-integrations/vendor-specific-fields.md @@ -0,0 +1,207 @@ +--- +id: vendor-specific-fields +title: Extension Fields +--- + +# Extension Fields + +The AI Gateway supports extension fields that allow you to specify unified or backend-specific parameters directly as inline fields in your OpenAI-compatible requests. These fields are applied during the translation process to the target backend's native API format. + +## Overview + +Extension fields enable you to: + +- Use advanced backend-specific features not available in the OpenAI API +- Use unified configuration fields that work across multiple providers not available in the OpenAI API + +### Vendor Extension Fields + +Vendor specific fields are specified as inline fields in your OpenAI request and are applied after the standard OpenAI-to-backend translation. + +### Unified Extension Fields + +For thinking/reasoning capabilities, you can use a unified `thinking` field that automatically translates to the correct backend-specific format: + +- **GCP Vertex AI (Gemini)**: Translates to `generationConfig.thinkingConfig` +- **GCP Anthropic**: Uses `thinking` field directly +- **AWS Bedrock**: Uses `thinking` field directly + +This unified approach allows you to write provider-agnostic requests while still leveraging thinking capabilities. + +## Supported Backends + +The following backends support extension fields: + +### GCP Vertex AI (Gemini) + +- **API Schema Name**: `GCPVertexAI` +- **Supported Fields**: + - `safetySettings`: Configure the safety settings for gemini models that translates to `SafetySetting`. [Gemini Docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/multimodal/configure-safety-filters) + - `thinking`: Configure thinking process for reasoning models that automatically translates to `generationConfig.thinkingConfig`. [Gemini Docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/thinking) +- **Supported Tools**: + - `google_search`: Enable Google Search grounding for Gemini models. Configuration options vary by platform: `exclude_domains` and `blocking_confidence` are Vertex AI only, while `time_range_filter` is Gemini API only. [Google Search Grounding Docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/grounding-with-google-search) + +### GCP Anthropic + +- **API Schema Name**: `GCPAnthropic` +- **Supported Fields**: + - `thinking`: Configuration for enabling Claude's extended thinking. [Anthropic Docs](https://docs.anthropic.com/en/api/messages#body-thinking) +- **Supported Tool Fields** (set on an entry of `tools[].function`): + - `eager_input_streaming`: Stream a tool's input as the model generates it, instead of buffering and validating each parameter value first. [Anthropic Docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/fine-grained-tool-streaming) + +### AWS Anthropic + +- **API Schema Name**: `AWSAnthropic` +- **Supported Tool Fields** (set on an entry of `tools[].function`): + - `eager_input_streaming`: Stream a tool's input as the model generates it, instead of buffering and validating each parameter value first. [Anthropic Docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/fine-grained-tool-streaming) + +### AWS Bedrock + +- **API Schema Name**: `AWSBedrock` +- **Supported Fields**: + - `thinking`: Configuration for enabling Anthropic Claude's extended thinking. [AWS Docs](https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-extended-thinking.html) + +## Usage + +Add extension fields directly as inline fields in your OpenAI request: + +### Using Unified Thinking Configuration + +The simplest way to enable thinking capabilities across all providers is to use the unified `thinking` field: + +```json +{ + "model": "gemini-2.5-pro", + "messages": [ + { + "role": "user", + "content": "Explain quantum computing and show me a simple code example." + } + ], + "temperature": 0.7, + "max_tokens": 2000, + "thinking": { + "type": "enabled", + "budget_tokens": 1000, + "includeThoughts": true + } +} +``` + +This configuration will work with any provider that supports thinking, automatically translating to the correct backend format. + +### Using Provider-Specific Fields + +For more fine-grained control or provider-specific features, you can use the vendor-specific fields like `safetySettings` for gemini models: + +```json +{ + "model": "gemini-1.5-pro", + "messages": [ + { + "role": "user", + "content": "Explain quantum computing and show me a simple code example." + } + ], + "temperature": 0.7, + "max_tokens": 2000, + "safetySettings": [ + { + "category": "HARM_CATEGORY_HARASSMENT", + "threshold": "BLOCK_ONLY_HIGH" + } + ] +} +``` + +### Using Google Search Grounding + +To enable Google Search grounding for Gemini models, add `google_search` to the tools array. + +For basic usage without configuration options: + +```json +{ + "model": "gemini-2.0-flash", + "messages": [ + { + "role": "user", + "content": "What are the latest developments in quantum computing?" + } + ], + "tools": [ + { + "type": "google_search" + } + ] +} +``` + +For Vertex AI, you can add filtering options: + +```json +{ + "model": "gemini-2.0-flash", + "messages": [ + { + "role": "user", + "content": "What are the latest developments in quantum computing?" + } + ], + "tools": [ + { + "type": "google_search", + "google_search": { + "exclude_domains": ["example.com"], + "blocking_confidence": "BLOCK_LOW_AND_ABOVE" + } + } + ] +} +``` + +### Using Eager Tool Input Streaming + +By default, Anthropic buffers each tool parameter value and validates it before emitting it to the stream. Setting `eager_input_streaming` on a tool makes the model stream that tool's input as it is generated, so `tool_calls[].function.arguments` fragments start arriving sooner. This matches the per-token streaming behavior of OpenAI and most other backends. + +```json +{ + "model": "gcp.claude-3-5-haiku", + "messages": [ + { + "role": "user", + "content": "What's the weather in New York?" + } + ], + "stream": true, + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "eager_input_streaming": true, + "parameters": { + "type": "object", + "properties": { + "location": { "type": "string" } + } + } + } + } + ] +} +``` + +The field is a tri-state. Omitting it defaults to buffered streaming, `true` enables eager streaming for that tool, and an explicit `false` also buffers but holds that even when the deprecated `fine-grained-tool-streaming-2025-05-14` beta header is present, which otherwise turns unset tools on. + +How arguments are accumulated does not change: clients concatenate the `arguments` fragments exactly as before. What changes is the guarantee. With eager streaming the server no longer validates the fragments, so the accumulated string is not guaranteed to be valid JSON — a response ending with `finish_reason: "length"` can cut a parameter off midway. Guard the parse and handle failure rather than assuming it succeeds. See [Accumulating tool input deltas](https://platform.claude.com/docs/en/agents-and-tools/tool-use/fine-grained-tool-streaming#accumulating-tool-input-deltas). + +### Field Conflicts + +Vendor fields override translated fields when conflicts occur. + +When using unified thinking configuration, the `thinking` field takes precedence over any provider-specific thinking configurations. + +### Unsupported Fields/Backends + +Fields and Backends other than specified in [Supported Backends](#supported-backends) will be ignored. diff --git a/site/versioned_docs/version-1.2/capabilities/mcp/index.md b/site/versioned_docs/version-1.2/capabilities/mcp/index.md new file mode 100644 index 0000000000..ad94230a97 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/mcp/index.md @@ -0,0 +1,475 @@ +--- +id: mcp +title: Model Context Protocol (MCP) Gateway +sidebar_position: 8 +--- + +Agent Router provides first-class support for [Model Context Protocol](https://modelcontextprotocol.io/) (MCP), enabling AI agents to securely connect to external tools and data sources. + +This guide provides an overview of the MCP Gateway capabilities and how to configure routing to MCP servers using the `MCPRoute` API. + +## Overview + +Agent Router's MCP support allows you to: + +- **Aggregate multiple MCP servers** into a single unified endpoint +- **Apply security policies** including OAuth authentication, fine-grained access control over the tool access, and upstream API key injection +- **Filter tools** to control which capabilities are exposed to clients +- **Leverage Envoy's networking** for load balancing, rate limiting, circuit breaking, and observability + +The MCP Gateway acts as a transparent proxy between MCP clients (AI agents like Claude, Goose, etc.) and backend MCP servers, providing the same production-grade features available for LLM traffic. + +## Key Features + +| Feature | Description | +| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Streamable HTTP Transport** | Full support for MCP's streamable HTTP transport, aligning with the [June 2025 MCP spec](https://modelcontextprotocol.io/specification/2025-06-18).
Efficient handling of stateful sessions and multi-part JSON-RPC messaging over persistent HTTP connections. | +| **Fine-Grained Authorization** | Native enforcement of [OAuth authentication flows](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization).
Implement granular access control using JWT claims, scopes, and CEL expressions. | +| **Server Multiplexing & Tool Routing** | Route tool calls to the right MCP backends, aggregating and filtering available tools based on gateway policy.
Dynamically merge streaming notifications from multiple MCP servers into a unified interface. | +| **Upstream Authentication** | Built-in upstream authentication primitives to securely connect to external MCP servers using API keys and header injection. | +| **Full MCP Spec Coverage** | Complete [June 2025 MCP spec](https://modelcontextprotocol.io/specification/2025-06-18) compliance, including support for tool calls, notifications, prompts, resources, and bi-directional server-to-client requests. | +| **Built-in Observability** | OpenTelemetry tracing and Prometheus metrics for all MCP requests, using the same observability stack as LLM traffic. | + +## Architecture + +The MCP Gateway is implemented as a lightweight proxy component within the Agent Router sidecar, leveraging Envoy's battle-tested networking stack for all connection handling. + +```mermaid +sequenceDiagram + participant Client as MCP Client
(AI Agent) + participant Gateway as Agent Router + participant MCP1 as MCP Server 1 + participant MCP2 as MCP Server 2 + + Client->>Gateway: Initialize session + Gateway->>MCP1: Initialize + Gateway->>MCP2: Initialize + MCP1-->>Gateway: Session ID 1 + MCP2-->>Gateway: Session ID 2 + Gateway-->>Client: Unified Session ID + + Client->>Gateway: List tools + Gateway->>MCP1: List tools + Gateway->>MCP2: List tools + MCP1-->>Gateway: Tools A, B + MCP2-->>Gateway: Tools X, Y + Gateway-->>Client: Filtered & merged tools + + Client->>Gateway: Call tool (server1__toolA) + Gateway->>MCP1: Call toolA + MCP1-->>Gateway: Result + Gateway-->>Client: Result +``` + +**Key architectural aspects:** + +- **Session Management**: The gateway creates unified sessions by encoding multiple backend session IDs, handling reconnection with `Last-Event-ID` support for SSE streams. +- **Notification Handling**: Long-lived SSE streams from multiple MCP servers are merged into a single stream for clients, with proper event ID reconstruction. +- **Request Routing**: Tool names are automatically prefixed with the backend name (e.g., `github__issue_read`) to route calls to the correct upstream server. + +For detailed architecture and design decisions, see the [MCP Gateway proposal](https://github.com/theagentrouter/agent-router/tree/main/docs/proposals/006-mcp-gateway). + +## Trying it out + +Before you begin, you'll need to complete the basic setup from the [Basic Usage](/docs/getting-started/basic-usage) guide, which includes installing Envoy Gateway and AI Gateway. + +### Basic MCPRoute Configuration + +The following example demonstrates a basic `MCPRoute` that proxies the GitHub MCP server: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: MCPRoute +metadata: + name: mcp-route + namespace: default +spec: + parentRefs: + - name: aigw-run + kind: Gateway + group: gateway.networking.k8s.io + path: "/mcp" # Clients connect to http://gateway-address/mcp + backendRefs: + - name: github + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp/x/issues/readonly" + securityPolicy: + apiKey: + secretRef: + name: github-token +--- +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: Backend +metadata: + name: github + namespace: default +spec: + endpoints: + - fqdn: + hostname: api.githubcopilot.com + port: 443 +--- +apiVersion: v1 +kind: Secret +metadata: + name: github-token + namespace: default +type: Opaque +stringData: + apiKey: ghp_your_token_here +``` + +Apply this configuration: + +```shell +kubectl apply -f mcp-route.yaml +``` + +Now clients can connect to `http:///mcp` and access GitHub tools. + +### Tool Filtering + +Control which tools are exposed using the `toolSelector` field. You can use exact matches or regular expressions: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: MCPRoute +metadata: + name: mcp-route + namespace: default +spec: + parentRefs: + - name: aigw-run + kind: Gateway + group: gateway.networking.k8s.io + backendRefs: + # GitHub: only expose issue-related tools + - name: github + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp/x/issues/readonly" + toolSelector: + includeRegex: + - .*issues?.* # Matches issue_read, list_issues, etc. + securityPolicy: + apiKey: + secretRef: + name: github-token + + # Context7: expose specific tools by exact name + - name: context7 + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp" + toolSelector: + include: + - resolve-library-id + - query-docs +``` + +:::note +The `toolSelector` field requires exactly one of `include` or `includeRegex` to be specified. If not specified, all tools from the MCP server are exposed. +::: + +### Server Multiplexing + +The gateway automatically aggregates tools from multiple MCP servers into a single unified interface: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: MCPRoute +metadata: + name: mcp-unified + namespace: default +spec: + parentRefs: + - name: aigw-run + kind: Gateway + group: gateway.networking.k8s.io + path: "/mcp" + backendRefs: + - name: github + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp/x/issues/readonly" + securityPolicy: + apiKey: + secretRef: + name: github-token + - name: context7 + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp" +``` + +Clients will see all tools with prefixed names: + +- `github__issue_read` +- `github__list_issues` +- `context7__resolve-library-id` +- `context7__query-docs` + +### Header Forwarding + +Forward HTTP headers from the client request to specific backend MCP servers. This enables per-user authentication passthrough (e.g., personal access tokens) without requiring OAuth: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: MCPRoute +metadata: + name: mcp-unified + namespace: default +spec: + parentRefs: + - name: aigw-run + kind: Gateway + group: gateway.networking.k8s.io + path: "/mcp" + backendRefs: + - name: atlassian + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp" + forwardHeaders: + - name: X-Atlassian-Jira-Personal-Token + - name: X-Atlassian-Jira-Url + - name: Authorization + backendHeader: X-Original-Auth # optional: rename the header + - name: github + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp" + securityPolicy: + apiKey: + secretRef: + name: github-token +``` + +Each `forwardHeaders` entry specifies: + +- `name` (required): The header to extract from the incoming client request. +- `backendHeader` (optional): A different header name to use when forwarding to the backend. If omitted, the original header name is used. + +Header **values** are forwarded verbatim. The gateway does not add an auth scheme such as `Bearer `. + +Headers are scoped per-backend — during fan-out operations like `tools/list`, only the backends with explicit `forwardHeaders` configuration receive the forwarded headers. Other backends in the same route are unaffected. + +To keep a default least-privilege service-account token on a backend while letting callers override it with a personal access token, set `securityPolicy.apiKey.injectionPolicy: IfNotPresent` and map the client token onto the same header with `forwardHeaders`. When the client omits that header, the gateway injects the configured API key (and prefixes it with `Bearer ` when the target is `Authorization`). When the client sends it, the forwarded value is preserved as-is. The default is `injectionPolicy: Always`, which always injects the configured credential. + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: MCPRoute +metadata: + name: mcp-unified + namespace: default +spec: + parentRefs: + - name: aigw-run + kind: Gateway + group: gateway.networking.k8s.io + backendRefs: + - name: github + kind: Backend + group: gateway.envoyproxy.io + securityPolicy: + apiKey: + secretRef: + name: github-sa-token # default least-privilege service account + injectionPolicy: IfNotPresent + forwardHeaders: + - name: X-GitHub-PAT + backendHeader: Authorization +``` + +For that example, send the full header value the backend expects, including the scheme: + +``` +X-GitHub-PAT: Bearer ghp_... +``` + +A raw token (`X-GitHub-PAT: ghp_...`) is copied onto `Authorization` unchanged, so the backend typically rejects it. Omit `X-GitHub-PAT` to use the injected service-account key instead. + +`injectionPolicy` applies only to header injection. Do not combine `injectionPolicy: IfNotPresent` with `queryParam`. If the MCPRoute itself uses OAuth or API-key client authentication, do not forward inbound `Authorization` (that is the gateway token). Use a dedicated client header and `backendHeader` to map it onto the backend credential header. + +### OAuth Authentication + +Protect your MCP Gateway with OAuth authentication following the [MCP Authorization specification](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization): + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: MCPRoute +metadata: + name: mcp-route + namespace: default +spec: + parentRefs: + - name: aigw-run + kind: Gateway + group: gateway.networking.k8s.io + backendRefs: + - name: github + kind: Backend + group: gateway.envoyproxy.io + path: "/mcp/readonly" + securityPolicy: + oauth: + issuer: "https://keycloak.example.com/realms/master" + audiences: + - "https://api.example.com/mcp" + protectedResourceMetadata: + resource: "https://api.example.com/mcp" + scopesSupported: + - "profile" + - "email" +``` + +#### Authorization servers with non-standard metadata locations + +By default the gateway discovers the authorization server's endpoints by probing the well-known +URIs derived from `issuer`, as described in [RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414#section-3). +Some authorization servers publish their metadata somewhere the issuer does not lead to, for example +at a versioned path. Point `authorizationServerMetadataUrl` at the document in that case: + +```yaml +securityPolicy: + oauth: + issuer: "https://example.com/api/idp/authn" + authorizationServerMetadataUrl: "https://example.com/api/idp/v4/authn/.well-known/openid-configuration" +``` + +`issuer` is unaffected by this field: it still identifies the authorization server in the protected +resource metadata the gateway publishes. When `jwks` is not set, the JWKS URI is also discovered from +the document fetched here. If that document cannot be fetched, the MCPRoute is not accepted and the +reason appears in its status conditions. + +The OAuth flow follows the MCP specification's authorization code flow with PKCE: + +```mermaid +sequenceDiagram + participant C as MCP Client + participant G as MCP Gateway + participant A as Authorization Server + + C->>G: MCP request without token + G->>C: HTTP 401 with WWW-Authenticate header + + C->>G: Request Protected Resource Metadata + G->>C: Return OAuth metadata + + C->>A: GET /.well-known/oauth-authorization-server + A->>C: Authorization server metadata + + Note over C: Generate PKCE parameters + C->>A: Authorization request + PKCE + Note over A: User authorizes + A->>C: Authorization code + + C->>A: Token request + code_verifier + A->>C: Access token + + C->>G: MCP request with access token + G->>G: Verify token (JWKS) + Note over G: Token validated + G->>C: MCP response +``` + +### Authorization Policies + +Agent Router supports fine-grained access control over tool access using a combination of: + +- **JWT Scopes & Claims**: Validate standard OAuth2 scopes and custom claims +- **Tool Selection**: Restrict access to specific tools +- **CEL Expressions**: Flexible, advanced matching using Common Expression Language (CEL) + +#### Configuration Structure + +Authorization is configured in the `MCPRoute` resource under `spec.securityPolicy.authorization`. + +```yaml +spec: + securityPolicy: + authorization: + rules: + - source: + jwt: + scopes: ["read"] + target: + tools: + - backend: "github" + tool: "list_issues" +``` + +#### Rule Evaluation + +Rules are evaluated in order. The first rule that matches the request (Target, Source, and CEL) determines the action (Allow/Deny). If no rules match, the `defaultAction` is applied. + +#### Matchers + +| Matcher | Description | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Target** | Matches specific tools. Can filter by `backend` and `tool` name. | +| **Source** | Matches JWT properties.
`scopes`: List of required scopes (all must be present).
`claims`: Key-value pairs. Arrays in claims match if _any_ value matches. | +| **CEL** | Advanced expression evaluated against the request context. | + +#### CEL Context + +The following variables are available in CEL expressions: + +| Variable | Description | +| --------------------- | ---------------------------------------------- | +| `request.method` | HTTP method (e.g., "POST") | +| `request.host` | Host header value | +| `request.path` | URL path | +| `request.headers` | Map of headers (lowercased keys, single value) | +| `request.auth.jwt` | Parsed JWT `{claims: ..., scopes: [...]}` | +| `request.mcp.method` | MCP JSON-RPC method (e.g., "tools/call") | +| `request.mcp.backend` | Target backend name | +| `request.mcp.tool` | Target tool name (for tool calls) | +| `request.mcp.params` | Parsed JSON-RPC parameters | + +#### Examples + +**Comprehensive Policy Example** + +This example demonstrates various matching strategies including token scopes, claims, tool targeting, and CEL expressions. + +```yaml +authorization: + rules: + - source: + jwt: + scopes: + - echo + target: + tools: + - backend: mcp-backend + tool: echo + cel: request.mcp.params.arguments.text.matches("^Hello, .*!$") && request.headers["x-tenant-id"] == "t-123" + - source: + jwt: + scopes: + - sum + claims: + - name: tenant + valueType: String + values: + - acme + - globex + - name: org.departments + valueType: StringArray + values: + - engineering + - development + target: + tools: + - backend: mcp-backend + tool: sum +``` + +## See Also + +- [MCP Gateway Proposal](https://github.com/theagentrouter/agent-router/tree/main/docs/proposals/006-mcp-gateway) - Detailed architecture and design decisions +- [MCP Specification](https://modelcontextprotocol.io/specification/2025-06-18) - Official Model Context Protocol documentation +- [MCP Example](https://github.com/theagentrouter/agent-router/tree/main/examples/mcp) - Complete working example +- [CLI MCP Configuration](/docs/cli/aigwrun#mcp-configuration) - Using MCP with `aigw run` standalone mode diff --git a/site/versioned_docs/version-1.2/capabilities/observability/accesslogs.md b/site/versioned_docs/version-1.2/capabilities/observability/accesslogs.md new file mode 100644 index 0000000000..75d6f1bd13 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/observability/accesslogs.md @@ -0,0 +1,275 @@ +--- +id: accesslogs +title: Access Logs With AI/LLM/MCP Metadata +sidebar_position: 8 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +It is possible to configure Envoy Access Logs to include metadata generated by the AI Gateway such as +the selected model and token consumption. This guide walks through the steps to configure the AI Gateway +Routes and MCP Routes to generate LLM token consumption and MCP metadata, and to configure the Envoy Access +Logs to include the AI/LLM/MCP details. + +## LLM Metadata in Access Logs + +Agent Router populates information in the filter dynamic metadata under the `io.envoy.ai_gateway` namespace. +This metadata includes information about the selected model, prompt and completion token usage, and other +details about the LLM request, and can be extracted and included in the Envoy Access Logs. + +### AIGatewayRoute configuration + +The contents of the dynamic metadata are configured in the `AIGatewayRoute` resource under the `llmRequestCosts` +field. The `llmRequestCosts` field configures a list of metadata keys to populate in the Envoy filter metadata, to +be later accessed in the access logs. For example: + +```yaml +apiVersion: ai-gateway.envoyproxy.io/v1alpha1 +kind: AIGatewayRoute +metadata: + name: ai-gateway-route +spec: + (...) + llmRequestCosts: + - metadataKey: llm_input_token + type: InputToken + - metadataKey: llm_output_token + type: OutputToken + - metadataKey: llm_total_token + type: TotalToken + rules: + (...) +``` + +### EnvoyProxy Configuration + +Once the `AIGatewayRoute` is configured to populate the dynamic metadata, the Envoy Access Logs can be configured +to include the metadata in the log entries. This can be done by configuring the access logs in the `EnvoyProxy` +resource: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: EnvoyProxy +metadata: + name: ai-access-logs + namespace: default +spec: + telemetry: + accessLog: + settings: + - sinks: + - type: File + file: + path: /dev/stdout + format: + type: JSON + json: + # AI specific fields. The properties in the dynamic metadata expressions must match the ones + # defined in the AIGatewayRoute llmRequestCosts field. + gen_ai.request.model: "%REQ(X-AI-EG-MODEL)%" + gen_ai.response.model: "%DYNAMIC_METADATA(io.envoy.ai_gateway:response_model)%" + gen_ai.request.model_override: "%DYNAMIC_METADATA(io.envoy.ai_gateway:model_name_override)%" + gen_ai.provider.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:backend_name)%" + gen_ai.usage.total_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_total_token)%" + gen_ai.usage.input_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_input_token)%" + gen_ai.usage.output_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_output_token)%" + # Common fields + start_time: "%START_TIME%" + method: "%REQ(:METHOD)%" + x-envoy-origin-path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" + protocol: "%PROTOCOL%" + response_code: "%RESPONSE_CODE%" + response_flags: "%RESPONSE_FLAGS%" + response_code_details: "%RESPONSE_CODE_DETAILS%" + connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%" + upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%" + bytes_received: "%BYTES_RECEIVED%" + bytes_sent: "%BYTES_SENT%" + duration: "%DURATION%" + x-envoy-upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%" + x-forwarded-for: "%REQ(X-FORWARDED-FOR)%" + user-agent: "%REQ(USER-AGENT)%" + x-request-id: "%REQ(X-REQUEST-ID)%" + ":authority": "%REQ(:AUTHORITY)%" + upstream_host: "%UPSTREAM_HOST%" + upstream_cluster: "%UPSTREAM_CLUSTER%" + upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%" + downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%" + downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%" + requested_server_name: "%REQUESTED_SERVER_NAME%" + route_name: "%ROUTE_NAME%" +``` + +In this example we're adding the `gen_ai.*` properties with the values extracted from the filter metadata populated +by the AI Gateway. The `EnvoyProxy` resource must be attached to the `Gateway` object or to the `GatewayClass`. For +example: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: envoy-ai-gateway-basic + namespace: default +spec: + gatewayClassName: envoy-ai-gateway-basic + listeners: + - name: http + protocol: HTTP + port: 80 + infrastructure: + parametersRef: + group: gateway.envoyproxy.io + kind: EnvoyProxy + name: ai-access-logs +``` + +With this configuration, the access log entries will include the AI Gateway metadata and look like this: + +```json +{ + ":authority": "api.router.tetrate.ai", + "bytes_received": 105, + "bytes_sent": 432, + "connection_termination_details": null, + "downstream_local_address": "127.0.0.1:1975", + "downstream_remote_address": "127.0.0.1:64484", + "duration": 1526, + "gen_ai.provider.name": "default/tars/route/aigw-run/rule/0/ref/0", + "gen_ai.request.model": "gpt-4o-mini", + "gen_ai.request.model_override": null, + "gen_ai.usage.input_tokens": 15, + "gen_ai.usage.output_tokens": 7, + "gen_ai.usage.total_tokens": 22, + "method": "POST", + "protocol": "HTTP/1.1", + "requested_server_name": null, + "response_code": 200, + "response_code_details": "via_upstream", + "response_flags": "-", + "route_name": "httproute/default/aigw-run/rule/0/match/0/*", + "start_time": "2025-08-30T16:51:37.894Z", + "upstream_cluster": "httproute/default/aigw-run/rule/0", + "upstream_host": "34.128.142.77:443", + "upstream_local_address": "192.168.1.38:64485", + "upstream_transport_failure_reason": null, + "user-agent": "curl/8.7.1", + "x-envoy-origin-path": "/v1/chat/completions", + "x-envoy-upstream-service-time": "451", + "x-forwarded-for": "192.168.1.38", + "x-request-id": "83b63da3-45a9-4196-98b8-1a7f697d6d8d" +} +``` + +### Trying it out + +You can deploy the example to quickly try the access log configuration against a local backend: + + +{`kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/access-log/basic.yaml`} + + +Once everything is applied you can send requests to the gateway and see the access logs in the gateway pod logs. + +## MCP Metadata in Access Logs + +Agent Router automatically populates MCP information in the filter dynamic metadata under the `io.envoy.ai_gateway` namespace. +This metadata includes information about the request, MCP session, MCP method call, etc, and can be extracted and +included in the Envoy Access Logs. + +### EnvoyProxy Configuration + +The Envoy Access Logs can be configured to include the MCP metadata in the log entries as shown in the following `EnvoyProxy` +resource: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: EnvoyProxy +metadata: + name: ai-access-logs + namespace: default +spec: + telemetry: + accessLog: + settings: + - sinks: + - type: File + file: + path: /dev/stdout + format: + type: JSON + json: + # MCP specific fields + jsonrpc.request.id: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_request_id)%" + mcp.session.id: "%REQ(MCP-SESSION-ID)%" + mcp.method.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_method)%" + mcp.tool.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_tool_name)%" + mcp.provider.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_backend)%" + # Common fields + start_time: "%START_TIME%" + method: "%REQ(:METHOD)%" + x-envoy-origin-path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" + protocol: "%PROTOCOL%" + response_code: "%RESPONSE_CODE%" + response_flags: "%RESPONSE_FLAGS%" + response_code_details: "%RESPONSE_CODE_DETAILS%" + connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%" + upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%" + bytes_received: "%BYTES_RECEIVED%" + bytes_sent: "%BYTES_SENT%" + duration: "%DURATION%" + x-envoy-upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%" + x-forwarded-for: "%REQ(X-FORWARDED-FOR)%" + user-agent: "%REQ(USER-AGENT)%" + x-request-id: "%REQ(X-REQUEST-ID)%" + ":authority": "%REQ(:AUTHORITY)%" + upstream_host: "%UPSTREAM_HOST%" + upstream_cluster: "%UPSTREAM_CLUSTER%" + upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%" + downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%" + downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%" + requested_server_name: "%REQUESTED_SERVER_NAME%" + route_name: "%ROUTE_NAME%" +``` + +In this example we're adding the `mcp_*` properties with the values extracted from the filter metadata populated +by the AI Gateway. The `EnvoyProxy` resource must be attached to the `Gateway` object or to the `GatewayClass` as +explained in the previous section. + +With this configuration, the access log entries will include the AI Gateway metadata and look like this: + +```json +{ + "bytes_received": 165, + "bytes_sent": 10255, + "connection_termination_details": null, + "downstream_local_address": "127.0.0.1:10088", + "downstream_remote_address": "127.0.0.1:58124", + "duration": 653, + "gen_ai.provider.name": null, + "gen_ai.request.model": null, + "gen_ai.request.model_override": null, + "gen_ai.usage.input_tokens": null, + "gen_ai.usage.output_tokens": null, + "mcp.provider.name": "context7", + "mcp.method.name": "tools/call", + "jsonrpc.request.id": "3", + "mcp.session.id": null, + "method": "POST", + "response_code": 200, + "start_time": "2025-10-08T10:42:11.430Z", + "upstream_cluster": "httproute/default/ai-eg-mcp-br-mcp-route-context7/rule/0", + "upstream_host": "13.217.64.66:443", + "upstream_local_address": "192.168.40.127:58172", + "upstream_transport_failure_reason": null, + "user-agent": "Go-http-client/1.1", + "x-envoy-origin-path": "/mcp", + "x-request-id": "cf2c0454-6290-498e-af6c-e32a611b5da1" +} +``` + +### Trying it out + +You can deploy the example to quickly try the access log configuration by following the +[MCP examples](https://github.com/theagentrouter/agent-router/tree/main/examples/mcp) and use any of the +provided configurations. diff --git a/site/versioned_docs/version-1.2/capabilities/observability/index.md b/site/versioned_docs/version-1.2/capabilities/observability/index.md new file mode 100644 index 0000000000..46d01969f7 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/observability/index.md @@ -0,0 +1,16 @@ +--- +id: observability +title: Observability +--- + +Agent Router extends the capabilities of Envoy Gateway, and as you run Envoy Gateway you have access to the foundational observability in the Envoy Gateway system. +We recommend you familiarize yourself with the [Envoy Gateway Observability Documentation](https://gateway.envoyproxy.io/docs/tasks/observability/). + +## AI/LLM Observability Features + +The Agent Router provides specialized observability capabilities for AI and LLM workloads: + +- **[GenAI Metrics](./metrics.md)** - Prometheus metrics following OpenTelemetry Gen AI semantic conventions for monitoring token usage, latency, and model performance. +- **[GenAI Tracing](./tracing.md)** - OpenTelemetry integration with OpenInference semantic conventions for LLM request tracing and evaluation. +- **[Access Logs with AI/LLM metadata](./accesslogs.md)** - AI metadata produced by the AI gateway (model name, token usage, etc.) can be included in the Envoy Access Logs. +- **[Gateway Configuration](../gateway-config.md)** - Per-gateway configuration of the external processor container, including environment variables for tracing and resource requirements. diff --git a/site/versioned_docs/version-1.2/capabilities/observability/metrics.md b/site/versioned_docs/version-1.2/capabilities/observability/metrics.md new file mode 100644 index 0000000000..0117e54c4a --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/observability/metrics.md @@ -0,0 +1,124 @@ +--- +id: metrics +title: AI/LLM Metrics +sidebar_position: 6 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +When using the Agent Router, it will collect AI specific metrics and expose them to Prometheus for monitoring by default. +This guide provides an overview of the metrics collected by the AI Gateway and how to monitor them using Prometheus. + +## Overview + +Agent Router is designed to intercept and process AI/LLM requests, that enables it to collect metrics for monitoring and observability. +Currently, it collects metrics and exports them to Prometheus for monitoring in the OpenTelemetry format as specified by the [OpenTelemetry Gen AI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/). +Not all metrics are supported yet, but the Agent Router will continue to add more metrics in the future. + +### Supported Endpoints + +Metrics are collected for the following LLM endpoints: + +- **`/v1/chat/completions`** - Chat completions (streaming and non-streaming) +- **`/v1/completions`** - Legacy text completions (streaming and non-streaming) +- **`/v1/embeddings`** - Text embeddings +- **`/cohere/v2/rerank`** - Rerank +- **`/typesafe/v1/systemone`** - TypeSafe System One (Jev) +- **`/anthropic/v1/messages`** - Anthropic messages (streaming and non-streaming) + +For example, the Agent Router collects metrics such as: + +- [**`gen_ai.client.token.usage`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiclienttokenusage): Number of tokens processed. The attribute `gen_ai.token.type` can be used to differentiate between input, output, and total tokens. +- [**`gen_ai.server.request.duration`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiserverrequestduration): Measured from the start of the received request headers in the Agent Router filter to the end of the processed response body processing. +- [**`gen_ai.server.time_to_first_token`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiservertime_to_first_token): Measured from the start of the received request headers in the Agent Router filter to the receiving of the first token in the response body handling. +- [**`gen_ai.server.time_per_output_token`**](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/#metric-gen_aiservertime_per_output_token): The latency between consecutive tokens, if supported, or by chunks/tokens otherwise. + +Each metric comes with some default attributes such as: + +- `gen_ai.operation.name` + - `chat`: For `/v1/chat/completions` endpoint. + - `completion`: For `/v1/completions` endpoint. + - `embedding`: For `/v1/embeddings` endpoint. + - `rerank`: For `/cohere/v2/rerank` endpoint. + - `systemone`: For `/typesafe/v1/systemone` endpoint. + - `image_generation`: For `/v1/images/generations` endpoint. + - `messages`: For `/anthropic/v1/messages` endpoint. +- `gen_ai.original.model` - The original model name from the request body +- `gen_ai.request.model` - The model name requested (may be overridden) +- `gen_ai.response.model` - The model name returned in the response +- `gen_ai.provider.name` - The provider name (e.g., `openai`, `anthropic`) +- `gen_ai.backend` - The `AIServiceBackend` that served the request, as `namespace/name` + +:::tip + +You can enrich the metrics with custom labels extracted from HTTP request headers. Use `controller.requestHeaderAttributes` for a base mapping shared with spans and access logs, and `controller.metricsRequestHeaderAttributes` for metrics-only mappings. Metrics never default to `session.id` because it is high-cardinality. See [values.yaml](https://github.com/theagentrouter/agent-router/blob/main/manifests/charts/ai-gateway-helm/values.yaml) for more details including other configurations. + +::: + +## Trying it out + +Before you begin, you'll need to complete the basic setup from the [Basic Usage](/docs/getting-started/basic-usage) guide. + +Then, you can install the prometheus using the following commands: + + +{`kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/monitoring/monitoring.yaml`} + + +Let's wait for a while until the Prometheus is up and running. + +```shell +kubectl wait --for=condition=ready pod -l app=prometheus -n monitoring +``` + +To access the Prometheus dashboard, you need to port-forward the Prometheus service to your local machine like this: + +```shell +kubectl port-forward -n monitoring svc/prometheus 9090:9090 +``` + +Now open your browser and navigate to `http://localhost:9090` to access the Prometheus dashboard to explore the metrics. + +Alternatively, you can make the following requests to see the raw metrics: + +```shell +curl http://localhost:9090/api/v1/query --data-urlencode \ + 'query=sum(gen_ai_client_token_usage_sum{gateway_envoyproxy_io_owning_gateway_name = "envoy-ai-gateway-basic"}) by (gen_ai_request_model, gen_ai_token_type)' \ + | jq '.data.result[]' +``` + +and then you would get the response like this, assuming you have made some requests with the model `gpt-4o-mini`: + +```json lines +{ + "metric": { + "gen_ai_request_model": "gpt-4o-mini", + "gen_ai_token_type": "input" + }, + "value": [ + 1743105857.684, + "12" + ] +} +{ + "metric": { + "gen_ai_request_model": "gpt-4o-mini", + "gen_ai_token_type": "output" + }, + "value": [ + 1743105857.684, + "13" + ] +} +{ + "metric": { + "gen_ai_request_model": "gpt-4o-mini", + "gen_ai_token_type": "total" + }, + "value": [ + 1743105857.684, + "25" + ] +} +``` diff --git a/site/versioned_docs/version-1.2/capabilities/observability/tracing.md b/site/versioned_docs/version-1.2/capabilities/observability/tracing.md new file mode 100644 index 0000000000..f62c46a3d9 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/observability/tracing.md @@ -0,0 +1,366 @@ +--- +id: tracing +title: GenAI Distributed Tracing +sidebar_position: 7 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +Agent Router's router joins and records distributed traces when supplied +with an [OpenTelemetry](https://opentelemetry.io/) collector endpoint. + +This guide provides an overview of the spans recorded by the AI Gateway and how +export them to your choice of OpenTelemetry collector. + +## Overview + +Agent Router's router joins and records distributed traces when supplied +with an [OpenTelemetry](https://opentelemetry.io/) collector endpoint. + +Requests to the OpenAI Chat Completions, Completions (legacy), and Embeddings +endpoints are recorded as Spans which include typical timing and request +details. In addition, there are GenAI attributes representing the LLM or +Embeddings call including full request and response details, defined by +[OpenInference semantic conventions][openinference]. + +OpenInference attributes default to include full request and response data for +both chat completions and embeddings. This can be toggled with configuration, +but when enabled allows systems like [Arize Phoenix][phoenix] to perform +evaluations of production requests captured in OpenTelemetry spans. + +For chat completions, this includes traditional LLM metrics such as correctness +and hallucination detection. For embeddings, it enables agentic RAG evaluations +focused on retrieval and semantic analysis. + +## Trying it out + +Before you begin, you'll need to complete the basic setup from the +[Basic Usage](/docs/getting-started/basic-usage) guide, which includes +installing Envoy Gateway and AI Gateway. + +### Install Phoenix for LLM observability + +```shell +# Install Phoenix using PostgreSQL storage. +helm install phoenix oci://registry-1.docker.io/arizephoenix/phoenix-helm \ + --namespace envoy-ai-gateway-system \ + --set auth.enableAuth=false \ + --set server.port=6006 + +# Wait for Phoenix to be ready (first run may take a few minutes to pull images) +kubectl wait --timeout=5m -n envoy-ai-gateway-system \ + pods -l app=phoenix --for=condition=Ready +``` + +### Configure AI Gateway with OpenTelemetry + +Upgrade your AI Gateway installation with [OpenTelemetry configuration][otel-config]: + + +{`helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \\ + --version v${vars.aigwVersion} \\ + --namespace envoy-ai-gateway-system \\ + --set "extProc.extraEnvVars[0].name=OTEL_EXPORTER_OTLP_ENDPOINT" \\ + --set "extProc.extraEnvVars[0].value=http://phoenix-svc.envoy-ai-gateway-system:6006" \\ + --set "extProc.extraEnvVars[1].name=OTEL_METRICS_EXPORTER" \\ + --set "extProc.extraEnvVars[1].value=none" +# OTEL_SERVICE_NAME defaults to "ai-gateway" if not set +# OTEL_METRICS_EXPORTER=none because Phoenix only supports traces, not metrics +# Note: Use fully-qualified service name because ext-proc runs in envoy-gateway-system namespace`} + + +Wait for the gateway pod to be ready: + +```shell +kubectl wait --for=condition=Ready -n envoy-gateway-system \ + pods -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic +``` + +### Generate traces + +Make requests to your AI Gateway to generate traces. Follow the instructions +from [Testing the Gateway](/docs/getting-started/basic-usage#testing-the-gateway) +in the Basic Usage guide to: + +1. Set up port forwarding (if needed) +2. Test the chat completions endpoint + +Each request will generate traces that are sent to Phoenix. + +### Check Phoenix is receiving traces + +```bash +kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces" +``` + +### Access Phoenix UI + +Port-forward to access the Phoenix dashboard: + +```shell +kubectl port-forward -n envoy-ai-gateway-system svc/phoenix-svc 6006:6006 +``` + +Then open http://localhost:6006 in your browser to explore the traces. + +## Semantic Conventions + +Spans are recorded using [OpenInference semantic conventions][openinference] by +default. You can instead emit the +[OpenTelemetry GenAI semantic conventions][otel-genai], which use `gen_ai.*` +attributes and are consumed by OpenTelemetry-native backends: + +```yaml +extProc: + extraEnvVars: + - name: AI_GATEWAY_TRACING_SEMCONV + value: "gen_ai" +``` + +| Value | Behavior | +| ------------------------ | ---------------------------------- | +| unset or `openinference` | OpenInference attributes (default) | +| `gen_ai` | OpenTelemetry GenAI attributes | + +Any other value fails startup rather than silently falling back, so a typo is +reported immediately instead of producing traces nobody is watching. + +Only one convention is emitted at a time. Choosing `gen_ai` changes the shape of +your spans, so update dashboards and alerts before switching: + +| | OpenInference | OpenTelemetry GenAI | +| --------- | ------------------------ | ----------------------------------------------- | +| Span name | `ChatCompletion` | `chat {model}` | +| Span kind | `INTERNAL` | `CLIENT` | +| Model | `llm.model_name` | `gen_ai.request.model`, `gen_ai.response.model` | +| Provider | `llm.system` | `gen_ai.provider.name` | +| Tokens | `llm.token_count.prompt` | `gen_ai.usage.input_tokens` | +| Messages | `llm.input_messages.N.*` | `gen_ai.input.messages` (single JSON value) | +| Errors | `exception` event | `error.type` attribute | + +Note that the GenAI conventions are still marked Development upstream, so +attribute names may change in future releases. + +Since span names include the model, they are higher cardinality than +OpenInference's fixed names. This matters for backends that index on span name. + +### Capturing message content with GenAI + +Unlike OpenInference, which records request and response content by default, the +GenAI conventions treat message content as opt-in because it routinely contains +sensitive data. Enable it explicitly: + +```yaml +extProc: + extraEnvVars: + - name: AI_GATEWAY_TRACING_SEMCONV + value: "gen_ai" + - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT + value: "true" +``` + +Token counts and sampling parameters are recorded either way, since the +conventions treat those as metadata rather than content. + +Message content is currently mapped for chat completions and Anthropic +messages. Other endpoints record operation, model, usage and sampling +parameters. Several — image generation, speech, transcription, translation and +rerank and TypeSafe System One — have no content attributes defined by the conventions at all. + +The `OPENINFERENCE_HIDE_*` variables described below apply only to the +OpenInference convention. They have no effect when `gen_ai` is selected. + +### MCP spans + +`AI_GATEWAY_TRACING_SEMCONV` selects the vocabulary for MCP spans as well as for +the LLM endpoints. OpenInference defines no MCP conventions, so the default keeps +the gateway-specific attributes MCP spans have always used; `gen_ai` opts into +the [OpenTelemetry MCP semantic conventions][otel-mcp]. + +| | default (`openinference`) | `gen_ai` | +| ------------- | ------------------------- | ---------------------------------------------- | +| Span name | `CallTool`, `ListTools` | `tools/call {tool}`, `tools/list` | +| Tool name | `mcp.tool.name` | `gen_ai.tool.name` + `gen_ai.operation.name` | +| Prompt name | `mcp.prompt.name` | `gen_ai.prompt.name` | +| Request ID | `mcp.request.id` | `jsonrpc.request.id` | +| Transport | `mcp.transport` | `network.transport`, `network.protocol.*` | +| Errors | `exception` event | `error.type`, `rpc.response.status_code` | +| Session | on the per-backend event | also `mcp.session.id` on the span | +| List sizes | not recorded | `mcp.tools.count`, `mcp.resources.count`, ... | +| Tool call I/O | not recorded | `gen_ai.tool.call.arguments`/`.result`, opt-in | + +Tool call arguments and results are message content, so under `gen_ai` they +follow the same `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` opt-in as +the LLM endpoints. The default convention never records them. + +:::note Deprecation +The gateway-specific MCP attributes are deprecated in favor of the OpenTelemetry +MCP conventions. They remain the default for now; a future release will announce +a version in which the default flips. Migrate by setting +`AI_GATEWAY_TRACING_SEMCONV=gen_ai` once your dashboards query the new names. +::: + +## Privacy Configuration + +:::note +This section applies to the default OpenInference convention. See +[Capturing message content with GenAI](#capturing-message-content-with-genai) +for the `gen_ai` equivalent. +::: + +Control sensitive data in traces by adding +[OpenInference configuration][openinference-config] to Helm values when you +reconfigure the AI Gateway. There is [similar config][openinference-embeddings] +for embeddings: + +For example, if you are using a `values.yaml` file instead of command line +arguments, you can add the following to control redaction: + +```yaml +extProc: + extraEnvVars: + # Base OTEL configuration... + # Hide sensitive data (all default to false) + - name: OPENINFERENCE_HIDE_INPUTS + value: "true" # Hide input messages to the LLM + - name: OPENINFERENCE_HIDE_OUTPUTS + value: "true" # Hide output messages from the LLM + # Reduce volume for embeddings (all default to false) + - name: OPENINFERENCE_HIDE_EMBEDDINGS_TEXT + value: "true" # Hide embeddings input + - name: OPENINFERENCE_HIDE_EMBEDDINGS_VECTORS + value: "true" # Hide embeddings output +``` + +Note: Hiding inputs/outputs prevents human or LLM-as-a-Judge evaluation of your +LLM requests, such as done with the [Phoenix Evals library][phoenix-evals]. + +## Session Tracking + +Sessions help track and organize related traces across multi-turn conversations +with your AI app. Maintaining context between interactions is key for +observability. + +With sessions, you can: + +- Track a conversation's full history in one thread. +- View inputs/outputs for a given agent. +- Monitor token usage and latency per conversation. + +By tagging spans with a consistent session ID, you get a connected view of +performance across a user's journey. + +The challenge is that requests to the gateway may not send traces, making +grouping difficult. Many GenAI frameworks allow you to set custom HTTP headers +when sending traffic to an LLM. Propagating sessions this way is simpler than +instrumenting applications with tracing code and can still achieve grouping. + +There's no standard name for session ID headers, but there is a common attribute +in OpenTelemetry, [session.id][otel-session], which has special handling in some +OpenTelemetry platforms such as [Phoenix][phoenix-session]. + +To bridge this gap, Agent Router lets you map HTTP request headers to +OpenTelemetry attributes. You can define a base mapping shared by metrics, +spans, and access logs, plus optional per-signal mappings for metrics, spans, +and access logs. + +- `controller.requestHeaderAttributes` +- `controller.spanRequestHeaderAttributes` +- `controller.metricsRequestHeaderAttributes` +- `controller.logRequestHeaderAttributes` + +`controller.spanRequestHeaderAttributes` and `controller.logRequestHeaderAttributes` default to `agent-session-id:session.id` when unset (set them to an empty string to disable the default). Metrics never default to `session.id`. + +Both of these use the same value format: a comma-separated list of +`:` pairs. For example, if your session ID header +is `agent-session-id`, you can map it to the standard OpenTelemetry attribute +`session.id` like this: `agent-session-id:session.id`. + +Some metrics systems will be able to do fine-grained aggregation, but not all. +Here's an example of keeping the default session mapping for spans/logs while +only adding a low-cardinality attribute to metrics: + + +{`helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \\ + --version v${vars.aigwVersion} \\ + --namespace envoy-ai-gateway-system \\ + --reuse-values \\ + --set "controller.metricsRequestHeaderAttributes=x-tenant-id:tenant.id"`} + + +## Cleanup + +To remove Phoenix and disable tracing: + + +{`# Uninstall Phoenix +helm uninstall phoenix -n envoy-ai-gateway-system + +# Disable tracing in AI Gateway + +helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \\ +--version v${vars.aigwVersion} \\ +--namespace envoy-ai-gateway-system \\ +--reuse-values \\ +--unset extProc.extraEnvVars`} + + +## Per-Gateway Configuration + +For deployments with multiple Gateways that need different tracing configurations, +use the `GatewayConfig` CRD instead of global Helm values. This allows you to: + +- Configure different OTEL endpoints for different Gateways +- Set per-gateway service names for better trace organization +- Override global tracing settings for specific Gateways + +### Example + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: production-tracing + namespace: default +spec: + extProc: + kubernetes: + env: + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: "http://production-collector:4317" + - name: OTEL_SERVICE_NAME + value: "ai-gateway-production" +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: production-gateway + annotations: + aigateway.envoyproxy.io/gateway-config: production-tracing +spec: + # ... +``` + +See the [Gateway Configuration](../gateway-config.md) guide for detailed information +on `GatewayConfig` usage, including environment variable precedence and shared configurations. + +## See Also + +- [Gateway Configuration](../gateway-config.md) - Per-gateway configuration using GatewayConfig +- [OpenInference Specification][openinference] - GenAI Semantic conventions for traces +- [OpenTelemetry Configuration][otel-config] - Environment variable reference +- [Arize Phoenix Documentation][phoenix] - LLM observability platform + +--- + +[openinference]: https://github.com/Arize-ai/openinference/tree/main/spec +[otel-genai]: https://github.com/open-telemetry/semantic-conventions-genai +[otel-mcp]: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md +[openinference-config]: https://github.com/Arize-ai/openinference/blob/main/spec/configuration.md +[openinference-embeddings]: https://github.com/Arize-ai/openinference/blob/main/spec/embedding_spans.md +[otel-config]: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ +[phoenix]: https://docs.arize.com/phoenix +[phoenix-evals]: https://arize.com/docs/phoenix/evaluation/llm-evals +[otel-session]: https://opentelemetry.io/docs/specs/semconv/registry/attributes/session/ +[phoenix-session]: https://arize.com/docs/phoenix/tracing/llm-traces/sessions diff --git a/site/versioned_docs/version-1.2/capabilities/scaling.md b/site/versioned_docs/version-1.2/capabilities/scaling.md new file mode 100644 index 0000000000..efef887b0e --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/scaling.md @@ -0,0 +1,117 @@ +--- +id: scaling +title: Scaling the AI Gateway Controller +sidebar_position: 3 +--- + +# Scaling the AI Gateway Controller + +The AI Gateway controller has two independent components with different scaling +characteristics: + +1. **Kubernetes controller** — reconciles CRDs and performs mutable operations + against the Kubernetes API (requires leader election). +2. **Extension server** — a gRPC server that handles Envoy Gateway extension + calls. It is read-only with respect to the Kubernetes API and scales + horizontally. + +Because Envoy Gateway's xDS server communicates with all Envoy instances, and +each EG replica in turn calls the AIGW extension server, the extension server +can become a bottleneck under load. Running multiple AIGW controller replicas +distributes this load across pods. + +:::note +Leader election applies only to the Kubernetes controller portion. The +extension server starts and serves traffic on every replica, including +non-leader pods, so all replicas handle extension server requests from the +moment they are ready. +::: + +## Recommended Configuration + +### Helm + +Set `controller.replicaCount` and ensure `controller.leaderElection.enabled` +is `true` (the default): + +```yaml +controller: + replicaCount: 2 + leaderElection: + enabled: true +``` + +Apply with: + +```shell +helm upgrade --install ai-gateway-helm \ + oci://docker.io/envoyproxy/ai-gateway-helm \ + --namespace envoy-ai-gateway-system \ + --values values.yaml +``` + +### Kubernetes Deployment (without Helm) + +Edit the controller `Deployment` directly: + +```shell +kubectl -n envoy-ai-gateway-system scale deployment ai-gateway-controller \ + --replicas=2 +``` + +Or patch the `Deployment` manifest: + +```yaml +spec: + replicas: 2 +``` + +## Horizontal Pod Autoscaler + +For dynamic workloads you can pair the above with an HPA. The extension server +is CPU-bound, so CPU utilization is a reasonable metric: + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: ai-gateway-controller + namespace: envoy-ai-gateway-system +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: ai-gateway-controller + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 +``` + +## Resource Requests and Limits + +Size each replica according to the number of Envoy instances it will serve. +A reasonable starting point: + +```yaml +controller: + resources: + requests: + cpu: "100m" + memory: "128Mi" + limits: + cpu: "500m" + memory: "256Mi" +``` + +Adjust based on observed usage in your environment. + +## See Also + +- [Gateway Configuration](./gateway-config.md) — per-gateway ext_proc settings +- [Observability](./observability/) — metrics to monitor controller load diff --git a/site/versioned_docs/version-1.2/capabilities/security/index.md b/site/versioned_docs/version-1.2/capabilities/security/index.md new file mode 100644 index 0000000000..f2d3c1fa68 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/security/index.md @@ -0,0 +1,32 @@ +--- +id: security +title: Security +--- + +# Security + +As Agent Router is built on Envoy Gateway, you can leverage the Envoy Gateway Security Policy by attaching them to the Gateway and/or generated HTTPRoutes. + +:::tip +View all **[Envoy Gateway Security Docs](https://gateway.envoyproxy.io/docs/tasks/security/)** to learn more what security configurations are available to you. +::: + +## Common Security Docs + +Below are a list of common security configurations that can be useful when securing your gateway leveraging Envoy Gateway configurations. + +### Access Control + +- [JWT Validation](https://gateway.envoyproxy.io/docs/tasks/security/jwt-authentication/) - _Validate signed JWT tokens_ +- [JWT Claim Based Authorization](https://gateway.envoyproxy.io/docs/tasks/security/jwt-claim-authorization/) - _Assert values of claims in JWT tokens_ +- [Mutual TLS](https://gateway.envoyproxy.io/docs/tasks/security/mutual-tls/) - _Certificate based access control_ +- [Integrate with External Authorization Service](https://gateway.envoyproxy.io/docs/tasks/security/ext-auth/) - _Useful for custom logic to your business_ +- [OIDC Provider Integration](https://gateway.envoyproxy.io/docs/tasks/security/oidc/) - _When you want to have integration with a login flow for an end-user_ +- [Require Basic Auth](https://gateway.envoyproxy.io/docs/tasks/security/basic-auth/) +- [Require API Key](https://gateway.envoyproxy.io/docs/tasks/security/apikey-auth/) +- [IP Allowlist/Denylist](https://gateway.envoyproxy.io/docs/tasks/security/restrict-ip-access/) + +### TLS + +- [Setup TLS Certificate](https://gateway.envoyproxy.io/docs/tasks/security/secure-gateways/) +- [Using TLS cert-manager](https://gateway.envoyproxy.io/docs/tasks/security/tls-cert-manager/) diff --git a/site/versioned_docs/version-1.2/capabilities/security/upstream-auth.mdx b/site/versioned_docs/version-1.2/capabilities/security/upstream-auth.mdx new file mode 100644 index 0000000000..654588f5ae --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/security/upstream-auth.mdx @@ -0,0 +1,194 @@ +--- +id: upstream-auth +title: Upstream Authentication +sidebar_position: 8 +--- +# Upstream Authentication + +## Overview + +Upstream Authentication secures the connectivity between the Gateway and the LLM Providers. This layer ensures secure communication and proper access control when the Gateway interacts with various AI service providers like AWS Bedrock, Azure OpenAI, Open AI, Mistral, Gemini, and other platforms. + +## Why Upstream Authentication Matters + +Upstream Authentication is essential for several reasons: + +1. **Security Layer**: It provides a secure authentication mechanism between the Gateway and upstream providers, preventing credentials sprawl across teams and organizations. + +1. **Credential Management**: It allows AI platform teams to handle the secure storage and management of credentials and use them in one place. This makes the revocation and rotation of credentials easier. + +1. **Compliance**: It helps organizations meet security and compliance requirements by maintaining proper authentication protocols and enforcing controlled usage of AI resources. + +## Enterprise Security Architecture + +In enterprise environments, security is implemented in multiple layers: + +1. **Client to Gateway Authentication** + - Clients must authenticate to the Gateway using appropriate methods (API keys, OAuth, etc.) aligned with the organization's security standards. + - This ensures only authorized clients can access the Gateway's services, and the Gateway enforces the access to the upstream providers and models. + + :::tip + Check out [Envoy Gateway Security Documentation](https://gateway.envoyproxy.io/docs/tasks/security/) for Client to Gateway security configuration options. + ::: + +2. **Gateway to Upstream Authentication** + - The Gateway must authenticate to upstream providers. + - This layer is managed by Upstream Authentication. + - Ensures secure communication between the Gateway and AI service providers. + +
+ ![Upstream Authentication](/diagrams/upstream-auth.png) +
+ +## Credential Management + +**Where the providers support short lived access credentials**, the Agent Router control plane supports automated credential management with the providers' identity system. This ensures a short-lived proof of authorization, such as an access token, is used when the request is sent to the upstream service. + +**For providers that support long lived access credentials**, the Agent Router control plane supports a manual credential management process. In these cases the credentials, like API keys, are stored in Kubernetes secrets and managed by the administrator. + +### Automated Credential Management + +The Gateway integrates with each provider's identity system to ensure secure, short-lived authentication: + +- **AWS Bedrock**: Uses OIDC integration with AWS STS to generate temporary credentials for each request +- **Azure OpenAI**: Leverages Entra ID (formerly Azure AD) to provide short-lived access tokens +- **GCP VertexAI**: Uses GCP workload federation with Google STS to generate temporary credentials for each request + + +In both cases, the Gateway automatically manages these credentials, ensuring that each request to upstream providers is sent with short-lived credentials. This approach significantly reduces the risk of credential exposure and aligns with enterprise security best practices. + +:::info +Learn more about connecting to [AWS Bedrock](/docs/getting-started/connect-providers/aws-bedrock) and [Azure OpenAI](/docs/getting-started/connect-providers/azure-openai) in the provider specific documentation. +::: + +### Manual Credential Management + +For providers that support long lived access credentials, the Agent Router control plane supports a manual credential management process. In these cases the credentials, like API keys, are stored in Kubernetes secrets and managed by the AI Gateway administrator. Agent Router will use the credentials from the secret to authenticate with the upstream service, attaching them to each request by securely retrieving them from the secret and attaching them to the request. + +:::info +Learn more about connecting to [OpenAI](/docs/getting-started/connect-providers/openai) and adding your API key to the secret. You can use the same approach for other providers that support long lived credentials. +::: + +## Per-Request Credential Override + +By default, a `BackendSecurityPolicy` attaches one credential to every request. The optional +`credentialOverride` field instead lets a trusted filter that runs ahead of the AI Gateway +processor — typically an external authorization service wired in via `ext_authz` — supply the +upstream credential for each request individually. This is how a multi-tenant gateway can bill +every tenant against their own provider account while still sharing routes and backends. + +The per-request credential can come from one of two sources. + +### From Dynamic Metadata (recommended) + +The trusted filter writes the credential into Envoy dynamic metadata, and the policy names the +metadata namespace to read it from: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: openai-per-tenant +spec: + targetRefs: + - group: aigateway.envoyproxy.io + kind: AIServiceBackend + name: openai + type: APIKey + apiKey: + secretRef: + name: openai-fallback-key + credentialOverride: + fromDynamicMetadata: + namespace: envoy.filters.http.ext_authz + key: x-aigw-api-key +``` + +For `AWSCredentials` the metadata value must be a struct rather than a string, because SigV4 +signing needs the whole credential (`sessionToken` is optional and omitted for long-lived +credentials): + +```yaml + credentialOverride: + fromDynamicMetadata: + namespace: envoy.filters.http.ext_authz + key: x-aigw-aws-credentials # { accessKeyId, secretAccessKey, sessionToken } +``` + +Envoy only forwards dynamic metadata namespaces the filter chain is configured to forward. The +namespace the policy reads from must be declared in the +[GatewayConfig](../gateway-config.md) referenced by the Gateway, under +`extProc.metadataForwardingNamespaces`: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: my-gateway-config +spec: + extProc: + metadataForwardingNamespaces: + - envoy.filters.http.ext_authz +``` + +The declared namespaces apply to all AI traffic of the Gateways referencing the config. The list +is deliberately not derived from the policies that read it: a `BackendSecurityPolicy` is namespaced +and can be written by a backend owner, while what Envoy hands to the external processor is the +Gateway owner's decision. Envoy forwards exactly what that owner declared, so a policy naming an +undeclared namespace behaves as if the metadata were absent, and the controller logs an error +naming it. The declaration governs the Gateway that makes it, so attaching a route to a Gateway +cannot widen what a different Gateway forwards. An external authorization service produces the +metadata through the `dynamic_metadata` field of its `CheckResponse`. + +:::note +Only **untyped** dynamic metadata is read. A filter that writes the credential as typed metadata +(a `google.protobuf.Any` under the same namespace) produces no value, and the request falls back +to the static credential or is rejected according to `fallbackToConfigured`. + +A policy reads a namespace Envoy forwards into the processor. `io.envoy.ai_gateway` travels the +other way: the processor writes it and Envoy receives it, for token-based rate limiting and +access logs. +::: + +Dynamic metadata is the recommended source. Clients cannot forge it, since only filters in the +gateway's own configuration can write it. And because the credential never appears as a request +header, there is nothing to strip, nothing for access logs to capture, and nothing that can +reach another backend on retries or fallback. + +### From Request Headers + +Alternatively, the trusted filter can inject the credential as a request header, which the +gateway reads and strips before forwarding the request to the backend it authenticated: + +```yaml + credentialOverride: + fromRequestHeaders: {} # reads x-aigw-api-key for type: APIKey +``` + +Because any client can set request headers, this source is only safe when the trusted boundary +overwrites or rejects client-supplied values of the configured header. The header also travels +with the request through the filter chain, and stripping is configured per backend — so on +routes with multiple backends (for example provider fallback), prefer `fromDynamicMetadata`, +which has neither exposure. + +### Fallback Behaviour + +`fallbackToConfigured` (default `true`) controls what happens when the source carries no +credential: fall back to the static credential configured in the policy, or, when set to +`false`, reject the request with a 401 without contacting the provider. A *partial* AWS +credential — an access key ID without a secret access key, or the reverse — always fails the +request rather than falling back, since it indicates a misconfigured credential producer. + +Falling back requires something to fall back *to*: if the policy has no static credential — +for example `GCPCredentials` relying on Application Default Credentials with no credentials +file or workload identity federation — the controller rejects `credentialOverride` unless +`fallbackToConfigured: false` is set explicitly. + +When enabling `fromDynamicMetadata` on a backend already serving traffic, prefer starting with +`fallbackToConfigured: true` and tightening to `false` afterwards: the processor configuration +and Envoy's forwarding configuration are delivered on separate channels, and with `false` a +request that lands in between is rejected with a 401. + +## Conclusion + +Upstream Authentication is a key component of the Agent Router's security architecture. It ensures secure communication between the Gateway and upstream AI service providers while supporting modern authentication methods and enterprise security requirements. Leverage Agent Router's Upstream Authentication to maintain a secure and compliant AI infrastructure in your enterprise environments. diff --git a/site/versioned_docs/version-1.2/capabilities/traffic/header-body-mutations.md b/site/versioned_docs/version-1.2/capabilities/traffic/header-body-mutations.md new file mode 100644 index 0000000000..58b3f065dd --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/traffic/header-body-mutations.md @@ -0,0 +1,187 @@ +--- +id: header-body-mutations +title: Header and Body Mutations +sidebar_position: 7 +--- + +# Header and Body Mutations + +Agent Router allows you to mutate HTTP headers and JSON request body fields before requests are sent to backends. This is useful for adding provider-specific headers, setting service tiers, or removing internal fields before forwarding to upstream providers. + +## Use Cases + +- Add custom headers required by specific backends (e.g., organization IDs, API keys). +- Set JSON body fields like `service_tier`, `max_tokens`, or `temperature` per backend. +- Remove internal or sensitive fields before forwarding requests to upstream providers. +- Override default values on a per-backend or per-route basis. + +## Where Mutations Can Be Configured + +Mutations can be configured at two levels: + +- **AIServiceBackend** — Mutations defined here apply to **all requests** routed to this backend, regardless of which route matched. +- **AIGatewayRoute backendRef** — Mutations defined here apply **only when traffic matches a specific route rule** and is sent to the referenced backend. + +:::note Precedence +When both route-level and backend-level mutations are defined, **route-level takes precedence** over backend-level for conflicting operations. Non-conflicting operations from both levels are applied together. +::: + +## Header Mutations + +Use `headerMutation` to add, overwrite, or remove HTTP headers on the outgoing request. + +### Setting Headers + +The `set` field overwrites an existing header or adds it if not present. A maximum of 16 entries is allowed. + +```yaml +headerMutation: + set: + - name: "x-custom-header" + value: "my-value" +``` + +For example, given an incoming request with `my-header: foo`, the following configuration changes it to `my-header: bar`: + +```yaml +headerMutation: + set: + - name: "my-header" + value: "bar" +``` + +### Removing Headers + +The `remove` field removes the specified headers from the request. Header names are **case-insensitive** (per [RFC 2616](https://datatracker.ietf.org/doc/html/rfc2616#section-4.2)). A maximum of 16 entries is allowed. + +```yaml +headerMutation: + remove: + - "x-internal-header" + - "x-debug-header" +``` + +## Body Mutations + +Use `bodyMutation` to add, overwrite, or remove top-level JSON fields in the HTTP request body. + +:::warning +Only **top-level** fields are currently supported. Nested field paths are not available. +::: + +### Setting Body Fields + +The `set` field overwrites an existing JSON field or adds it if not present. A maximum of 16 entries is allowed. + +```yaml +bodyMutation: + set: + - path: "service_tier" + value: '"scale"' + - path: "max_tokens" + value: "4096" + - path: "temperature" + value: "0.7" +``` + +The `value` field is parsed as raw JSON. This means different value types require different formatting: + +| Type | Example `value` | Result in JSON body | +| ------- | -------------------- | ------------------- | +| String | `'"scale"'` | `"scale"` | +| Number | `"42"` | `42` | +| Boolean | `"true"` | `true` | +| Object | `'{"key": "value"}'` | `{"key": "value"}` | +| Array | `"[1, 2, 3]"` | `[1, 2, 3]` | +| Null | `"null"` | `null` | + +:::tip +String values require inner quotes. For example, to set a field to the string `"scale"`, use `'"scale"'` in YAML. Numeric and boolean values do not need inner quotes. +::: + +### Removing Body Fields + +The `remove` field removes the specified top-level fields from the request body. A maximum of 16 entries is allowed. + +```yaml +bodyMutation: + remove: + - "internal_flag" + - "debug_mode" +``` + +## Complete Examples + +### Example 1: AIServiceBackend with Mutations + +This example configures an `AIServiceBackend` that adds a custom organization header and sets `service_tier` while removing an internal tracking field for all requests routed to this backend: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1alpha1 +kind: AIServiceBackend +metadata: + name: my-openai-backend +spec: + schema: + name: OpenAI + backendRef: + name: my-openai-backend + kind: Backend + group: gateway.envoyproxy.io + headerMutation: + set: + - name: "x-custom-org" + value: "my-org-id" + bodyMutation: + set: + - path: "service_tier" + value: '"scale"' + remove: + - "internal_tracking_id" +``` + +### Example 2: AIGatewayRoute backendRef with Mutations + +This example configures route-level mutations that apply only when requests match a specific route rule. Here, requests for `gpt-4` get a premium header and an increased `max_tokens` value: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1alpha1 +kind: AIGatewayRoute +metadata: + name: my-route +spec: + parentRefs: + - name: my-gateway + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gpt-4 + backendRefs: + - name: my-openai-backend + headerMutation: + set: + - name: "x-route-specific" + value: "premium" + bodyMutation: + set: + - path: "max_tokens" + value: "8192" +``` + +## Precedence Rules + +:::note +When both route-level and backend-level mutations are defined for the same backend: + +- **Route-level mutations take precedence** over backend-level mutations for conflicting operations (e.g., both setting the same header name or body field path). +- **Non-conflicting operations from both levels are applied together.** For example, if the backend-level sets header `x-org` and the route-level sets header `x-tier`, both headers are added to the request. + ::: + +## References + +- [AIServiceBackend](../../api/api.mdx#aiservicebackend) +- [AIGatewayRoute](../../api/api.mdx#aigatewayroute) diff --git a/site/versioned_docs/version-1.2/capabilities/traffic/index.md b/site/versioned_docs/version-1.2/capabilities/traffic/index.md new file mode 100644 index 0000000000..e260399999 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/traffic/index.md @@ -0,0 +1,6 @@ +--- +id: traffic +title: Traffic Handling +--- + +This section provides information about traffic routing related capabilities in Agent Router, including [Quota Policy](./quota-policy.md) for controlling total token consumption budgets across time windows. diff --git a/site/versioned_docs/version-1.2/capabilities/traffic/model-virtualization.md b/site/versioned_docs/version-1.2/capabilities/traffic/model-virtualization.md new file mode 100644 index 0000000000..8ccabf77aa --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/traffic/model-virtualization.md @@ -0,0 +1,156 @@ +--- +id: model-name-virtualization +title: Model Name Virtualization +sidebar_position: 7 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Agent Router provides an advanced model name virtualization capability that allows you to manage and route requests to different AI models seamlessly. +This guide covers the key feature and configuration for model virtualization. + +## Motivation + +It is not uncommon for multiple AI providers to offer a similar or identical model, such as Llama-3-70b, etc. +However, each provider tends to have its own unique naming convention for the same model. +For example, `Claude 4 Sonnet` is hosted both on GCP and AWS Bedrock, but they have different model names: + +- GCP: `claude-sonnet-4@20250514`, etc. +- AWS Bedrock: `anthropic.claude-sonnet-4-20250514-v1:0` + +Even within the same provider, model names may vary based on deployment configurations or versions. +For example, an OpenAI Platform request to `gpt-5-nano` might result in a response from `gpt-5-nano-2025-08-07`. +Azure OpenAI uses deployment names in the URL path rather than model names in the request body, so you won't know the model in use until you get the response. + +From downstream GenAI applications' perspective, it is beneficial to have a unified model name that abstracts away these differences. We do this while still availing the authoritative model name that served a response. + +## Provider-specific Virtualization + +AI providers handle model naming and execution differently to balance flexibility, determinism, and optimization. This section categorizes these behaviors into virtualization types, enabling Agent Router to abstract differences for unified downstream access while retaining executed model details where possible. + +- **Automatic Routing & Resolution**: Providers select and optimize models at runtime behind generic identifiers, returning the actual executed version for reproducibility and debugging. +- **Static Model Execution**: Uses direct mapping to immutable identifiers or fixed versions for deterministic execution, ensuring requested models run exactly as specified, ideal for consistent embeddings. +- **Deterministic Snapshot Mapping**: Resolves models via endpoint/URL with dated snapshots (no response model field), providing consistent cross-platform behavior through timestamped versions. +- **URI-Based Resolution**: Ignores request body model field, using URI path for deployment-based routing while returning the actual versioned model name in responses. +- **Third-Party Delegation**: Delegates embeddings to specialized third-party services with separate APIs, integrating ecosystem expertise while focusing on core capabilities. + +| Upstream | API Type | Virtualization Type | Request Model Example | Response Model Example | +| ---------------- | ---------------- | ------------------------------ | ----------------------------------------- | -------------------------- | +| **awsbedrock** | Chat Completions | Static Model Execution | `anthropic.claude-sonnet-4-20250514-v1:0` | N/A (no model field) | +| **awsbedrock** | Embeddings | Static Model Execution | `amazon.titan-embed-text-v2:0` | N/A (no model field) | +| **azureopenai** | Chat Completions | URI-Based Resolution | `{any-value}` (ignored) | `gpt-5-nano-2025-08-07` | +| **azureopenai** | Embeddings | URI-Based Resolution | `{any-value}` (ignored) | `text-embedding-ada-002-2` | +| **gcpanthropic** | Chat Completions | Deterministic Snapshot Mapping | `claude-sonnet-4@20250514` | N/A (no model field) | +| **gcpanthropic** | Embeddings | Third-Party Delegation | `voyage-3.5` | N/A (no model field) | +| **gcpvertexai** | Chat Completions | Deterministic Snapshot Mapping | `gemini-1.5-pro-002` | N/A (no model field) | +| **gcpvertexai** | Embeddings | Static Model Execution | `text-embedding-004` | N/A (no model field) | +| **openai** | Chat Completions | Automatic Routing & Resolution | `gpt-5-nano` | `gpt-5-nano-2025-08-07` | +| **openai** | Embeddings | Static Model Execution | `text-embedding-3-small` | `text-embedding-3-small` | + +### Azure OpenAI URI-Based Behavior + +Azure OpenAI has a unique approach where [the model field in the request JSON is completely ignored][azure-model-ignored]. The deployment name in the URI path determines which model is actually used, and the response returns the actual versioned model name: + +- **URI Format**: `https://{resource}.openai.azure.com/openai/deployments/{deployment-name}/chat/completions` +- **Request Body**: `{"model": "anything"}` ← This value is ignored +- **Response**: `{"model": "gpt-5-nano-2025-08-07"}` ← Returns the actual versioned model name + +This means [Azure OpenAI doesn't read the model field from the request JSON at all][azure-deployment-resolution] - it uses the deployment name from the URI path exclusively, but returns the real underlying versioned model identifier in the response. + +### AWS Bedrock Static Execution + +AWS Bedrock uses static model execution with immutable versioned identifiers. The Converse API does not return a model field in responses. Each model ID like `anthropic.claude-sonnet-4-20250514-v1:0` represents a specific, frozen model version with no automatic updates or routing. + +### GCP Anthropic Embeddings Delegation + +GCP Anthropic delegates embeddings to [Voyage AI's specialized embedding models][anthropic-embeddings-voyage] rather than providing Claude-native embeddings. Available models include: + +- `voyage-3.5` - Balanced performance embedding model +- `voyage-3-large` - Best quality general-purpose embeddings +- `voyage-code-3` - Optimized for code retrieval +- `voyage-law-2` - Legal domain-specific embeddings + +This delegation approach allows Anthropic to focus on language model capabilities while leveraging domain expertise for embeddings. + +### OpenAI Virtualization + +OpenAI performs automatic model routing where [generic identifiers like `gpt-5-nano` may route to different model versions][openai-model-routing] like `gpt-5-nano-2025-08-07` based on availability and optimization. + +### GCP Provider Behavior + +Both GCP Anthropic and GCP Vertex AI use endpoint-based model resolution where [the model is specified in the URI path rather than returned in the response body][gcp-endpoint-resolution]. This provides deterministic behavior with timestamped snapshots. + +## Virtualization with modelNameOverride API + +In our top level AIGatewayRoute configuration, you can specify a `modelNameOverride` inside [AIGatewayRouteBackendRef](/api/api.mdx#aigatewayrouterulebackendref) on each route rule to override the model name that is sent to the upstream AI provider. +This feature is primarily designed for scenarios where you want to dynamically change the model name based on the actual AI provider the request is being sent to. + +The example configuration looks like this: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: test-route +spec: + targetRefs: [...] + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: claude-4-sonnet + backendRefs: + - name: aws-backend + modelNameOverride: anthropic.claude-sonnet-4-20250514-v1:0 + weight: 50 + - name: gcp-backend + modelNameOverride: claude-sonnet-4@20250514 + weight: 50 +``` + +This configuration allows downstream applications to use a unified model name `claude-4-sonnet` while splitting traffic between the AWS Bedrock and GCP AI providers based on the specified `modelNameOverride`. +This is what the word "Virtualization" means in this context: abstracting away the differences in model names across different AI providers and providing a unified interface for downstream applications. +It also can be thought of as "one-to-many" aliasing of model names, where one unified model name can map to multiple different model names on different providers depending on the routing path. + +## Virtualization for fallback scenarios + +As we see in the [Provider Fallback](./provider-fallback) page, Agent Router allows you to fallback to a different AI provider if the primary one fails. +However, sometimes we want to fallback to a different model on the same provider. +For example, it is natural to set up the Agent Router in a way that if the primary expensive model fails (rate limit, etc), Envoy retries the request to a less expensive model on the same provider. +More concretely, if the request to `gpt-5-nano` fails, we want to retry it with `gpt-5-nano-mini` on the same OpenAI provider. + +`modelNameOverride` can also be used in this scenario to achieve the desired behavior. The configuration would look like this: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: test-route +spec: + targetRefs: [...] + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gpt-5-nano + backendRefs: + - name: openai-backend + # This doesn't specify modelNameOverride, so it will use the default model name `gpt-5-nano` in the request. + priority: 0 + - name: openai-backend + modelNameOverride: gpt-5-nano-mini + priority: 1 +``` + +With this configuration, assuming the retry is properly configured as per the [Provider Fallback](./provider-fallback) page, if the request to `gpt-5-nano` fails, Agent Router will automatically retry the request to `gpt-5-nano-mini` on the same OpenAI provider without requiring any changes to the downstream application. + +--- + +[azure-model-ignored]: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/chatgpt +[azure-deployment-resolution]: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/faq +[anthropic-embeddings-voyage]: https://docs.anthropic.com/en/docs/build-with-claude/embeddings +[openai-model-routing]: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/chatgpt +[gcp-endpoint-resolution]: https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude diff --git a/site/versioned_docs/version-1.2/capabilities/traffic/provider-fallback.md b/site/versioned_docs/version-1.2/capabilities/traffic/provider-fallback.md new file mode 100644 index 0000000000..5958c5b50f --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/traffic/provider-fallback.md @@ -0,0 +1,112 @@ +--- +id: provider-fallback +title: Provider Fallback +sidebar_position: 6 +--- + +# Provider Fallback + +Agent Router supports provider fallback to ensure high availability and reliability for AI/LLM workloads. With fallback, you can configure multiple upstream providers for a single route, so that if the primary provider fails (due to network errors, 5xx responses, or other health check failures), traffic is automatically routed to a healthy fallback provider. + +## When to Use Fallback + +- To ensure uninterrupted service when a primary AI/LLM provider is unavailable. +- To provide redundancy across multiple cloud or on-premise model providers. +- To implement active-active or active-passive failover strategies for critical AI workloads. + +## How Fallback Works + +- **Primary and Fallback Backends:** You can specify a prioritized list of backends in your `AIGatewayRoute` using `backendRefs`. The first backend is treated as primary, and subsequent backends are considered fallbacks. +- **Retry Policy:** Fallback is triggered based on retry policies, which can be configured using the [`BackendTrafficPolicy`](https://gateway.envoyproxy.io/contributions/design/backend-traffic-policy/) API. +- **Automatic Failover:** When the primary backend becomes unhealthy, Agent Router automatically shifts traffic to the next healthy fallback backend. + +## Example + +Below is an example configuration that demonstrates provider fallback from a failing upstream to AWS Bedrock: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: provider-fallback + namespace: default +spec: + parentRefs: + - name: provider-fallback + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: us.meta.llama3-2-1b-instruct-v1:0 + backendRefs: + - name: provider-fallback-always-failing-upstream # Primary backend (expected to fail) + priority: 0 + - name: provider-fallback-aws # Fallback backend + priority: 1 +``` + +The corresponding `Backend` resources: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: Backend +metadata: + name: provider-fallback-always-failing-upstream + namespace: default +spec: + endpoints: + - fqdn: + hostname: provider-fallback-always-failing-upstream.default.svc.cluster.local + port: 443 +--- +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: Backend +metadata: + name: provider-fallback-aws + namespace: default +spec: + endpoints: + - fqdn: + hostname: bedrock-runtime.us-east-1.amazonaws.com + port: 443 +``` + +## Configuring Fallback Behavior + +Attach a `BackendTrafficPolicy` to the generated `HTTPRoute` to control retry behavior: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: BackendTrafficPolicy +metadata: + name: provider-fallback +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: HTTPRoute + name: provider-fallback # HTTPRoute is created with the same name as AIGatewayRoute + retry: + # This ensures that only one attempt is made per priority. + # For example, if the primary backend fails, it will not retry on the same backend. + numAttemptsPerPriority: 1 + numRetries: 5 + perRetry: + backOff: + baseInterval: 100ms + maxInterval: 10s + timeout: 30s + retryOn: + httpStatusCodes: + - 500 + triggers: + - connect-failure + - retriable-status-codes +``` + +## References + +- [Provider Fallback Example](https://github.com/theagentrouter/agent-router/tree/main/examples/provider_fallback) +- [`BackendTrafficPolicy` API Design](https://gateway.envoyproxy.io/contributions/design/backend-traffic-policy/) diff --git a/site/versioned_docs/version-1.2/capabilities/traffic/quota-policy.md b/site/versioned_docs/version-1.2/capabilities/traffic/quota-policy.md new file mode 100644 index 0000000000..ee7f54c78f --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/traffic/quota-policy.md @@ -0,0 +1,369 @@ +--- +id: quota-policy +title: Quota Policy +sidebar_position: 6 +--- + +# Quota Policy + +`QuotaPolicy` enables token-based quota management for AI inference services in Agent Router. +When all related backend's quota are exceeded, requests are rejected with a `429 Too Many Requests` status code. + +:::note QuotaPolicy vs. usage-based rate limiting +`QuotaPolicy` manages the platform's upstream consumption budget: how many tokens the gateway may +spend against a provider backend and model across all routes and consumers. [Usage-based rate +limiting](./usage-based-ratelimiting.md) manages consumer usage: how much a client may consume on a +route. Both can count tokens and return `429 Too Many Requests`, but they answer different questions: +whether the platform has exhausted its budget for a provider/model, or whether a client has exceeded +its allowance. See [Choosing a policy](#choosing-a-policy) for details. +::: + +## Overview + +Key features of QuotaPolicy: + +- **Per-model token quotas** — assign token budgets to individual models served by an `AIServiceBackend`. +- **CEL cost expressions** — weight input, output, cached, and reasoning tokens differently when + computing how much a request burns down a quota. +- **Client-selector bucket rules** — carve out per-tenant or per-header quotas using request attributes. +- **Shadow mode** — evaluate quota rules without enforcing them, for safe rollout. + +## Choosing a Policy + +Both policies use Envoy's rate limit protocol and Redis-backed counters, but `QuotaPolicy` is enforced +by a dedicated AI Gateway rate limit service. Usage-based rate limiting uses Envoy Gateway's global +rate limit service. Choose between them based on the budget's intent and scope: + +| | `QuotaPolicy` | Usage-based rate limiting | +| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| Configuration | One AI Gateway policy containing the token cost and quota buckets | `llmRequestCosts` on an `AIGatewayRoute`, plus an Envoy Gateway `BackendTrafficPolicy` | +| Scope | Backend- and model-scoped - the budget follows an `AIServiceBackend` across every route that sends traffic to it | Route/Gateway-scoped - budgets are keyed by client descriptors on a route | +| Token accounting | One default or custom CEL cost per model | Multiple metadata keys can track and limit input, output, total, or custom token costs separately | +| Client budgets | Default and header-selected buckets, with optional shadow mode | Envoy Gateway client selectors and rate limit rules | +| Enforcement | In `Shared` mode, a request is denied only when all applicable quota buckets are exhausted; the quota is evaluated for the selected backend | A request is denied when any matched rate limit is exceeded; limits are evaluated from the route's client descriptors | +| Time windows | Exactly one second, minute, hour, or day | One second, minute, hour, day, month, or year | + +Use `QuotaPolicy` when the platform needs to protect its provider budget for a backend and model. +For example, a platform team can give every tenant a separate daily budget for an expensive model, +set a shared backend budget alongside those tenant budgets, and evaluate a new tenant rule in shadow +mode. The policy stays with the `AIServiceBackend` even when multiple routes send traffic to it. + +Use usage-based rate limiting when you need to define each client's allowance on a route. It is also +the natural choice when you need lower-level Envoy Gateway controls, route-specific token cost +metadata, separate limits for input and output tokens, or a monthly or yearly window. + +## How It Works + +1. A `QuotaPolicy` is attached to one or more `AIServiceBackend` resources via `targetRefs`. +2. For each completed request, the token cost is computed using the configured cost expression + (defaults to `total_tokens`). +3. The cost is charged against the matching quota bucket (the per-model default bucket, or a matching + bucket rule). +4. When all related quota buckets for that model are exceeded, subsequent matching requests receive `429 Too Many Requests`. + +:::tip Prerequisites +Quota enforcement requires two components that are not deployed by the AI Gateway Helm chart today: + +1. **Redis** stores quota counters. See the + [redis.yaml example](https://github.com/theagentrouter/agent-router/blob/main/examples/token_ratelimit/redis.yaml) + for a simple deployment. +2. **A dedicated rate limit service** evaluates the `ai-gateway-quota` domain. It must use the AI + Gateway controller's xDS server for configuration, with node ID `envoy-ai-gateway-ratelimit`, and + listen at the controller's `quotaRateLimitServiceAddr`. The + [quota E2E manifest](https://github.com/theagentrouter/agent-router/blob/main/tests/e2e/testdata/backend_quota_ratelimit.yaml) + provides a deployment example. + +Envoy Gateway's rate-limit addon is a separate service used by usage-based rate limiting. It is not +required for a QuotaPolicy-only deployment, although both services can use the same Redis instance. + +By default, `controller.quotaRateLimitFailureModeDeny` is `false`. If the dedicated service is absent +or unreachable, quota checks fail open and requests continue without enforcement. Set it to `true` +if unavailable quota enforcement should reject requests instead. +::: + +## Configuration + +### Per-Model Quotas + +Use `perModelQuotas` to apply a token budget to a specific model served by the targeted backend(s). + +:::warning The model name must match the route +A `perModelQuotas` entry only applies when its `modelName` matches the `modelNameOverride` set on the +`AIGatewayRoute` rule's `backendRefs` for the targeted backend. If they do not match, the quota is +silently **not** applied. +::: + +Given an `AIGatewayRoute` that routes a model to the backend: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1alpha1 +kind: AIGatewayRoute +metadata: + name: my-route +spec: + rules: + - backendRefs: + - name: my-backend + modelNameOverride: my-model # <-- the QuotaPolicy modelName must match this +``` + +attach a `QuotaPolicy` to the backend: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1alpha1 +kind: QuotaPolicy +metadata: + name: my-quota-policy +spec: + targetRefs: + - group: aigateway.envoyproxy.io + kind: AIServiceBackend + name: my-backend + perModelQuotas: + - modelName: "my-model" + quota: + mode: Shared + defaultBucket: + limit: 10000 # Maximum tokens allowed in the window. + duration: "1h" # Sliding window. +``` + +You can attach quotas for multiple models, each with its own budget: + +```yaml +perModelQuotas: + - modelName: gpt-4 + quota: + defaultBucket: + limit: 10000 # Strict limit for the expensive model. + duration: "1h" + - modelName: gpt-3.5-turbo + quota: + defaultBucket: + limit: 100000 # Higher limit for the cost-effective model. + duration: "1h" +``` + +:::note +When multiple `QuotaPolicy` resources define the same model for the same `AIServiceBackend`, the +policy whose namespace/name sorts alphabetically first takes precedence. +::: + +### Custom Cost Expression + +By default, a request's cost is its `total_tokens`. You can override this with a +[CEL](https://github.com/google/cel-spec) expression that weights token types differently. The +following variables are available in a `costExpression`: + +| Variable | Type | Description | +| ----------------------------- | ------ | ------------------------------------------------ | +| `input_tokens` | uint | Prompt / input tokens. | +| `output_tokens` | uint | Completion / output tokens. | +| `total_tokens` | uint | Total tokens (the default cost). | +| `cached_input_tokens` | uint | Input tokens served from the provider's cache. | +| `cache_creation_input_tokens` | uint | Input tokens charged for writing to the cache. | +| `reasoning_tokens` | uint | Reasoning tokens (for reasoning-capable models). | +| `model` | string | The resolved model name. | +| `backend` | string | The serving backend name. | +| `route_name` | string | The route name. | + +```yaml +perModelQuotas: + - modelName: gpt-4 + quota: + # Cached input tokens count as 1/10 of a regular input token; + # output tokens count 6x. + costExpression: "input_tokens + cached_input_tokens / 10u + output_tokens * 6u" + defaultBucket: + limit: 50000 + duration: "1h" +``` + +:::tip +Use a custom cost expression when token types have significantly different costs with your provider — +for example, output tokens are typically more expensive than input tokens. + +The token variables are **unsigned integers**, so numeric literals must carry a `u` suffix (for +example `output_tokens * 6u`) and the expression must evaluate to a non-negative integer. Integer +division truncates (`cached_input_tokens / 10u`); for an exact fractional weight, cast through +floating point — for example `uint(double(cached_input_tokens) * 0.1)`. +::: + +### Bucket Mode + +The `mode` field on a per-model `quota` controls how the `defaultBucket` and matching `bucketRules` +interact when a request matches one or more rules. + +Currently only **`Shared`** mode is supported (it is also the default, so the field can be omitted): + +- The request is charged to **all** matching `bucketRules` **and** the `defaultBucket`. +- The request is allowed only if quota is available in **at least one** matching bucket. + +```yaml +perModelQuotas: + - modelName: gpt-4 + quota: + mode: Shared # Default — may be omitted. + defaultBucket: + limit: 10000 + duration: "1h" +``` + +:::note +An exclusive bucket mode (charging matching rules **or** the default bucket, but not both) is planned +but not yet available. The only accepted value today is `Shared`. +::: + +### Client Selectors with Bucket Rules + +Bucket rules let you carve out dedicated quotas for specific clients identified by request attributes +such as headers. This is useful for multi-tenant deployments where each tenant needs its own budget. +Because the mode is `Shared`, a request that matches a bucket rule is charged against **both** that +rule's bucket **and** the `defaultBucket`. + +```yaml +perModelQuotas: + - modelName: gpt-4 + quota: + defaultBucket: + limit: 10000 # Shared budget across all tenants. + duration: "1h" + bucketRules: + # Premium tenant gets a dedicated, higher per-tenant budget. + - clientSelectors: + - headers: + - name: x-tenant-id + type: Exact + value: premium-tenant + quota: + limit: 50000 + duration: "1h" +``` + +Use `type: Distinct` to create a separate bucket for every unique value of a header — for example, a +per-tenant budget keyed on the tenant ID: + +```yaml +bucketRules: + - clientSelectors: + - headers: + - name: x-tenant-id + type: Distinct # One bucket per unique tenant ID. + quota: + limit: 5000 + duration: "1h" +``` + +Supported header match types are `Exact`, `Distinct`, and `RegularExpression`. + +`clientSelectors` reuse the Envoy Gateway [`RateLimitSelectCondition`](https://gateway.envoyproxy.io/docs/api/extension_types/#ratelimitselectcondition) type, but QuotaPolicy currently applies only the `headers` matcher. The other fields on that type (`sourceCIDR`, `methods`, `path`, `queryParams`) are accepted by the schema but **not yet honored** for quota buckets. + +### Shadow Mode + +Shadow mode lets you test a bucket rule without rejecting traffic. When `shadowMode` is enabled on a +bucket rule, all quota checks are performed (cache lookups, counter updates, telemetry), but the +outcome is never enforced — the request always succeeds even if the quota is exceeded. + +```yaml +bucketRules: + - clientSelectors: + - headers: + - name: x-tenant-id + type: Distinct + quota: + limit: 5000 + duration: "1h" + shadowMode: true # Evaluate but do not enforce. +``` + +:::tip +Use shadow mode when rolling out a new quota rule. Monitor the telemetry to confirm the limit is set +correctly, then enable enforcement by removing `shadowMode` (or setting it to `false`). +::: + +Shadow mode is configured per bucket rule. It cannot be set on the `defaultBucket`. + +## Duration Format + +The `duration` field selects the sliding-window size. It must be exactly one of the following values: + +| Value | Window | +| ------ | ---------- | +| `"1s"` | One second | +| `"1m"` | One minute | +| `"1h"` | One hour | +| `"1d"` | One day | + +The window is fixed-size — arbitrary multiples such as `"30s"` or `"15m"` are **not** valid and will +be rejected by the CRD schema. Choose the `limit` to express your budget within one of these windows. + +The one-day maximum is a current `QuotaPolicy` API restriction, not an inherent limitation of token +accounting. Token-based rate limiting configured through an Envoy Gateway `BackendTrafficPolicy` +also supports `Month` and `Year` units. Use [usage-based rate limiting](./usage-based-ratelimiting.md) +when a token budget needs either of those longer windows. + +## Service-Wide Quota + +:::warning Not yet available +The API exposes a backend-wide `serviceQuota` field — intended to apply a single budget across all +models on a backend — but it is **not yet enforced** (it is currently a known TODO in the API). +Configuring it has no effect on traffic, so use [Per-Model Quotas](#per-model-quotas) to enforce +token quotas today. This section will document `serviceQuota` once enforcement lands. +::: + +## Full Example + +The following `QuotaPolicy` combines per-model quotas, custom cost expressions, bucket rules with +client selectors, and shadow mode. + +```yaml +apiVersion: aigateway.envoyproxy.io/v1alpha1 +kind: QuotaPolicy +metadata: + name: full-quota-policy +spec: + targetRefs: + - group: aigateway.envoyproxy.io + kind: AIServiceBackend + name: my-backend + + perModelQuotas: + - modelName: gpt-4 + quota: + costExpression: "input_tokens + cached_input_tokens / 10u + output_tokens * 6u" + mode: Shared + defaultBucket: + limit: 10000 + duration: "1h" + bucketRules: + # Premium tenant gets a dedicated, higher quota. + - clientSelectors: + - headers: + - name: x-tenant-id + type: Exact + value: premium-tenant + quota: + limit: 50000 + duration: "1h" + # Track per-tenant usage in shadow mode (no enforcement). + - clientSelectors: + - headers: + - name: x-tenant-id + type: Distinct + quota: + limit: 5000 + duration: "1h" + shadowMode: true + + - modelName: gpt-3.5-turbo + quota: + defaultBucket: + limit: 100000 + duration: "1h" +``` + +## References + +- [API Reference](../../api/api.mdx) +- [Usage-Based Rate Limiting](./usage-based-ratelimiting.md) +- [Provider Fallback](./provider-fallback.md) diff --git a/site/versioned_docs/version-1.2/capabilities/traffic/usage-based-ratelimiting.md b/site/versioned_docs/version-1.2/capabilities/traffic/usage-based-ratelimiting.md new file mode 100644 index 0000000000..5677e7a840 --- /dev/null +++ b/site/versioned_docs/version-1.2/capabilities/traffic/usage-based-ratelimiting.md @@ -0,0 +1,336 @@ +--- +id: usage-based-ratelimiting +title: Usage-based Rate Limiting +sidebar_position: 5 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide focuses on AI Gateway's specific capabilities for token-based rate limiting in LLM requests. For general rate limiting concepts and configurations, refer to [Envoy Gateway's Rate Limiting documentation](https://gateway.envoyproxy.io/docs/tasks/traffic/global-rate-limit/). + +:::info Quota Policy vs. usage-based rate limiting +`QuotaPolicy` manages the platform's upstream consumption budget for a provider backend and model +across all routes and consumers. Usage-based rate limiting (this page) manages each client's allowance +on a route. Both can count tokens and return `429 Too Many Requests`. See [Choosing a policy](./quota-policy.md#choosing-a-policy) +for a comparison of enforcement behavior, scope, and time windows. +::: + +## Overview + +AI Gateway leverages Envoy Gateway's Global Rate Limit API to provide token-based rate limiting for LLM requests. Key features include: + +- Token usage tracking based on model and user identifiers +- Configuration for tracking input, output, and total token metadata from LLM responses +- Model-specific rate limiting using AI Gateway headers (`x-ai-eg-model`) which is inserted by the AI Gateway filter with the model name extracted from the request body. +- Support for custom token cost calculations using CEL expressions + +## Token Usage Behavior + +AI Gateway has specific behavior for token tracking and rate limiting: + +1. **Token Extraction**: AI Gateway automatically extracts token usage from LLM responses that follow the OpenAI schema format. The token counts are stored in the metadata specified in your `llmRequestCosts` configuration. + +2. **Rate Limit Timing**: Token usage is charged after the response completes. When a request is received: + - Envoy checks the token budget using usage that has already been charged + - If the existing charged usage is over the limit, the request is rejected with a 429 status code + - If the request is admitted, AI Gateway extracts token usage from the completed response and charges it towards the total + +3. **Token Types**: + - `InputToken`: Counts tokens in the request prompt + - `CachedInputToken`: Counts _cached_ input tokens in the request prompt + - `OutputToken`: Counts tokens in the model's response + - `TotalToken`: Combines both input and output tokens + - `CEL`: Allows custom token calculations using CEL expressions + +4. **Multiple Rate Limits**: You can configure multiple rate limit rules for the same user-model combination. For example: + - Limit total tokens per hour + - Separate limits for input and output tokens + - Custom limits using CEL expressions + +5. **Per Route Custom Token Calculation**: The `llmRequestCosts` defined in your `AIGatewayRoute` spec are scoped exclusively to that specific route. This means multiple `AIGatewayRoute` resources can define the exact same metadata keys, but the gateway will independently apply the correct calculations based on the route's name and namespace. + +To map the request to the correct calculation, the AI Gateway parses the route from the Envoy Gateway Metadata. If that metadata is not present, it falls back to parsing the route name from the route configuration, assuming Envoy Gateway has generated the name in the following format: `httproute///rule/`. + +:::note +For model providers with OpenAI schema transformations (like AWS Bedrock), AI Gateway automatically captures token usage through its request/response transformer. This enables consistent token tracking and rate limiting across different AI services using a unified OpenAI-compatible format. +::: + +### Streaming responses + +For streaming responses, token usage is only known after the stream completes and the final usage +data has been observed. AI Gateway does not interrupt an already admitted stream if that stream +pushes the token count over the configured limit. Instead, the stream completes, its token usage is +charged at stream end, and later matching requests are rejected with `429 Too Many Requests` until +enough usage expires from the rate-limit window. + +For example, with a 1,000-token hourly limit, a request that is admitted while the bucket is empty +can stream 1,200 tokens successfully. The bucket is then 200 tokens over the limit, and subsequent +matching requests are limited until the window has enough capacity again. + +## Configuration + +:::tip Prerequisites + +Rate limiting requires two components to be configured: + +1. **Redis Deployment**: A Redis instance must be running to store rate limit data. See the [redis.yaml example](https://github.com/theagentrouter/agent-router/blob/main/examples/token_ratelimit/redis.yaml) for a simple deployment. + +2. **Envoy Gateway Configuration**: Envoy Gateway must be configured at installation time to enable rate limiting and point to your Redis instance. See [Envoy Gateway Installation Guide](../../getting-started/prerequisites.md#additional-features-rate-limiting-inferencepool-etc) + +::: + +### 1. Configure Token Tracking + +AI Gateway automatically tracks token usage for each request. Configure which token counts you want to track in your `AIGatewayRoute`: + +```yaml +spec: + llmRequestCosts: + - metadataKey: llm_input_token + type: InputToken # Counts tokens in the request + - metadataKey: llm_cached_input_token + type: CachedInputToken # Counts cached input tokens in the request prompt + - metadataKey: llm_output_token + type: OutputToken # Counts tokens in the response + - metadataKey: llm_total_token + type: TotalToken # Tracks combined usage +``` + +For advanced token calculations specific to your use case: + +```yaml +spec: + llmRequestCosts: + - metadataKey: custom_cost + type: CEL + cel: "(input_tokens - cached_input_tokens) + (cached_input_tokens * 0.1) + output_tokens * 1.5" # Example: Weight cached tokens less and weight output tokens more heavily +``` + +LLMRequestCosts can be defined on a per-route level. + +### 2. Configure Rate Limits + +AI Gateway uses Envoy Gateway's Global Rate Limit API to configure rate limits. Rate limits should be defined using a combination of user and model identifiers to properly control costs at the model level. Configure this using a `BackendTrafficPolicy`: + +#### Example: Cost-Based Model Rate Limiting + +The following example demonstrates a common use case where different models have different token limits based on their costs. This is useful when: + +- You want to limit expensive models (like GPT-4) more strictly than cheaper ones +- You need to implement different quotas for different tiers of service +- You want to prevent cost overruns while still allowing flexibility with cheaper models + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: BackendTrafficPolicy +metadata: + name: model-specific-token-limit-policy + namespace: default +spec: + targetRefs: + - name: envoy-ai-gateway-token-ratelimit + kind: Gateway + group: gateway.networking.k8s.io + rateLimit: + type: Global + global: + rules: + # Rate limit rule for GPT-4: 1000 total tokens per hour per user + # Stricter limit due to higher cost per token + - clientSelectors: + - headers: + - name: x-tenant-id + type: Distinct + - name: x-ai-eg-model + type: Exact + value: gpt-4 + limit: + requests: 1000 # 1000 total tokens per hour + unit: Hour + cost: + request: + from: Number + number: 0 # Set to 0 so only token usage counts + response: + from: Metadata + metadata: + namespace: io.envoy.ai_gateway + key: llm_total_token # Uses total tokens from the responses + # Rate limit rule for GPT-3.5: 5000 total tokens per hour per user + # Higher limit since the model is more cost-effective + - clientSelectors: + - headers: + - name: x-tenant-id + type: Distinct + - name: x-ai-eg-model + type: Exact + value: gpt-3.5-turbo + limit: + requests: 5000 # 5000 total tokens per hour (higher limit for less expensive model) + unit: Hour + cost: + request: + from: Number + number: 0 # Set to 0 so only token usage counts + response: + from: Metadata + metadata: + namespace: io.envoy.ai_gateway + key: llm_total_token # Uses total tokens from the response +``` + +:::warning +When configuring rate limits: + +1. Always set the request cost number to 0 to ensure only token usage counts towards the limit +2. Set appropriate limits for different models based on their costs and capabilities +3. Ensure both user and model identifiers are used in rate limiting rules + ::: + +## Per-Tenant Limits Without Per-Tenant Policies + +The rate limits above use a static limit value defined in the `BackendTrafficPolicy` (for example `requests: 1000`). If each tenant needs a different number, that forces a policy per tenant — and, because the policy attaches to routes, often a set of routes per tenant too. At a few thousand tenants that is the dominant cost of the configuration. + +Envoy can instead read the limit value from **dynamic metadata** on each request, so one shared policy serves every tenant and the tenant's own number arrives with the request. Envoy Gateway exposes this as `limit.fromMetadata` on the rate limit rule. + +Nothing about this requires AI Gateway. Any Envoy filter that writes dynamic metadata can supply the number, and the policy points directly at whatever namespace that filter writes to. AI Gateway's part is the other half — turning the LLM's token usage into the **cost** charged against the budget, which is covered above and is the piece nothing else can do. + +:::note Requires Envoy Gateway with `limit.fromMetadata` +`limit.fromMetadata` was added in [envoyproxy/gateway#9216](https://github.com/envoyproxy/gateway/pull/9216). Older releases silently drop the field, which leaves the static `requests` in force. Check your Envoy Gateway version before relying on it. +::: + +### 1. Emit the limit from a preceding filter + +The limit must be a struct with `requests_per_unit` and `unit`, which is what Envoy's rate limit override reads. `unit` is one of `SECOND`, `MINUTE`, `HOUR`, `DAY`, `MONTH`, `YEAR`. + +With `ext_authz`, the auth server already resolves the tenant, so it can return the budget in the same Check response. Envoy files everything a Check response returns under the `envoy.filters.http.ext_authz` namespace: + +```json +{ + "dynamic_metadata": { + "fields": { + "total_limit": { + "struct_value": { + "fields": { + "requests_per_unit": { "number_value": 100000 }, + "unit": { "string_value": "HOUR" } + } + } + } + } + } +} +``` + +Dynamic metadata is deliberate here rather than a request header: it can only be written by filters in the chain, never by the downstream client, so a tenant cannot raise its own limit. No header stripping is needed to make it safe. + +### 2. Read it in a `BackendTrafficPolicy` + +Point `limit.fromMetadata` at the namespace and key the filter wrote. The static `requests`/`unit` stay as the default that applies when the metadata is absent: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: BackendTrafficPolicy +metadata: + name: tenant-token-budget + namespace: default +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: envoy-ai-gateway + rateLimit: + type: Global + global: + rules: + - clientSelectors: + - headers: + - name: x-tenant-id + type: Distinct + limit: + requests: 1000 # default when ext_authz says nothing + unit: Hour + fromMetadata: + namespace: envoy.filters.http.ext_authz # where ext_authz writes + key: total_limit + cost: + request: { from: Number, number: 0 } # charge tokens, not requests + response: + from: Metadata + metadata: + namespace: io.envoy.ai_gateway + key: llm_total_token # emitted by globalLLMRequestCosts +``` + +`Distinct` on `x-tenant-id` gives each tenant its own bucket, and `fromMetadata` gives each bucket its own size. One policy, arbitrary per-tenant numbers, no per-tenant objects. + +### 3. Separate input, output and total budgets + +A tenant plan is usually stated as a token triplet rather than one number. Declare one cost key per token kind on the `GatewayConfig`: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: GatewayConfig +metadata: + name: envoy-ai-gateway + namespace: default +spec: + globalLLMRequestCosts: + - metadataKey: llm_input_token + type: InputToken + - metadataKey: llm_output_token + type: OutputToken + - metadataKey: llm_total_token + type: TotalToken +``` + +Have the auth server return `input_limit`, `output_limit` and `total_limit`, then add one rule per kind, each pairing its limit key with the matching cost key. A tenant stops at whichever budget it exhausts first. + +### Behavior to plan for + +| Metadata value | Effect | +| -------------------------------------------- | ------------------------------------------------ | +| a valid `{ requests_per_unit, unit }` struct | becomes the limit for that request | +| absent | the static `requests`/`unit` on the rule applies | +| `requests_per_unit: 0` | a real limit of `0` — see below | + +:::warning Missing metadata fails open to the static default +When the metadata is absent — the auth server didn't set it, or was bypassed — the request falls back to the static `requests`/`unit`. Because that direction is permissive, keep the static value a **conservative ceiling** rather than a placeholder. +::: + +:::caution A `requests_per_unit` of `0` suspends the tenant +`0` is a real limit, and it is the deliberate way to suspend a tenant. Exactly when it starts denying depends on how you charge cost: + +- **Token cost** (`cost.request.from: Number, number: 0`, charged from the response) — the budget is checked before anything is charged, so the request already in flight is admitted and everything after it is denied. +- **Request count** (no `cost` block, one unit per request) — denied immediately, including the first request. + +So a source meaning _"I have no opinion about this request"_ must **omit** the field entirely. Emitting `0` from an uninitialized field, a failed lookup, or a tenant record with no quota set suspends that tenant instead of falling back. +::: + +:::note When the source can't produce the struct +Some sources emit their own shape and can't be changed — `jwt_authn` writes claims verbatim, and a third-party auth sidecar may only return a string like `"100000/HOUR"`. Envoy's override reads only the `{ requests_per_unit, unit }` struct, so in that case you need a filter in between to transcode it, such as a small Lua filter writing the struct into its own metadata namespace and pointing `fromMetadata` there. +::: + +## Making Requests + +For proper cost control and rate limiting, requests must include: + +- `x-tenant-id`: Identifies the user making the request + +Example request: + +```shell +curl -H "Content-Type: application/json" \ + -H "x-tenant-id: user123" \ + -d '{ + "model": "gpt-4", + "messages": [ + { + "role": "user", + "content": "Hello!" + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` diff --git a/site/versioned_docs/version-1.2/cli/index.md b/site/versioned_docs/version-1.2/cli/index.md new file mode 100644 index 0000000000..afcc71fa0f --- /dev/null +++ b/site/versioned_docs/version-1.2/cli/index.md @@ -0,0 +1,17 @@ +--- +id: cli +title: "Agent Router CLI" +sidebar_position: 3 +--- + +# Agent Router CLI (aigw) + +The Agent Router CLI, `aigw`, is a command-line interface that provides a set of useful tools for using the gateway. + +:::warning +The CLI is experimental and currently under active development. +::: + +Currently, you can do the following with the `aigw` CLI: + +- **Run**: Run the Agent Router locally as a standalone proxy with a given configuration file without any dependencies such as docker or Kubernetes. diff --git a/site/versioned_docs/version-1.2/cli/installation.md b/site/versioned_docs/version-1.2/cli/installation.md new file mode 100644 index 0000000000..3600f957a4 --- /dev/null +++ b/site/versioned_docs/version-1.2/cli/installation.md @@ -0,0 +1,100 @@ +--- +id: aigwinstall +title: Installation +sidebar_position: 1 +--- + +## Official CLI binaries + +Each release includes the binaries for the `aigw` CLI build for different platforms.
+They can be downloaded directly from the corresponding release in the +[GitHub releases page](https://github.com/theagentrouter/agent-router/releases). + +## Using the Docker image + +You can also use the official Docker images to run the CLI without installing it locally. +The CLI images are available at: https://hub.docker.com/r/envoyproxy/ai-gateway-cli/tags + +To run the CLI using Docker, you only need to expose the port where the standalone `aigw` listens to +and configure the environment variables for the credentials. If you want to use a custom configuration file, +you can mount it as a volume. + +The following example runs the AI Gateway with the default configuration for the [OpenAI provider](../getting-started/connect-providers/openai.md): + +```shell +docker run --rm -p 1975:1975 -e OPENAI_API_KEY=OPENAI_API_KEY envoyproxy/ai-gateway-cli run +``` + +## Building the latest version + +To use the latest version, you can use the following commands to clone the repo and build the CLI: + +```shell +git clone https://github.com/theagentrouter/agent-router.git +cd agent-router +go install ./cmd/aigw +``` + +:::tip +`go install` command installs a binary in the `$(go env GOPATH)/bin` directory. +Make sure that the `$(go env GOPATH)/bin` directory is in your `PATH` environment variable. + +For example, you can add the following line to your shell profile (e.g., `~/.bashrc`, `~/.zshrc`, etc.): + +```sh +export PATH=$PATH:$(go env GOPATH)/bin +``` + +::: + +Now, you can check if the installation was successful by running the following command: + +```sh +aigw --help +``` + +This will display the help message for the `aigw` CLI like this: + +``` +Usage: aigw [flags] + +Envoy AI Gateway CLI + +Flags: + -h, --help Show context-sensitive help. + +Commands: + version [flags] + Show version. + + run [] [flags] + Run the AI Gateway locally for given configuration. + +Run "aigw --help" for more information on a command. +``` + +## Configuration + +The [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defines standard locations for user-specific files: + +- **Config files**: User-specific configuration (persistent, shared) +- **Data files**: Downloaded binaries (persistent, shared) +- **State files**: Logs and configs per run (persistent, debugging) +- **Runtime files**: Ephemeral files like sockets (deleted on reboot) + +`aigw` adopts these conventions to separate configuration, downloaded Envoy binaries, logs, and ephemeral runtime files. + +| Environment Variable | Default Path | CLI Flag | +| -------------------- | --------------------- | --------------- | +| `AIGW_CONFIG_HOME` | `~/.config/aigw` | `--config-home` | +| `AIGW_DATA_HOME` | `~/.local/share/aigw` | `--data-home` | +| `AIGW_STATE_HOME` | `~/.local/state/aigw` | `--state-home` | +| `AIGW_RUNTIME_DIR` | `/tmp/aigw-${UID}` | `--runtime-dir` | + +**Priority**: CLI flags > Environment variables > Defaults + +## What's next? + +The following sections provide more information about each of the CLI commands: + +- [aigw run](./run.md): Run the AI Gateway locally for a given configuration. diff --git a/site/versioned_docs/version-1.2/cli/run.md b/site/versioned_docs/version-1.2/cli/run.md new file mode 100644 index 0000000000..5b9b2802aa --- /dev/null +++ b/site/versioned_docs/version-1.2/cli/run.md @@ -0,0 +1,383 @@ +--- +id: aigwrun +title: aigw run +sidebar_position: 2 +--- + +# `aigw run` + +## Overview + +This command runs the Agent Router locally as a standalone proxy with a given configuration file without any dependencies such as docker or Kubernetes. +Since the project is primarily focused on the Kubernetes environment, this command is useful for testing the configuration locally before deploying it to a Kubernetes cluster. +Not only does it help in testing the configuration, but it is also useful in a local development environment of the provider-agnostic AI applications. + +:::warning +Currently, `aigw run` supports Linux and macOS. +::: + +## Quick Start + +The simplest way to get started is to have `aigw` generate a configuration for +you using environment variables. `aigw` supports auto-configuration using the same +environment variables as the OpenAI SDK, making it easy to integrate with existing +tooling. Here are some examples: + +```bash +# OpenAI +OPENAI_API_KEY=sk-your-key aigw run +# Azure OpenAI +AZURE_OPENAI_ENDPOINT=https://example.openai.azure.com \ + AZURE_OPENAI_API_KEY=your-key \ + OPENAI_API_VERSION=2024-12-01-preview \ + aigw run +# Tetrate Agent Router Service (TARS) +OPENAI_BASE_URL=https://api.router.tetrate.ai/v1 OPENAI_API_KEY=sk-your-key aigw run +# Ollama running locally +OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_API_KEY=unused aigw run +``` + +Now, the AI Gateway is running locally with the default configuration serving at `localhost:1975`. + +Then, open a new terminal and run the following curl commands to test the AI Gateway. +For example, use `qwen2.5:0.5b` model to route to Ollama, assuming Ollama is running locally and `ollama pull qwen2.5:0.5b` is executed to pull the model. + +```shell +curl -H "Content-Type: application/json" -XPOST http://localhost:1975/v1/chat/completions \ + -d '{"model": "qwen2.5:0.5b","messages": [{"role": "user", "content": "Say this is a test!"}]}' +``` + +### Supported Environment Variables + +The following environment variables are compatible with the OpenAI SDK: + +**OpenAI / OpenAI-compatible backends:** + +When `OPENAI_API_KEY` is set, the following environment variables are read: + +| Variable | Required | Example | Description | +| ----------------- | -------- | --------------------------- | ---------------------------------------------------- | +| `OPENAI_API_KEY` | Yes | `sk-proj-...` | API key for authentication (use "unused" for Ollama) | +| `OPENAI_BASE_URL` | No | `https://api.openai.com/v1` | Base URL of your OpenAI-compatible backend | + +**Azure OpenAI:** + +When `AZURE_OPENAI_API_KEY` is set, the following environment variables are read: + +| Variable | Required | Example | Description | +| ----------------------- | -------- | ---------------------------------- | ------------------------------------------- | +| `AZURE_OPENAI_ENDPOINT` | Yes | `https://example.openai.azure.com` | Your Azure endpoint, including the resource | +| `AZURE_OPENAI_API_KEY` | Yes | `abc123...` | API key for authentication | +| `OPENAI_API_VERSION` | Yes | `2024-12-01-preview` | Azure OpenAI API version | + +**Optional headers (both OpenAI and Azure OpenAI):** + +| Variable | Example | Description | +| ------------------- | ---------- | ---------------------------------------------------------------------------------------------- | +| `OPENAI_ORG_ID` | `org-...` | Organization ID - adds `OpenAI-Organization` request header for billing and access control | +| `OPENAI_PROJECT_ID` | `proj_...` | Project ID - adds `OpenAI-Project` request header for project-level billing and access control | + +## Custom Configuration + +To run the AI Gateway with a custom configuration, provide the path to the configuration file as an argument to the `aigw run` command. +For example, to run the AI Gateway with a custom configuration file named `config.yaml`, run the following command: + +```shell +aigw run config.yaml +``` + +The configuration uses the same API as the Agent Router custom resources definitions. See [API Reference](../api/) for more information. + +The best way to start customizing the configuration is to start with an [example configuration](https://github.com/theagentrouter/agent-router/tree/main/examples) and modify it as needed. + +### Modify an Example Configuration + +First, download an example configuration to use as a starting point: + +```shell +curl -o ollama.yaml https://raw.githubusercontent.com/theagentrouter/agent-router/refs/heads/{vars.aigwGitRef}/examples/aigw/ollama.yaml +``` + +Next, let's say change the model matcher from `.*` (match all) to specifically match `deepseek-r1:1.5b` and save the configuration to `custom.yaml`: + +```diff +--- ollama.yaml ++++ custom.yaml +@@ -88,7 +88,7 @@ + - headers: + - type: RegularExpression + name: x-ai-eg-model +- value: .* ++ value: deepseek-r1:1.5b +``` + +You can also use environment variable substitution (`envsubst`) to allow small +changes without needing to copy a file. For example, you could use this syntax +instead, to default the model to the `CHAT_MODEL` variable. + +```diff +--- ollama.yaml ++++ custom.yaml +@@ -88,7 +88,7 @@ + - headers: + - type: RegularExpression + name: x-ai-eg-model +- value: .* ++ value: ${CHAT_MODEL:=deepseek-r1:1.5b} +``` + +### Run with the Custom Configuration + +Now, run the AI Gateway with the custom configuration by running the following command: + +```shell +aigw run custom.yaml +``` + +Now, the AI Gateway is running locally with the custom configuration serving at `localhost:1975`. + +```shell +curl -H "Content-Type: application/json" -XPOST http://localhost:1975/v1/chat/completions \ + -d '{"model": "deepseek-r1:1.5b","messages": [{"role": "user", "content": "Say this is a test!"}]}' +``` + +## MCP Configuration + +`aigw run` supports running as an [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) Gateway, allowing AI agents to connect to multiple MCP servers through a unified endpoint. The gateway aggregates tools from multiple backends, applies security policies, and provides observability for MCP traffic. + +### Using MCP Servers Configuration File + +Use the `--mcp-config` flag to load MCP server configuration from a JSON file. The format follows the canonical [MCP servers configuration](https://modelcontextprotocol.io/docs/getting-started/intro#configuring-mcp-servers) used by clients like Claude Desktop, Cursor, and VS Code: + +```json +{ + "mcpServers": { + "context7": { + "type": "http", + "url": "https://mcp.context7.com/mcp" + }, + "github": { + "type": "http", + "url": "https://api.githubcopilot.com/mcp/readonly", + "headers": { + "Authorization": "Bearer ${GITHUB_ACCESS_TOKEN}" + }, + "includeTools": ["issue_read", "list_issues"] + } + } +} +``` + +Start the MCP Gateway: + +```shell +aigw run --mcp-config mcp-servers.json +``` + +The gateway will be available at `http://localhost:1975/mcp`. Configure your MCP client to connect to this URL. + +### Using Inline JSON Configuration + +For quick testing or when the configuration is provided programmatically, use the `--mcp-json` flag: + +```shell +aigw run --mcp-json '{"mcpServers":{"context7":{"type":"http","url":"https://mcp.context7.com/mcp"}}}' +``` + +### MCP Servers Configuration Format + +Each server configuration in the `mcpServers` object supports the following properties: + +**`type`** (string, required) + +The transport protocol for the MCP server. Supported values: `"http"`, `"streamable-http"`, `"streamable_http"`, or `"streamableHttp"`. + +**`url`** (string, required) + +The full endpoint URL for the MCP server, including protocol, hostname, and path. + +**`headers`** (object, optional) + +HTTP headers to send with requests to the MCP server. Authorization headers with Bearer tokens are automatically extracted and injected as API keys. + +Example: + +```json +"headers": { + "Authorization": "Bearer ${GITHUB_TOKEN}", + "X-Custom-Header": "value" +} +``` + +Header values support environment variable substitution using `${VAR}` syntax. For example, `${GITHUB_TOKEN}` will be replaced at runtime with the value of the `GITHUB_TOKEN` environment variable. + +**`includeTools`** (array of strings, optional) + +List of specific tool names to expose from this server. If not specified, all tools from the server are exposed. + +This follows the same convention as [Gemini CLI's tool filtering](https://google-gemini.github.io/gemini-cli/docs/tools/mcp-server.html#optional). + +Example: + +```json +"includeTools": ["issue_read", "list_issues", "search_issues"] +``` + +### Testing the MCP Gateway + +Use the [MCP Inspector](https://github.com/modelcontextprotocol/inspector) to test your gateway: + +```shell +# List available tools +npx @modelcontextprotocol/inspector@0.16.8 --cli http://localhost:1975/mcp --method tools/list + +# Call a tool +npx @modelcontextprotocol/inspector@0.16.8 \ + --cli http://localhost:1975/mcp \ + --method tools/call \ + --tool-name context7__resolve-library-id \ + --tool-arg libraryName="react" +``` + +:::note +Tool names are automatically prefixed with the server name (e.g., `github__issue_read`, `context7__resolve-library-id`) to route calls to the correct backend. +::: + +### Advanced MCP Configuration + +For production deployments with advanced features like OAuth authentication, fine-grained access control, and Kubernetes integration, use the [`MCPRoute` API](/docs/capabilities/mcp/). The same configuration can be used in both `aigw run` standalone mode and Kubernetes environments. + +### Admin Endpoints + +While running, `aigw` serves admin endpoints on port `1064` by default: + +- `/metrics`: Prometheus metrics endpoint for [LLM/AI metrics](../capabilities/observability/metrics.md) +- `/health`: Health check endpoint that verifies the external processor is healthy + +## OpenTelemetry + +Agent Router's router joins and records distributed traces when supplied +with an [OpenTelemetry](https://opentelemetry.io/) collector endpoint. + +Requests to the OpenAI Chat Completions and Embeddings endpoints are recorded +as Spans which include typical timing and request details. In addition, there +are GenAI attributes representing the LLM or Embeddings call including full +request and response details, defined by [OpenInference semantic conventions][openinference]. + +OpenInference attributes default to include full request and response data for +both chat completions and embeddings. This can be toggled with configuration, +but when enabled allows systems like [Arize Phoenix][phoenix] to perform +evaluations of production requests captured in OpenTelemetry spans. + +For chat completions, this includes traditional LLM metrics such as correctness +and hallucination detection. For embeddings, it enables agentic RAG evaluations +focused on retrieval and semantic analysis. + +### OpenTelemetry configuration + +`aigw run` supports OpenTelemetry tracing via environment variables: + +- **[OTEL SDK][otel-env]**: OTLP exporter configuration that controls span + and metrics export such as: + - `OTEL_EXPORTER_OTLP_ENDPOINT`: Collector endpoint (e.g., `http://phoenix:6006`) + - `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`: Override traces endpoint separately + - `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`: Override metrics endpoint separately + - `OTEL_TRACES_EXPORTER`: Traces exporter type (e.g., `console`, `otlp`, `none`) + - `OTEL_METRICS_EXPORTER`: Metrics exporter type (e.g., `console`, `otlp`, `none`) + - `OTEL_BSP_SCHEDULE_DELAY`: Batch span processor delay (default: 5000ms) + - `OTEL_METRIC_EXPORT_INTERVAL`: Metrics export interval (default: 60000ms) + +- **Semantic Convention**: Select which span attributes are recorded. See + [Tracing][tracing] for the attribute differences. + - `AI_GATEWAY_TRACING_SEMCONV`: `openinference` (default) or `gen_ai` for the + OpenTelemetry GenAI conventions. An unrecognized value fails startup. + - `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`: Capture message content + when `gen_ai` is selected (default: `false`). The GenAI conventions treat + content as opt-in, unlike OpenInference. + - This also selects the vocabulary for MCP spans: the default keeps the + gateway-specific `mcp.*` attributes, `gen_ai` opts into the OpenTelemetry MCP + conventions. + +- **[OpenInference][openinference-config]**: Control sensitive data redaction, + such as below. These apply only when `AI_GATEWAY_TRACING_SEMCONV` is + `openinference`. There is [similar config][openinference-embeddings] for embeddings: + - `OPENINFERENCE_HIDE_INPUTS`: Hide input messages/prompts (default: `false`) + - `OPENINFERENCE_HIDE_OUTPUTS`: Hide output messages/completions (default: `false`) + - `OPENINFERENCE_HIDE_EMBEDDINGS_TEXT`: Hide embeddings input (default: `false`) + - `OPENINFERENCE_HIDE_EMBEDDINGS_VECTORS`: Hide embeddings output (default: `false`) + +- **Header Mapping**: Map HTTP request headers to span attributes, metric labels, and access log attributes. See [Session Tracking][session-tracking] for more details. + - `OTEL_AIGW_REQUEST_HEADER_ATTRIBUTES`: Base mapping applied to metrics, spans, and access logs. Example: `x-tenant-id:tenant.id` + - `OTEL_AIGW_SPAN_REQUEST_HEADER_ATTRIBUTES`: Span-only mapping (merged with base). Default: `agent-session-id:session.id` when unset (set to empty to disable). Example: `x-forwarded-proto:url.scheme` + - `OTEL_AIGW_METRICS_REQUEST_HEADER_ATTRIBUTES`: Metrics-only mapping (merged with base). Example: `x-tenant-id:tenant.id` (metrics never default to `session.id`) + - `OTEL_AIGW_LOG_REQUEST_HEADER_ATTRIBUTES`: Access-log-only mapping (merged with base). Default: `agent-session-id:session.id` when unset (set to empty to disable). Example: `x-forwarded-proto:url.scheme` + +See [docker-compose-otel.yaml][docker-compose-otel.yaml] for a complete example configuration. + +## Configuration + +### File Locations + +`aigw run` uses the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) to organize files: + +| Environment Variable | Default Path | Purpose | +| -------------------- | --------------------- | --------------------------- | +| `AIGW_CONFIG_HOME` | `~/.config/aigw` | User configuration files | +| `AIGW_DATA_HOME` | `~/.local/share/aigw` | Downloaded Envoy binaries | +| `AIGW_STATE_HOME` | `~/.local/state/aigw` | Persistent logs and configs | +| `AIGW_RUNTIME_DIR` | `/tmp/aigw-${UID}` | Ephemeral runtime files | + +See [Installation - Configuration](./installation.md#configuration) for more details about XDG directories. + +### File Mappings + +Each invocation creates a unique run identifier (`runID`) in format `YYYYMMDD_HHMMSS_UUU` to isolate concurrent runs: + +| File Type | Purpose | Path | Type | +| ------------------------- | ---------------------------------------- | ---------------------------------------------------------------- | ------- | +| Default Config | Configuration file location | `${AIGW_CONFIG_HOME}/config.yaml` | CONFIG | +| Envoy Version Preference | Selected Envoy version (via func-e) | `${AIGW_CONFIG_HOME}/envoy-version` | CONFIG | +| Envoy Binaries | Downloaded executables (via func-e) | `${AIGW_DATA_HOME}/envoy-versions/{version}/bin/envoy` | DATA | +| AIGW Logs | Gateway logs and stderr output | `${AIGW_STATE_HOME}/runs/{runID}/aigw.log` | STATE | +| Envoy Gateway Config | Generated EG configuration | `${AIGW_STATE_HOME}/runs/{runID}/envoy-gateway-config.yaml` | STATE | +| Envoy Gateway Resources | Generated EG resources (Gateway, Routes) | `${AIGW_STATE_HOME}/runs/{runID}/envoy-ai-gateway-resources/...` | STATE | +| External Processor Config | Generated extproc configuration | `${AIGW_STATE_HOME}/runs/{runID}/extproc-config.yaml` | STATE | +| Envoy Run Logs (func-e) | Envoy stdout/stderr (via func-e) | `${AIGW_STATE_HOME}/envoy-runs/{runID}/stdout.log,stderr.log` | STATE | +| UDS Socket | Unix domain socket for extproc | `${AIGW_RUNTIME_DIR}/{runID}/uds.sock` | RUNTIME | +| Admin Address (func-e) | Envoy admin endpoint (via func-e) | `${AIGW_RUNTIME_DIR}/{runID}/admin-address.txt` | RUNTIME | + +**File Categories:** + +- **CONFIG**: User-specific configuration (persistent, shared across runs) +- **DATA**: Downloaded binaries (persistent, shared across runs) +- **STATE**: Per-run logs and configs (persistent for debugging) +- **RUNTIME**: Ephemeral files like sockets (cleaned on reboot) + +### `runID` + +By default, `aigw run` generates a timestamp-based `runID` for each invocation. You can customize this for predictable paths: + +```shell +# Use run ID "0" for Docker/Kubernetes deployments +aigw run --run-id=0 + +# Or via environment variable +AIGW_RUN_ID=production aigw run +``` + +Custom run IDs: + +- Enable predictable file paths in containers +- Allow correlation across multiple runs with the same ID +- Must not contain path separators (`/` or `\`) + +--- + +[openinference]: https://github.com/Arize-ai/openinference/tree/main/spec +[phoenix]: https://docs.arize.com/phoenix +[otel-env]: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ +[openinference-config]: https://github.com/Arize-ai/openinference/blob/main/spec/configuration.md +[openinference-embeddings]: https://github.com/Arize-ai/openinference/blob/main/spec/embedding_spans.md +[docker-compose-otel.yaml]: https://github.com/theagentrouter/agent-router/blob/main/cmd/aigw/docker-compose-otel.yaml +[session-tracking]: ../capabilities/observability/tracing.md#session-tracking +[tracing]: ../capabilities/observability/tracing.md#semantic-conventions diff --git a/site/versioned_docs/version-1.2/compatibility.md b/site/versioned_docs/version-1.2/compatibility.md new file mode 100644 index 0000000000..3ec7741756 --- /dev/null +++ b/site/versioned_docs/version-1.2/compatibility.md @@ -0,0 +1,79 @@ +--- +id: compatibility +title: Compatibility Matrix +sidebar_position: 6 +--- + +# Compatibility Matrix + +This document provides compatibility information for Agent Router releases with their dependencies. + +| Agent Router | Envoy Gateway | Kubernetes | Gateway API | Inference Extension | Support Status | +| ------------ | ----------------------------- | ---------- | ----------- | ------------------- | -------------- | +| main | v1.9.2+ (Envoy Proxy v1.39.x) | v1.32+ | v1.6.1 | v1.6.2 | Development | +| v1.2.x | v1.9.2+ (Envoy Proxy v1.39.x) | v1.32+ | v1.6.1 | v1.6.2 | Supported | +| v1.1.x | v1.8.1+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.6.0 | Supported | +| v1.0.x | v1.8.1+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.0.2 | Supported | +| v0.7.x | v1.8.x+ (Envoy Proxy v1.38.x) | v1.32+ | v1.5.x | v1.0.2 | Supported | +| v0.6.x | v1.7.x+ (Envoy Proxy v1.37.x) | v1.32+ | v1.4.x | v1.0.2 | Supported | +| v0.5.x | v1.6.x+ (Envoy Proxy v1.35.x) | v1.32+ | v1.4.x | v1.0.2 | Supported | +| others | N/A | N/A | N/A | N/A | End of Life | + +Note that "compatibility" means that these specific combinations have been tested and verified to work together. +Other versions may work but are not officially supported. +Please refer to our [Support Policy](https://github.com/theagentrouter/agent-router/blob/main/RELEASES.md#support-policy) for more details +on how we manage releases and support for different versions. + +To upgrade to a new Agent Router version, make sure upgrade your dependencies accordingly to maintain compatibility, especially make sure that +Envoy Gateway and Gateway API versions are up-to-date as per the compatibility matrix above. Then, upgrade the AI Gateway using the standard helm upgrade process. + +## Upgrading and Migration + +### Helm Upgrade Commands + +```bash +# 1. Upgrade CRDs +helm upgrade aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \ + --version \ + envoy-ai-gateway-system < NEW_VERSION > -n + +# 2. Upgrade application +helm upgrade aieg oci://docker.io/envoyproxy/ai-gateway-helm \ + --version \ + envoy-ai-gateway-system < NEW_VERSION > -n +``` + +### Migrating from v1alpha1 to v1beta1 + +AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, GatewayConfig, and MCPRoute support both v1alpha1 (deprecated) and v1beta1. When you upgrade: + +- **Existing v1alpha1 resources** continue to work. The API server can serve them via both v1alpha1 and v1beta1 endpoints. +- **Storage version migration** is not automatic. To migrate existing resources to v1beta1 storage, you must manually re-apply them or use the storage migration API. +- **New resources** should use `apiVersion: aigateway.envoyproxy.io/v1beta1`. +- **QuotaPolicy** remains v1alpha1-only (no v1beta1 available). + +#### Migrating Storage Version + +Storage version migration requires manual action. After upgrading the CRDs, existing resources remain stored in v1alpha1 format until you explicitly trigger migration. The Kubernetes API server can serve these resources via both v1alpha1 and v1beta1 endpoints, but the underlying storage version in etcd won't change automatically. + +To migrate existing resources from v1alpha1 to v1beta1 storage version: + +**Re-apply resources** (recommended for small deployments) + +Update your manifests to use v1beta1 and re-apply them with `kubectl apply`. + +For more information about storage version migration in Kubernetes, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/storage-version/#migrating-to-a-different-storage-version). + +To adopt v1beta1 in your manifests, change the `apiVersion` field and re-apply: + +```yaml +# Before (v1alpha1, deprecated) +apiVersion: aigateway.envoyproxy.io/v1alpha1 +kind: AIGatewayRoute +# ... + +# After (v1beta1, preferred) +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +# ... +``` diff --git a/site/versioned_docs/version-1.2/concepts/architecture/control-plane.md b/site/versioned_docs/version-1.2/concepts/architecture/control-plane.md new file mode 100644 index 0000000000..2d5d5af806 --- /dev/null +++ b/site/versioned_docs/version-1.2/concepts/architecture/control-plane.md @@ -0,0 +1,135 @@ +--- +id: control-plane +title: Control Plane Explained +sidebar_position: 3 +--- + +# Control Plane Explained + +The control plane is responsible for configuring and managing the system. It consists of several key components working together to manage the AI Gateway configuration. + +```mermaid +graph TB + %% Define styles for different component categories + classDef aiGatewayStyle fill:#f9f,stroke:#E74C3C,color:#000 + classDef envoyGatewayStyle fill:#ddf,stroke:#none,color:#000 + classDef aiGatewayResourceStyle fill:#f9f,stroke:#E74C3C,color:#000 + classDef envoyGatewayResourceStyle fill:#ddf,stroke:#none,color:#000 + classDef generatedResourceStyle fill:#ddf,stroke:#none,color:#000 + classDef envoyProxyStyle fill:#F5CBA7,stroke:#D35400,color:#000 + classDef aiGatewayExtProcStyle fill:#f9f,stroke:#E74C3C,color:#000 + + subgraph "UserAppliedResources"["User Applied Resources"] + subgraph "AI Gateway Resources" + AIGatewayRoute:::aiGatewayResourceStyle + AIServiceBackend:::aiGatewayResourceStyle + BackendSecurityPolicy:::aiGatewayResourceStyle + end + subgraph "Envoy Gateway resources" + Gateway:::envoyGatewayResourceStyle + Backend:::envoyGatewayResourceStyle + EGDots[...]:::envoyGatewayResourceStyle + end + end + style UserAppliedResources fill:none,stroke-dasharray:5 + + AIGateway((Agent Router + Controller)):::aiGatewayStyle + EnvoyGateway((Envoy Gateway + Controller)):::envoyGatewayStyle + subgraph "Generated Resources" + HTTPRoute:::generatedResourceStyle + HTTPRouteFilter:::generatedResourceStyle + ExtProcConfigSecret[AI Gateway ExtProc Secret]:::aiGatewayExtProcStyle + end + + subgraph "EnvoyProxyPod"[Envoy Proxy Pod] + EnvoyProxy[Envoy Proxy]:::envoyProxyStyle + AIExtProc[AI Gateway ExtProc]:::aiGatewayExtProcStyle + end + style EnvoyProxyPod fill:none,stroke:5 + + AIGatewayRoute -.->AIGateway + AIServiceBackend -.->AIGateway + BackendSecurityPolicy -.->AIGateway + Gateway -.->|Watching|AIGateway + + Gateway -.->EnvoyGateway + Backend -.->EnvoyGateway + EGDots -.->EnvoyGateway + + AIGateway -.->HTTPRoute + AIGateway -.->HTTPRouteFilter + AIGateway -.->ExtProcConfigSecret + HTTPRoute -.->EnvoyGateway + HTTPRouteFilter -.->EnvoyGateway + ExtProcConfigSecret -.->AIExtProc + + EnvoyGateway -.->|xDS/Envoy config|EnvoyProxy + AIGateway <-.->|Extension Server protocol|EnvoyGateway + AIGateway -.->|Inserting ExtProc as Sidecar|AIExtProc +``` + +## How It Works + +The control plane operates through a chain of components that work together to manage the configuration: + +1. The Agent Router controller watches AI Gateway Custom Resources (CRs) +2. When changes are detected, it updates/generates the Envoy Gateway configuration +3. The Envoy Gateway communicates with the Agent Router controller via the [Envoy Gateway Extension server] protocol, and the Agent Router controller then fine-tunes the xDS configuration before the Envoy Gateway applies it to the Envoy Proxy. +4. The data plane (Envoy Proxy) processes AI traffic based on this configuration where the AI Gateway ExtProc runs as a sidecar to handle AI-specific processing. The sidecar container is inserted by the AI Gateway controller into the Envoy Proxy Pod. + +This architecture ensures a clear separation of concerns, where the AI Gateway controller focuses on AI-specific configuration while leveraging Envoy Gateway for general proxy management. + +## Components + +### 1. Agent Router Controller + +The AI Gateway Controller manages AI-specific components and configurations: + +#### ExtProc Management + +- Creates and updates ExtProc Secrets with processing rules as well as credentials +- Inserts the AI Gateway ExtProc as a [sidecar container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) in the Envoy Proxy Pod via the [Kubernetes Admission Webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/). The container mounts the ExtProc config secret and communicates with the Envoy Proxy to process AI traffic. + +#### Resource Management + +- Watches AI Gateway Custom Resources (CRs) +- Creates and manages `HTTPRoute` and `HTTPRouteFilter` resources +- Manages backend security policies and authentication, including the credentials rotation + +#### Integration with Envoy Gateway + +- Works alongside Envoy Gateway Controller (not directly configuring Envoy) +- Creates resources, such as `HTTPRoute`, that Envoy Gateway translates into Envoy configuration with AI-specific processing rules +- Serves the [Envoy Gateway Extension Server] to fine-tune the Envoy configuration (xDS) +- Enables token-based rate limiting through metadata + +### 2. Envoy Gateway Controller + +- Manages the core Envoy configuration through xDS +- Handles service discovery and load balancing +- Manages TLS certificates +- Translates Gateway API resources into Envoy configuration + +## Notable Rationale + +- As explained, [Envoy Gateway Extension server] is used for fine-tuning the xDS configuration to implement our features. This allows us to leverage Envoy Gateway for core proxy management while still customizing the configuration for AI-specific needs. + - For example, upstream filters (HTTP filter attached per Backend) is not supported in Envoy Gateway, so the AI Gateway controller uses the extension server to achieve this, which is essential to implement per-model priority routing. + - Another example is that backendRef level priority configuration is not supported in Gateway API or Envoy Gateway, so the AI Gateway controller uses the extension server to insert this configuration into the xDS. +- Delegating core proxy management to Envoy Gateway allows us to focus on AI-specific features and configurations, reducing duplication of effort and leveraging the strengths of both controllers. + - For example, we don't need to re-implement Gateway API resource translation, service discovery, load balancing, and TLS management. +- Inserting the ExtProc as a sidecar container brings the following benefits: + - Performance: Running the ExtProc as a sidecar within the same Pod as Envoy Proxy reduces network latency and overhead, leading to faster processing of AI traffic. Notably, the communication between Envoy Proxy and the ExtProc happens over a local Unix Domain Socket (UDS), not over the network. + - Simplified Deployment: Managing the ExtProc as part of the Envoy Proxy Pod simplifies deployment and scaling, as both components can be managed together. + - Session Affinity: Running as a sidecar naturally ensures the session affinity between Envoy Proxy and the ExtProc which is crucial for implementing provider fallbacks. As we explain in [Data Plane and Traffic Flow], we have two phases in the request processing: + Router-level and Upstream-level processing to handle provider fallbacks. Since Envoy does not support session affinity for External Processor access as of this writing, without the session affinity, the Upstream-level ExtProc might end up talking to a different ExtProc instance which does not have the context of the router level processing (mainly the original request body and headers). + +## Next Steps + +To learn more: + +- Understand the [Data Plane and Traffic Flow] + +[Envoy Gateway Extension server]: https://gateway.envoyproxy.io/docs/tasks/extensibility/extension-server/ +[Data Plane and Traffic Flow]: ./data-plane.md diff --git a/site/versioned_docs/version-1.2/concepts/architecture/data-plane.md b/site/versioned_docs/version-1.2/concepts/architecture/data-plane.md new file mode 100644 index 0000000000..5559d3e4b6 --- /dev/null +++ b/site/versioned_docs/version-1.2/concepts/architecture/data-plane.md @@ -0,0 +1,134 @@ +--- +id: data-plane +title: Data Plane and Traffic Flow +sidebar_position: 4 +--- + +# Data Plane and Traffic Flow + +The data plane handles the actual request traffic, with the External Processor (ExtProc) playing a central role in managing AI-specific processing. + +## Components + +The data plane consists of several key components: + +### 1. Envoy Proxy + +The core proxy that handles all incoming traffic and integrates with: + +- External Processor for AI-specific processing +- Rate Limit Service for token-based rate limiting +- Various AI providers as backends + +### 2. AI Gateway External Processor + +A specialized extension service of Envoy Proxy that handles AI-specific processing needs. It performs three main functions: + +1. **Request Processing** + - Routes requests to appropriate AI providers + - Handles model selection and validation + - Manages provider-specific authentication + - Supports different API formats (OpenAI, AWS Bedrock) + +2. **Token Management** + - Tracks token usage from AI providers + - Handles both streaming and non-streaming responses + - Provides usage data for rate limiting decisions + +3. **Provider Integration** + - Transforms requests between different AI provider formats + - Normalizes responses to a consistent format + - Manages provider-specific requirements + +### 3. Rate Limit Service + +Handles token-based rate limiting by: + +- Tracking token usage across requests +- Enforcing rate limits based on token consumption +- Managing rate limit budgets + +## Request Processing Flow + +```mermaid +sequenceDiagram + participant Client as Client (OpenAI SDK) + participant Envoy as Envoy Proxy + participant RLS as Rate Limit Service + participant Processor as AI Gateway External Processor + participant Provider as AI Provider / Upstream + + Client->>Envoy: Request + Envoy->>Processor: Router-level ExtProc Request + Note over Processor: Extract Model Name + Processor-->>Envoy: ClearRouteCache; + Envoy->>RLS: Check Rate Limit + RLS-->>Envoy: ; + loop Retry/Fallback loop + Note over Envoy: Select Upstream/Endpoint + Envoy->>Processor: Upstream level ExtProc Request + Note over Processor: Request-Transform & Upstream Authnz + Processor-->>Envoy: ; + Envoy->>Provider: Forward Request + Provider-->>Envoy: Response + end + Envoy->>Processor: Process Response + Note over Processor: Response Transform & Extract Token Usage + Processor-->>Envoy: Add Usage Metadata + Envoy->>RLS: Reduce Rate Limit budget + RLS-->>Envoy: ; + Envoy->>Client: Response +``` + +The data plane processes requests through several key steps: + +### 1. Request Path + +1. **Routing**: Calculates the destination AI provider based on: + - Request path + - Headers + - Model name extracted from the request body + +2. **Request Transformation**: Prepares the request for the provider: + - Request body transformation + - Request path modification + - Format adaptation + +3. **Upstream Authorization**: Handles provider authentication: + - API key management + - Header modifications + - Authentication token handling + +4. **Token Rate Limiting Check**: Checks the request against the Rate Limit Service: + - Validates token usage + - Enforces rate limits based on configured budgets + +### 2. Response Path + +1. **Response Transformation**: + - Transforms provider response for client compatibility + - Normalizes response format + - Handles streaming responses + +2. **Token Usage Management**: + - Extracts token usage from responses + - Calculates usage based on configuration + - Stores usage in per-request dynamic metadata + - Enables rate limiting based on token consumption + +## Notable Rationale + +- Why the External Processor is separated into two phases (Router-level and Upstream-level): + - In Envoy, retry/fallback happens after the router filter at the upstream level. For example, when the upstream server returns 5xx, Envoy does not invoke the router level filter again. + Instead, it invokes only the upstream level filters. In our case, retry/fallback will make the requests to totally different AI providers. For example, on the first try, it goes to OpenAI, and on the second try, it goes to AWS Bedrock. + In this case, we need to do different request transformations and upstream authorizations. So, this logic needs to be in the upstream level filter. +- Why the External Processor? + - The External Processor is the most powerful battle-tested and production-ready extension point in Envoy. It allows us to implement complex logic without modifying Envoy's core codebase. + - [Dynamic Modules](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/dynamic_modules) could be a future alternative as it offers better performance as well as less complexity in the overall architecture. The work is tracked in [envoyproxy/ai-gateway#90](https://github.com/theagentrouter/agent-router/issues/90). + +## Next Steps + +To learn more: + +- Explore the [System Architecture](./system-architecture.md) +- Check out our [Getting Started](../../getting-started/index.md) guide for hands-on experience diff --git a/site/versioned_docs/version-1.2/concepts/architecture/index.md b/site/versioned_docs/version-1.2/concepts/architecture/index.md new file mode 100644 index 0000000000..c7d338d37e --- /dev/null +++ b/site/versioned_docs/version-1.2/concepts/architecture/index.md @@ -0,0 +1,47 @@ +--- +id: architecture +title: Architecture +sidebar_position: 2 +--- + +# Architecture + +This section provides a detailed look at the architectural components of Agent Router. Understanding the architecture will help you better deploy, configure, and maintain your gateway installation. + +## Overview + +Agent Router follows a modern cloud-native architecture with distinct control and data planes. This separation of concerns allows for better scalability, maintainability, and flexibility in deployment options. + +Agent Router integrates with Envoy Gateway for the control plane and Envoy Proxy for the data plane. + +## Key Concepts + +### Control Plane + +A control plane is a component that manages the configuration of the data plane. We utilize Envoy Gateway as a central control plane, and Agent Router works in conjunction with it to manage the data plane configuration. + +### Data Plane + +The data plane is the component that sits in the request path and processes the requests. In the context of Agent Router, the data plane consists of Envoy Proxy and the AI Gateway external processor that processes the AI requests. + +## In This Section + +1. [System Architecture Overview](./system-architecture.md) + - High-level architecture overview + - Control and data plane separation + +2. [Control Plane Explained](./control-plane.md) + - AI Gateway Controller + - Envoy Gateway Controller + - Configuration management + - Resource orchestration + +3. [Data Plane and Traffic Flow](./data-plane.md) + - External Processor functionality + - Request processing flow + +## What's Next + +After understanding the architecture: + +- Check out our [Getting Started](../../getting-started/index.md) guide for hands-on experience diff --git a/site/versioned_docs/version-1.2/concepts/architecture/system-architecture.md b/site/versioned_docs/version-1.2/concepts/architecture/system-architecture.md new file mode 100644 index 0000000000..b95e3e63c3 --- /dev/null +++ b/site/versioned_docs/version-1.2/concepts/architecture/system-architecture.md @@ -0,0 +1,66 @@ +--- +id: system-architecture +title: System Architecture Overview +sidebar_position: 1 +--- + +# System Architecture Overview + +Agent Router follows a modern cloud-native architecture pattern with distinct control and data planes. Let's explore how these components work together to manage AI traffic effectively. + +The architecture is divided into two main planes: + +1. **Control Plane**: Responsible for configuring and managing the system + - Kubernetes API Server serves as the configuration interface + - AI Gateway Controller manages AI-specific resources and configurations + - Envoy Gateway Controller handles core proxy configuration and xDS + +2. **Data Plane**: Handles the actual request traffic + - Envoy Proxy and External Processor work together to handle AI traffic + - Envoy Proxy routes and manages all incoming requests + - External Processor handles AI-specific transformations and validations + - Rate Limit Service manages token-based rate limiting + +The Control Plane configures the Data Plane through several paths: + +- AI Gateway Controller configures the External Processor as well as fine-tuning xDS configurations via Envoy Gateway extension server mechanism. +- Envoy Gateway Controller configures Envoy Proxy through xDS as well as Rate Limit Service. + +```mermaid +graph TB + subgraph "Control Plane" + K8s[Kubernetes API Server] + Controller[AI Gateway Controller] + EGController[Envoy Gateway Controller] + + Controller --> |GWAPI/EG resources|K8s + K8s -->|AI Gateway resources| Controller + K8s -->|GWAPI/EG resources| EGController + Controller <-->|xDS fine-tune| EGController + + style Controller fill:#f9f,stroke:#333,stroke-width:2px + style EGController fill:#ddf,stroke:#333 + end + + subgraph "Data Plane" + Envoy[Envoy Proxy] + ExtProc[External Processor] + RateLimit[Rate Limit Service] + + Envoy --> RateLimit + Envoy --> ExtProc + + style Envoy fill:#f9f,stroke:#333,stroke-width:2px + style ExtProc fill:#f9f,stroke:#333,stroke-width:2px + style RateLimit fill:#dfd,stroke:#333 + end + + Controller -.->|Configures| ExtProc + EGController -.->|Configures| RateLimit + EGController -.->|Updates Config| Envoy +``` + +For detailed information about specific components: + +- Learn more about the [Control Plane](./control-plane.md) components +- Understand the [Data Plane](./data-plane.md) in detail diff --git a/site/versioned_docs/version-1.2/concepts/index.md b/site/versioned_docs/version-1.2/concepts/index.md new file mode 100644 index 0000000000..fbb68c7a63 --- /dev/null +++ b/site/versioned_docs/version-1.2/concepts/index.md @@ -0,0 +1,19 @@ +--- +id: concepts +title: Concepts +sidebar_position: 4 +--- + +# Concepts + +Welcome to the Concepts section of Agent Router documentation. This section will help you understand the core components and how they work together to manage AI traffic effectively. + +## Architecture + +- [System Architecture Overview](./architecture/system-architecture.md) - High-level overview of control and data planes +- [Control Plane](./architecture/control-plane.md) - How configuration and management works +- [Data Plane](./architecture/data-plane.md) - Request processing and traffic flow + +## Resources + +- [Resources Overview](./resources.md) - Understanding the AI Gateway resources and their relationships diff --git a/site/versioned_docs/version-1.2/concepts/resources.md b/site/versioned_docs/version-1.2/concepts/resources.md new file mode 100644 index 0000000000..3320e7de1d --- /dev/null +++ b/site/versioned_docs/version-1.2/concepts/resources.md @@ -0,0 +1,55 @@ +--- +id: resources +title: Resources +sidebar_position: 2 +--- + +# Resources + +The Agent Router uses several custom resources to manage AI traffic. Here's an overview of the key resources and how they relate to each other: + +## Resource Reference + +| Resource | Purpose | API Reference | +| --------------------- | ---------------------------------------------------- | ------------------------------------------------------------- | +| AIGatewayRoute | Defines unified API and routing rules for AI traffic | [AIGatewayRoute](../api/api.mdx#aigatewayroute) | +| AIServiceBackend | Represents individual AI service backends | [AIServiceBackend](../api/api.mdx#aiservicebackend) | +| BackendSecurityPolicy | Configures authentication for backend access | [BackendSecurityPolicy](../api/api.mdx#backendsecuritypolicy) | + +## Core Resources + +### AIGatewayRoute + +A resource that defines a unified AI API for a Gateway, allowing clients to interact with multiple AI backends using a single schema. + +- Specifies the input API schema for client requests +- Contains routing rules to direct traffic to appropriate backends +- Manages request/response transformations between different API schemas +- Can track LLM request costs (like token usage) + +### AIServiceBackend + +Represents a single AI service backend that handles traffic with a specific API schema. + +- Defines the output API schema the backend expects +- References a Kubernetes Service or Envoy Gateway Backend +- Can reference a BackendSecurityPolicy for authentication + +### BackendSecurityPolicy + +Configures authentication and authorization rules for backend access. + +- API Key authentication +- AWS credentials authentication + +## Resource Relationships + +```mermaid +graph TD + A[AIGatewayRoute] -->|references| B[AIServiceBackend] + B -->|references| C[K8s Service/Backend] + B -->|references| D[BackendSecurityPolicy] + D -->|contains| E[API Key/AWS Credentials] +``` + +The AIGatewayRoute acts as the entry point, defining how client requests are processed and routed to one or more AIServiceBackends. Each AIServiceBackend can reference a BackendSecurityPolicy, which provides the necessary credentials for accessing the underlying AI service. diff --git a/site/versioned_docs/version-1.2/getting-started/basic-usage.md b/site/versioned_docs/version-1.2/getting-started/basic-usage.md new file mode 100644 index 0000000000..90fff547f9 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/basic-usage.md @@ -0,0 +1,302 @@ +--- +id: basic-usage +title: Basic Usage +sidebar_position: 4 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; +import vars from '../\_vars.json'; + +This guide will help you set up a basic AI Gateway configuration and make your first request. + +For Windows users, note that you are able to use Windows Subsystem for Linux (WSL) to run the commands below if they do not work on the Windows command prompt. + +## Setting Up Your Environment + +### Deploy Basic Configuration + +Let's start by deploying a basic AI Gateway setup that includes a test backend: + + +{`kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/basic.yaml`} + + +Wait for the Gateway pod to be ready: + +```shell +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system \ + --for=condition=Ready +``` + +:::tip + +In the `basic.yaml` file, you will find "ClientTrafficPolicy" resource that configures the buffer limit on the client to the gateway connection. +By default, Envoy Gateway configures 32KB as the buffer limit, which is not enough for most AI model responses, hence it's configured to 50MB in the example. +Envoy Gateway provides more advanced policies, not limited to the buffer limit. Please refer to the [Client Traffic Policies](https://gateway.envoyproxy.io/latest/tasks/traffic/client-traffic-policy/) documentation for more details. + +**HTTP/2 window size settings:** If you are sending requests over HTTP/2, Envoy imposes additional +flow-control limits via `initialStreamWindowSize` and `initialConnectionWindowSize`. +These act as a secondary cap on request body size, independent of `bufferLimit`, and can cause +`413 request_payload_too_large` errors even after the buffer limit is raised. +To avoid this, set both values to the Envoy defaults (`16Mi` and `24Mi`) in the `ClientTrafficPolicy`: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: ClientTrafficPolicy +metadata: + name: client-buffer-limit + namespace: default +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: envoy-ai-gateway-basic + connection: + bufferLimit: 50Mi + http2: + initialStreamWindowSize: 16Mi + initialConnectionWindowSize: 24Mi +``` + +See the Envoy API reference for +[`initial_stream_window_size`](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-http2protocoloptions-initial-stream-window-size) +and +[`initial_connection_window_size`](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-http2protocoloptions-initial-connection-window-size) +for more details. + +::: + +## Configure `$GATEWAY_URL` + +First, check if your Gateway has an external IP address assigned: + +```shell +kubectl get svc -n envoy-gateway-system \ + --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic +``` + +You'll see output similar to this: + +``` +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) +eg-envoy-ai-gateway LoadBalancer 10.96.61.234 80:31234/TCP +``` + +Choose one of these options based on the EXTERNAL-IP status: + + + + +If the EXTERNAL-IP shows an actual IP address (not ``), you can access the gateway directly: + +First, save the external IP and set the gateway URL: + +```shell +export GATEWAY_URL=$(kubectl get gateway/envoy-ai-gateway-basic -o jsonpath='{.status.addresses[0].value}') +``` + +Verify the URL is available: + +```shell +echo $GATEWAY_URL +``` + + + + +If the EXTERNAL-IP shows `` or your cluster doesn't support LoadBalancer services, use port forwarding. + +First, set the gateway URL: + +```shell +export GATEWAY_URL="http://localhost:8080" +``` + +Then set up port forwarding (this will block the terminal): + +```shell +export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system \ + --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -o jsonpath='{.items[0].metadata.name}') + +kubectl port-forward -n envoy-gateway-system svc/$ENVOY_SERVICE 8080:80 +``` + + + + +## Testing the Gateway + +### Making Test Requests + +Open a new terminal and send test requests to the AI Gateway using the `GATEWAY_URL` we set up: + +#### Test Chat Completions + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "some-cool-self-hosted-model", + "messages": [ + { + "role": "system", + "content": "Hi." + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +#### Test Completions (Legacy) + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "some-cool-self-hosted-model", + "prompt": "def fib(n):\n if n <= 1:\n return n\n else:\n return fib(n-1) + fib(n-2)", + "max_tokens": 25, + "temperature": 0.4, + "top_p": 0.9 + }' \ + $GATEWAY_URL/v1/completions +``` + +#### Test Embeddings + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "some-cool-self-hosted-model", + "input": "Hello, world!" + }' \ + $GATEWAY_URL/v1/embeddings +``` + +#### Test Rerank + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "some-cool-self-hosted-model", + "query": "What is the capital of France?", + "documents": [ + "Paris is the capital of France.", + "Berlin is the capital of Germany." + ] + }' \ + $GATEWAY_URL/cohere/v2/rerank +``` + +:::tip + +If you're opening a new terminal, you'll need to set the `GATEWAY_URL` variable again. + +::: + +### Expected Responses + +#### Chat Completions Response + +You should receive a response like: + +```json +{ + "choices": [ + { + "message": { + "content": "I'll be back." + } + } + ] +} +``` + +#### Completions Response + +For legacy completions requests, you should receive a response like: + +```json +{ + "choices": [ + { + "text": " + fib(n-3) + fib(n-4)\n\ndef fib(n):\n if n <= 1:\n return", + "index": 0, + "finish_reason": "length" + } + ] +} +``` + +#### Embeddings Response + +For embeddings requests, you should receive a response like: + +```json +{ + "object": "list", + "data": [ + { + "object": "embedding", + "embedding": [0.1, 0.2, 0.3, 0.4, 0.5], + "index": 0 + } + ], + "model": "some-cool-self-hosted-model", + "usage": { + "prompt_tokens": 3, + "total_tokens": 3 + } +} +``` + +#### Rerank Response + +You should receive a response like: + +```json +{ + "results": [ + { "index": 0, "relevance_score": 0.98 }, + { "index": 1, "relevance_score": 0.12 } + ] +} +``` + +:::note + +These responses come from a mock backend. The model `some-cool-self-hosted-model` is configured to return test responses. +For real AI model responses, see the [Connect Providers](./connect-providers) guide. + +::: + +### Understanding the Response Formats + +The basic setup includes a mock backend that demonstrates the API structure but doesn't provide real AI responses. The response formats follow the standard OpenAI API formats: + +**Chat Completions:** + +- A `choices` array containing responses +- Each message having a `role` and `content` + +**Embeddings:** + +- A `data` array containing embedding objects +- Each embedding having a vector array and index +- Usage information showing token consumption + +**Rerank:** + +- A `results` array containing ranked results for the provided documents +- Each result includes the original document `index` and a `relevance_score` +- Higher `relevance_score` indicates higher relevance to the query + +## Next Steps + +Now that you've tested the basic setup, you can: + +- Configure [real AI model backends](./connect-providers) like OpenAI or AWS Bedrock +- Explore the [API Reference](../api/) for more details about available endpoints diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/anthropic.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/anthropic.md new file mode 100644 index 0000000000..7e92344926 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/anthropic.md @@ -0,0 +1,149 @@ +--- +id: anthropic +title: Connect Anthropic +sidebar_position: 6 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +# Connect Anthropic + +This guide will help you configure Agent Router to work with Anthropic's models. + +## Prerequisites + +Before you begin, you'll need: + +- An Anthropic API key from [Anthropic's console](https://console.anthropic.com) +- Basic setup completed from the [Basic Usage](../basic-usage.md) guide +- Basic configuration removed as described in the [Advanced Configuration](./index.md) overview + +## Configuration Steps + +:::info Ready to proceed? +Ensure you have followed the steps in [Connect Providers](../connect-providers/) +::: + +### 1. Download configuration template + + +{`curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/anthropic.yaml`} + + +### 2. Configure Anthropic Credentials + +Edit the `anthropic.yaml` file to replace the Anthropic placeholder value: + +- Find the section containing `ANTHROPIC_API_KEY` +- Replace it with your actual Anthropic API key + +:::caution Security Note +Make sure to keep your API key secure and never commit it to version control. +The key will be stored in a Kubernetes secret. +::: + +### 3. Apply Configuration + +Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, +then the secret credential update will be picked up automatically in a few seconds. + +```shell +kubectl apply -f anthropic.yaml + +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system \ + --for=condition=Ready +``` + +### 4. Test the Configuration + +You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. +See the [Basic Usage](../basic-usage.md) page for instructions. + +#### Test Chat Completions + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "claude-sonnet-4-5", + "messages": [ + { + "role": "user", + "content": "Hi." + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +## Troubleshooting + +If you encounter issues: + +1. Verify your API key is correct and active + +2. Check pod status: + + ```shell + kubectl get pods -n envoy-gateway-system + ``` + +3. View controller logs: + + ```shell + kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller + ``` + +4. View External Processor Logs + + ```shell + kubectl logs -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic -c ai-gateway-extproc + ``` + +5. Common errors: + - 401: Invalid API key + - 429: Rate limit exceeded + - 503: Anthropic service unavailable + +## Configuring More Models + +To use more models, add more [AIGatewayRouteRule]s to the `anthropic.yaml` file with the model name in the `value` field. + +For example, let's add [claude-haiku-3-5](https://docs.anthropic.com/en/docs/about-claude/models#claude-3.5-haiku) as a chat completion model: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1alpha1 +kind: AIGatewayRoute +metadata: + name: envoy-ai-gateway-basic-anthropic + namespace: default +spec: + parentRefs: + - name: envoy-ai-gateway-basic + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: claude-sonnet-4-5 + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: claude-haiku-3-5 + backendRefs: + - name: envoy-ai-gateway-basic-anthropic +``` + +## Next Steps + +After configuring Anthropic: + +- [Connect OpenAI](./openai.md) to add another provider +- [Connect AWS Bedrock](./aws-bedrock.md) to add another provider + +[AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/aws-bedrock.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/aws-bedrock.md new file mode 100644 index 0000000000..9874dc365f --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/aws-bedrock.md @@ -0,0 +1,385 @@ +--- +id: aws-bedrock +title: Connect AWS Bedrock +sidebar_position: 3 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +# Connect AWS Bedrock + +This guide will help you configure Agent Router to work with AWS Bedrock's foundation models, including Llama, Anthropic Claude, and other models available on AWS Bedrock. + +## Prerequisites + +- AWS credentials with access to Bedrock +- Basic setup completed from the [Basic Usage](../basic-usage.md) guide +- Basic configuration removed as described in the [Advanced Configuration](./index.md) overview +- Enabled model access to "Llama 3.2 1B Instruct" in the `us-east-1` region (see [AWS Bedrock Model Access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html)) + +## Authentication Methods + +Agent Router supports the AWS SDK default credential chain, which includes: + +1. **EKS Pod Identity** - Recommended for production on EKS (v1.24+) +2. **IRSA (IAM Roles for Service Accounts)** - Recommended for production on EKS (all versions) +3. **Static Credentials** - For development, testing, or non-EKS environments + +## Setup Instructions + +### Option 1: EKS Pod Identity (Recommended for EKS 1.24+) + +EKS Pod Identity is the simplest way to grant AWS permissions to your pods without managing static credentials. + +**Step 1: Set up AWS IAM** + +Follow the official AWS documentation to configure EKS Pod Identity: + +- [EKS Pod Identity Agent installation](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html) +- [Create IAM role and policy](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html) + +Your IAM policy needs these permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream", + "bedrock:ListFoundationModels", + "aws-marketplace:ViewSubscriptions" + ], + "Resource": "*" + } + ] +} +``` + +The Pod Identity association should reference: + +- **Namespace**: `envoy-gateway-system` +- **ServiceAccount**: `ai-gateway-dataplane-aws` + +**Step 2: Apply AI Gateway configuration** + + +{`kubectl apply -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/aws-pod-identity.yaml + +kubectl wait pods --timeout=2m \\ +-l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \\ +-n envoy-gateway-system --for=condition=Ready`} + + +**Step 3: Test** + +```shell +export GATEWAY_URL=$(kubectl get gateway envoy-ai-gateway-basic -n default -o jsonpath='{.status.addresses[0].value}') + +curl -H "Content-Type: application/json" -d '{ + "model": "us.meta.llama3-2-1b-instruct-v1:0", + "messages": [{"role": "user", "content": "Hello!"}] +}' http://$GATEWAY_URL/v1/chat/completions +``` + +--- + +### Option 2: IRSA (IAM Roles for Service Accounts) + +IRSA works on all EKS versions and uses OIDC federation for authentication. + +**Step 1: Set up AWS IAM** + +Follow the official AWS documentation to configure IRSA: + +- [Enable OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) +- [Create IAM role with trust policy](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) + +Your IAM policy needs the same Bedrock permissions as Pod Identity above. + +The trust policy should allow the ServiceAccount `system:serviceaccount:envoy-gateway-system:ai-gateway-dataplane-aws`. + +**Step 2: Download and configure** + + +{`curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/aws-irsa.yaml`} + + +Edit `aws-irsa.yaml` and replace `ACCOUNT_ID` with your AWS account ID in the ServiceAccount annotation: + +```yaml +eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/AIGatewayBedrockRole +``` + +**Step 3: Apply and test** + +```shell +kubectl apply -f aws-irsa.yaml + +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system --for=condition=Ready + +# Test (use same curl command as Pod Identity above) +export GATEWAY_URL=$(kubectl get gateway envoy-ai-gateway-basic -n default -o jsonpath='{.status.addresses[0].value}') +curl -H "Content-Type: application/json" -d '{ + "model": "us.meta.llama3-2-1b-instruct-v1:0", + "messages": [{"role": "user", "content": "Hello!"}] +}' http://$GATEWAY_URL/v1/chat/completions +``` + +--- + +### Option 3: Static Credentials + +Use AWS access key ID and secret for authentication. Suitable for development, testing, or non-EKS environments. + +:::caution Production Warning +Static credentials are not recommended for production. Use EKS Pod Identity or IRSA instead. +::: + +**Step 1: Download and configure** + +```shell +curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/{vars.aigwGitRef}/examples/basic/aws.yaml +``` + +Edit `aws.yaml` and replace the credential placeholders: + +- `AWS_ACCESS_KEY_ID`: Your AWS access key ID +- `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key + +**Step 2: Apply and test** + +```shell +kubectl apply -f aws.yaml + +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system --for=condition=Ready + +# Test (use same curl command as above) +export GATEWAY_URL=$(kubectl get gateway envoy-ai-gateway-basic -n default -o jsonpath='{.status.addresses[0].value}') +curl -H "Content-Type: application/json" -d '{ + "model": "us.meta.llama3-2-1b-instruct-v1:0", + "messages": [{"role": "user", "content": "Hello!"}] +}' http://$GATEWAY_URL/v1/chat/completions +``` + +You can also access an Anthropic model with native Anthropic messages endpoint: + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", + "messages": [ + { + "role": "user", + "content": "What is capital of France?" + } + ], + "max_tokens": 100 + }' \ + $GATEWAY_URL/anthropic/v1/messages +``` + +Expected output: + +```json +{ + "id": "msg_01XFDUDYJgAACzvnptvVoYEL", + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "The capital of France is Paris." + } + ], + "model": "claude-3-5-sonnet-20241022", + "stop_reason": "end_turn", + "usage": { + "input_tokens": 13, + "output_tokens": 8 + } +} +``` + +## Troubleshooting + +If you encounter issues: + +1. **Verify authentication is configured correctly** + - For **EKS Pod Identity**: Check Pod Identity association exists + ```shell + aws eks list-pod-identity-associations --cluster-name YOUR_CLUSTER_NAME + ``` + - For **IRSA**: Check ServiceAccount annotation + ```shell + kubectl get sa ai-gateway-dataplane-aws -n envoy-gateway-system -o yaml + ``` + - For **Static Credentials**: Verify secret exists + ```shell + kubectl get secret -n default + ``` + +2. **Check pod logs** + + ```shell + POD_NAME=$(kubectl get pod -n envoy-gateway-system \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -o jsonpath='{.items[0].metadata.name}') + kubectl logs -n envoy-gateway-system $POD_NAME -c extproc + ``` + +3. **Common errors** + - **401/403**: Invalid credentials or insufficient IAM permissions + - **404**: Model not found or not available in the specified region + - **AssumeRole errors**: Check IAM role trust policy + +For more details on AWS authentication, see: + +- [AWS SDK credential chain](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html) +- [EKS Pod Identity documentation](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) +- [IRSA documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) + +## Configuring More Models + +To use additional models, add more [AIGatewayRouteRule]s to your configuration with the [model ID] in the `value` field. For example, to use [Claude 3 Sonnet]: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: envoy-ai-gateway-basic-aws + namespace: default +spec: + parentRefs: + - name: envoy-ai-gateway-basic + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: anthropic.claude-3-sonnet-20240229-v1:0 + backendRefs: + - name: envoy-ai-gateway-basic-aws +``` + +## Using Anthropic Native API + +When using Anthropic models on AWS Bedrock, you have two options: + +1. **OpenAI-compatible format** (`/v1/chat/completions`) - Works with most models but may not support all Anthropic-specific features. Note that `max_tokens` is required by the Anthropic API — if your clients do not include it, you must inject it via a request body mutation. +2. **Native Anthropic API** (`/anthropic/v1/messages`) - Provides full access to Anthropic-specific features (only for Anthropic models) + +### Streaming with Native Anthropic API + +The native Anthropic API also supports streaming responses: + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", + "messages": [ + { + "role": "user", + "content": "Count from 1 to 5." + } + ], + "max_tokens": 100, + "stream": true + }' \ + $GATEWAY_URL/anthropic/v1/messages +``` + +## Advanced Features with Anthropic Models + +Since the gateway supports the native Anthropic API, you have full access to Anthropic-specific features: + +### Extended Thinking + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", + "messages": [ + { + "role": "user", + "content": "Solve this puzzle: A farmer needs to cross a river with a fox, chicken, and bag of grain. The boat can only hold the farmer and one item. How does the farmer get everything across safely?" + } + ], + "max_tokens": 1000, + "thinking": { + "type": "enabled", + "budget_tokens": 5000 + } + }' \ + $GATEWAY_URL/anthropic/v1/messages +``` + +### Prompt Caching + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", + "system": [ + { + "type": "text", + "text": "You are an AI assistant specialized in Python programming. You help users write clean, efficient Python code.", + "cache_control": {"type": "ephemeral"} + } + ], + "messages": [ + { + "role": "user", + "content": "Write a function to calculate fibonacci numbers." + } + ], + "max_tokens": 500 + }' \ + $GATEWAY_URL/anthropic/v1/messages +``` + +### Tool Use (Function Calling) + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "anthropic.claude-3-5-sonnet-20241022-v2:0", + "messages": [ + { + "role": "user", + "content": "What is the weather in San Francisco?" + } + ], + "max_tokens": 500, + "tools": [ + { + "name": "get_weather", + "description": "Get the current weather in a given location", + "input_schema": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA" + } + }, + "required": ["location"] + } + } + ] + }' \ + $GATEWAY_URL/anthropic/v1/messages +``` + +[AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule +[model ID]: https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html +[Claude 3 Sonnet]: https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/azure-openai.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/azure-openai.md new file mode 100644 index 0000000000..01b49d2ec6 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/azure-openai.md @@ -0,0 +1,138 @@ +--- +id: azure-openai +title: Connect Azure OpenAI +sidebar_position: 3 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +# Connect Azure OpenAI + +This guide will help you configure Agent Router to work with Azure OpenAI's foundation models. + +There are two ways to do the [Azure OpenAI authentication](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#authentication): Microsoft Entra ID and API Key. + +We will use Microsoft Entra ID to authenticate an application to use the Azure OpenAI service. You can obtain an access token using the OAuth 2.0 client credentials grant flow. This process involves registering the application in Microsoft Entra ID (formerly Azure Active Directory), configuring the appropriate permissions, and acquiring a token from the Microsoft identity platform. The access token is then used as proof of authorization in API requests to the Azure OpenAI endpoint. + +For detailed steps, refer to the official [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow#get-a-token). + +API Key authentication is not supported yet. + +## Prerequisites + +Before you begin, you'll need: + +- Azure credentials with access to OpenAI service. +- Basic setup completed from the [Basic Usage](../basic-usage.md) guide +- Basic configuration removed as described in the [Advanced Configuration](./index.md) overview + +## Azure Credential Setup + +1. An Azure account with OpenAI service access enabled +2. Your Azure tenant ID, client ID, and client secret key +3. Enabled model access to "GPT-4o" + +## Configuration Steps + +### 1. Download configuration template + + +{`curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/azure_openai.yaml`} + + +### 2. Configure Azure Credentials + +Edit the `azure_openai.yaml` file to replace these placeholder values: + +- `AZURE_TENANT_ID`: Your Azure tenant ID +- `AZURE_CLIENT_ID`: Your Azure client ID +- `AZURE_CLIENT_SECRET`: Your Azure client secret + +:::caution Security Note +Keep your Azure credentials secure and never commit them to version control. +The credentials will be stored in Kubernetes secrets. +::: + +### 3. Apply Configuration + +Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, then the secret credential update will be picked up automatically in a few seconds. + +```shell +kubectl apply -f azure_openai.yaml + +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system \ + --for=condition=Ready +``` + +### 4. Test the Configuration + +You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. +See the [Basic Usage](../basic-usage.md) page for instructions. + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4o", + "messages": [ + { + "role": "user", + "content": "Hi." + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +## Troubleshoot + +If you encounter issues: + +1. Verify your Azure credentials are correct and active +2. Check pod status + +```shell +kubectl get pods -n envoy-gateway-system +``` + +3. View controller logs: + +```shell +kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller +``` + +4. Common errors: + - 401/403: Invalid credentials or insufficient permissions + - 404: Model not found or not available in region + - 429: Rate limit exceeded + +## Configuring More Models + +To use more models, add more [AIGatewayRouteRule]s to the `azure_openai.yaml` file with the [model ID] in the `value` field. For example, to use [GPT-4.5 Preview] + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: envoy-ai-gateway-basic-azure + namespace: default +spec: + parentRefs: + - name: envoy-ai-gateway-basic + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gpt-4.5-preview + backendRefs: + - name: envoy-ai-gateway-basic-azure +``` + +[AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule +[model ID]: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models +[GPT-4.5 Preview]: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=global-standard%2Cstandard-chat-completions#gpt-45-preview diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/bitdeer.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/bitdeer.md new file mode 100644 index 0000000000..606925cf2d --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/bitdeer.md @@ -0,0 +1,163 @@ +--- +id: bitdeer +title: Connect Bitdeer AI +sidebar_position: 7 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +# Connect Bitdeer AI + +This guide will help you configure Agent Router to work with Bitdeer AI's serverless models through its OpenAI-compatible API. + +## Prerequisites + +Before you begin, you'll need: + +- An API key from the [Bitdeer AI Console](https://www.bitdeer.ai/model/apikeys) +- Basic setup completed from the [Basic Usage](../basic-usage.md) guide +- Basic configuration removed as described in the [Advanced Configuration](./index.md) overview + +## Configuration Steps + +:::info +Ready to proceed? Ensure you have followed the steps in [Connect Providers](./index.md). +::: + +### 1. Download configuration template + +{`curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/bitdeer.yaml`} + +### 2. Configure Bitdeer AI Credentials + +Edit the `bitdeer.yaml` file to replace the Bitdeer placeholder value: + +- Find the section containing `BITDEER_API_KEY` +- Replace it with your actual Bitdeer AI API key + +:::caution Security Note +Make sure to keep your API key secure and never commit it to version control. The key will be stored in a Kubernetes secret. +::: + +### 3. Apply Configuration + +Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, the secret credential update will be picked up automatically in a few seconds. + +```shell +kubectl apply -f bitdeer.yaml +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system \ + --for=condition=Ready +``` + +The template sends requests to Bitdeer AI's OpenAI-compatible endpoint at `https://api-inference.bitdeer.ai/v1`. + +### 4. Test the Configuration + +You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. See the [Basic Usage](../basic-usage.md) page for instructions. + +#### Test Chat Completions + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ai/DeepSeek-V4-Pro", + "messages": [ + { "role": "user", "content": "Explain serverless inference in one sentence." } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +#### Test Streaming + +```shell +curl -N -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ai/DeepSeek-V4-Pro", + "messages": [ + { "role": "user", "content": "Write a short poem about inference." } + ], + "stream": true + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +## Supported Models + +The template configures `deepseek-ai/DeepSeek-V4-Pro`. Other model IDs include `Qwen/Qwen3.5-397B-A17B` and `moonshotai/Kimi-K2.6`. + +See the [Bitdeer AI Model Catalog](https://developers.bitdeer.ai/docs/api/models-catalog) for the current list and each model's supported capabilities. + +## Troubleshooting + +If you encounter issues: + +- Verify your API key is correct and active +- Confirm the requested model ID appears in the Bitdeer AI Model Catalog +- Check pod status: + + ```shell + kubectl get pods -n envoy-gateway-system + ``` + +- View controller logs: + + ```shell + kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller + ``` + +- View External Processor logs: + + ```shell + kubectl logs -n envoy-gateway-system \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -c ai-gateway-extproc + ``` + +Common errors: + +- `401`: Invalid API key +- `429`: Rate limit exceeded +- `503`: Bitdeer AI service unavailable + +## Configuring More Models + +To use more models, add more matching rules to the `AIGatewayRoute` in `bitdeer.yaml`. Set each rule's `x-ai-eg-model` header value to a model ID from the [Bitdeer AI Model Catalog](https://developers.bitdeer.ai/docs/api/models-catalog). + +For example, add Qwen and Kimi models alongside the default DeepSeek model: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: envoy-ai-gateway-basic-bitdeer + namespace: default +spec: + parentRefs: + - name: envoy-ai-gateway-basic + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: deepseek-ai/DeepSeek-V4-Pro + - headers: + - type: Exact + name: x-ai-eg-model + value: Qwen/Qwen3.5-397B-A17B + - headers: + - type: Exact + name: x-ai-eg-model + value: moonshotai/Kimi-K2.6 + backendRefs: + - name: envoy-ai-gateway-basic-bitdeer +``` + +## Next Steps + +After configuring Bitdeer AI, return to [Connect Providers](./index.md) to add another provider. diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/gcp-vertexai.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/gcp-vertexai.md new file mode 100644 index 0000000000..026600f8c5 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/gcp-vertexai.md @@ -0,0 +1,220 @@ +--- +id: gcp-vertexai +title: Connect GCP VertexAI +sidebar_position: 3 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +# Connect GCP VertexAI + +This guide will help you configure Agent Router to work with GCP VertexAI's Gemini and Anthropic models. + +## Prerequisites + +Before you begin, you'll need: + +- GCP credentials with access to GCP VertexAI +- Basic setup completed from the [Basic Usage](../basic-usage.md) guide +- Basic configuration removed as described in the [Advanced Configuration](./index.md) overview + +## Authentication Options + +Agent Router supports three authentication methods for GCP VertexAI: + +1. **Application Default Credentials (ADC)** - Recommended for GKE with Workload Identity +2. **Service Account Key Files** - For explicit JSON credentials +3. **Workload Identity Federation** - For cross-cloud authentication + +### Option 1: Application Default Credentials (Recommended for GKE) + +When running on GKE with Workload Identity, configure only `projectName` and `region`: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: BackendSecurityPolicy +metadata: + name: gcp-credentials +spec: + targetRefs: + - group: aigateway.envoyproxy.io + kind: AIServiceBackend + name: your-backend + type: GCPCredentials + gcpCredentials: + projectName: YOUR_PROJECT_NAME + region: us-central1 +``` + +ADC automatically handles credential rotation via GKE Workload Identity. + +### Option 2: Service Account Key Files + +For non-GKE environments, use a service account key file: + +1. Your GCP project id and name. +2. In your GCP project, enable VertexAI API access. +3. Create a GCP service account and generate the JSON key file. + +:::caution Security Note +Service account key files should be avoided in production when possible. Use ADC/Workload Identity instead. +::: + +## Configuration Steps + +### 1. Download configuration template + + +{`curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/gcp_vertex.yaml`} + + +### 2. Configure GCP Credentials + +Edit the `gcp_vertex.yaml` file to replace these placeholder values: + +- `GCP_PROJECT_NAME`: Your GCP project name +- `GCP_REGION`: GCP region +- Update the generated service account key JSON string in the secret + +:::caution Security Note +Make sure to keep your GCP service account credentials secure and never commit them to version control. +The credentials will be stored in Kubernetes secrets. +::: + +### 3. Apply Configuration + +Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, +then the secret credential update will be picked up automatically in a few seconds. + +```shell +kubectl apply -f gcp_vertex.yaml + +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system \ + --for=condition=Ready +``` + +### 4. Test the Configuration + +You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. +See the [Basic Usage](../basic-usage.md) page for instructions. + +To access a Gemini model with chat completion endpoint: + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gemini-2.5-flash", + "messages": [ + { + "role": "user", + "content": "Hi." + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +To access an Anthropic model with chat completion endpoint: + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "claude-3-7-sonnet@20250219", + "messages": [ + { + "role": "user", + "content": "What is capital of France?" + } + ], + "max_completion_tokens": 100 + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +Expected output: + +```json +{ + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "message": { + "content": "The capital of France is Paris. Paris is not only the capital city but also the largest city in France, known for its cultural significance, historic landmarks like the Eiffel Tower and the Louvre Museum, and its influence in fashion, art, and cuisine.", + "role": "assistant" + } + } + ], + "object": "chat.completion", + "usage": { "completion_tokens": 58, "prompt_tokens": 13, "total_tokens": 71 } +} +``` + +You can also access an Anthropic model with native Anthropic messages endpoint: + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "claude-3-7-sonnet@20250219", + "messages": [ + { + "role": "user", + "content": "What is capital of France?" + } + ], + "max_tokens": 100 + }' \ + $GATEWAY_URL/anthropic/v1/messages +``` + +## Troubleshooting + +If you encounter issues: + +1. Verify your GCP credentials are correct and active +2. Check pod status: + ```shell + kubectl get pods -n envoy-gateway-system + ``` +3. View controller logs: + ```shell + kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller + ``` +4. Common errors: + - 401/403: Invalid credentials or insufficient permissions + - 404: Model not found or not available in a region + - 429: Rate limit exceeded + +## Configuring More Models + +To use more models, add more [AIGatewayRouteRule]s to the `gcp_vertex.yaml` file with the [model ID] in the `value` field. For example, to use [Claude 3 Sonnet] + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: envoy-ai-gateway-basic-gcp-gemini + namespace: default +spec: + schema: + name: OpenAI + parentRefs: + - name: envoy-ai-gateway-basic + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: gemini-2.5-flash-pro + backendRefs: + - name: envoy-ai-gateway-basic-gcp +``` + +[AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule +[model ID]: https://cloud.google.com/vertex-ai/generative-ai/docs/models +[Anthropic Claude]: https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/index.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/index.md new file mode 100644 index 0000000000..4d3d934941 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/index.md @@ -0,0 +1,51 @@ +--- +id: connect-providers +title: Connect Providers +sidebar_position: 5 +--- + +# Connect Providers + +After setting up the basic AI Gateway with the mock backend, you can configure it to work with real AI model providers. This section will guide you through connecting different AI providers to your gateway. + +## Example Providers + +In this getting started guide you'll find quickstart setups to connect to the following providers: + +- [Tetrate Agent Router Service (TARS)](./tars.md) - Connect to Tetrate Agent Router Service's models +- [Bitdeer AI](./bitdeer.md) - Connect to Bitdeer AI's OpenAI-compatible inference models +- [OpenAI](./openai.md) - Connect to OpenAI's GPT models +- [Anthropic](./anthropic.md) - Connect to Anthropic's Claude models +- [AWS Bedrock](./aws-bedrock.md) - Access AWS Bedrock's suite of foundation models +- [Azure OpenAI](./azure-openai.md) - Access Azure OpenAI's suite of foundation models +- [GCP VertexAI](./gcp-vertexai.md) - Access GCP Gemini and Anthropic models on VertexAI + +:::tip +To learn how to connect to providers see [Connecting to AI Providers](/docs/capabilities/llm-integrations/connect-providers) and you can view all [Supported Providers](/docs/capabilities/llm-integrations/supported-providers). +::: + +## Before You Begin + +Before configuring any provider, complete the [Basic Usage](../basic-usage.md) guide. + +## Security Best Practices + +When configuring AI providers, keep these security considerations in mind: + +- Store credentials securely using Kubernetes secrets +- Never commit API keys or credentials to version control +- Regularly rotate your credentials +- Use the principle of least privilege when setting up access +- Monitor usage and set up appropriate rate limits + +## Next Steps + +Choose your provider to get started: + +- [Connect Tetrate Agent Router Service (TARS)](./tars.md) +- [Connect Bitdeer AI](./bitdeer.md) +- [Connect OpenAI](./openai.md) +- [Connect Anthropic](./anthropic.md) +- [Connect AWS Bedrock](./aws-bedrock.md) +- [Connect Azure OpenAI](./azure-openai.md) +- [Connect GCP VertexAI](./gcp-vertexai.md) diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/openai.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/openai.md new file mode 100644 index 0000000000..523f3d5637 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/openai.md @@ -0,0 +1,166 @@ +--- +id: openai +title: Connect OpenAI +sidebar_position: 2 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +# Connect OpenAI + +This guide will help you configure Agent Router to work with OpenAI's models. + +## Prerequisites + +Before you begin, you'll need: + +- An OpenAI API key from [OpenAI's platform](https://platform.openai.com) +- Basic setup completed from the [Basic Usage](../basic-usage.md) guide +- Basic configuration removed as described in the [Advanced Configuration](./index.md) overview + +## Configuration Steps + +:::info Ready to proceed? +Ensure you have followed the steps in [Connect Providers](../connect-providers/) +::: + +### 1. Download configuration template + + +{`curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/openai.yaml`} + + +### 2. Configure OpenAI Credentials + +Edit the `openai.yaml` file to replace the OpenAI placeholder value: + +- Find the section containing `OPENAI_API_KEY` +- Replace it with your actual OpenAI API key + +:::caution Security Note +Make sure to keep your API key secure and never commit it to version control. +The key will be stored in a Kubernetes secret. +::: + +### 3. Apply Configuration + +Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, +then the secret credential update will be picked up automatically in a few seconds. + +```shell +kubectl apply -f openai.yaml + +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system \ + --for=condition=Ready +``` + +### 4. Test the Configuration + +You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. +See the [Basic Usage](../basic-usage.md) page for instructions. + +#### Test Chat Completions + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4o-mini", + "messages": [ + { + "role": "user", + "content": "Hi." + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +#### Test Completions (Legacy) + +OpenAI supports the legacy completions endpoint with specific models: + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "babbage-002", + "prompt": "def fib(n):\n if n <= 1:\n return n\n else:\n return fib(n-1) + fib(n-2)", + "max_tokens": 25, + "temperature": 0.4, + "top_p": 0.9 + }' \ + $GATEWAY_URL/v1/completions +``` + +## Troubleshooting + +If you encounter issues: + +1. Verify your API key is correct and active + +2. Check pod status: + + ```shell + kubectl get pods -n envoy-gateway-system + ``` + +3. View controller logs: + + ```shell + kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller + ``` + +4. View External Processor Logs + + ```shell + kubectl logs -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic -c ai-gateway-extproc + ``` + +5. Common errors: + - 401: Invalid API key + - 429: Rate limit exceeded + - 503: OpenAI service unavailable + +## Configuring More Models + +To use more models, add more [AIGatewayRouteRule]s to the `openai.yaml` file with the [model alias] in the `value` field. + +For example, let's add [o1] as a chat completion model, and [text-embedding-ada-002](https://platform.openai.com/docs/models/text-embedding-ada-002) as embedding models: + +```yaml +apiVersion: aigateway.envoyproxy.io/v1beta1 +kind: AIGatewayRoute +metadata: + name: envoy-ai-gateway-basic-openai + namespace: default +spec: + parentRefs: + - name: envoy-ai-gateway-basic + kind: Gateway + group: gateway.networking.k8s.io + rules: + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: o1 + - matches: + - headers: + - type: Exact + name: x-ai-eg-model + value: text-embedding-ada-002 + backendRefs: + - name: envoy-ai-gateway-basic-openai +``` + +## Next Steps + +After configuring OpenAI: + +- [Connect AWS Bedrock](./aws-bedrock.md) to add another provider + +[AIGatewayRouteRule]: ../../api/api.mdx#aigatewayrouterule +[model alias]: https://platform.openai.com/docs/models#current-model-aliases +[o1]: https://platform.openai.com/docs/models#o1 diff --git a/site/versioned_docs/version-1.2/getting-started/connect-providers/tars.md b/site/versioned_docs/version-1.2/getting-started/connect-providers/tars.md new file mode 100644 index 0000000000..73cc74c346 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/connect-providers/tars.md @@ -0,0 +1,124 @@ +--- +id: tars +title: Connect Tetrate Agent Router Service (TARS) +sidebar_position: 4 +--- + +import CodeBlock from '@theme/CodeBlock'; +import vars from '../../\_vars.json'; + +# Connect Tetrate Agent Router Service (TARS) + +This guide will help you configure Agent Router to work with Tetrate Agent Router Service (TARS)'s models. + +## Prerequisites + +Before you begin, you'll need: + +- An API key from [Tetrate Agent Router Service's platform](https://router.tetrate.ai/) +- Basic setup completed from the [Basic Usage](../basic-usage.md) guide +- Basic configuration removed as described in the [Advanced Configuration](./index.md) overview + +## Configuration Steps + +:::info Ready to proceed? +Ensure you have followed the steps in [Connect Providers](../connect-providers/) +::: + +### 1. Download configuration template + + +{`curl -O https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/basic/tars.yaml`} + + +### 2. Configure Tetrate Agent Router Service (TARS) Credentials + +Edit the `tars.yaml` file to replace the TARS placeholder value: + +- Find the section containing `TARS_API_KEY` +- Replace it with your actual TARS API key + +:::caution Security Note +Make sure to keep your API key secure and never commit it to version control. +The key will be stored in a Kubernetes secret. +::: + +### 3. Apply Configuration + +Apply the updated configuration and wait for the Gateway pod to be ready. If you already have a Gateway running, +then the secret credential update will be picked up automatically in a few seconds. + +```shell +kubectl apply -f tars.yaml + +kubectl wait pods --timeout=2m \ + -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic \ + -n envoy-gateway-system \ + --for=condition=Ready +``` + +### 4. Test the Configuration + +You should have set `$GATEWAY_URL` as part of the basic setup before connecting to providers. +See the [Basic Usage](../basic-usage.md) page for instructions. + +#### Test Chat Completions + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4o-mini", + "messages": [ + { + "role": "user", + "content": "Hi." + } + ] + }' \ + $GATEWAY_URL/v1/chat/completions +``` + +#### Test Completions (Legacy) + +TARS fully supports the legacy completions endpoint: + +```shell +curl -H "Content-Type: application/json" \ + -d '{ + "model": "babbage-002", + "prompt": "def fib(n):\n if n <= 1:\n return n\n else:\n return fib(n-1) + fib(n-2)", + "max_tokens": 25, + "temperature": 0.4, + "top_p": 0.9 + }' \ + $GATEWAY_URL/v1/completions +``` + +## Troubleshooting + +If you encounter issues: + +1. Verify your API key is correct and active + +2. Check pod status: + + ```shell + kubectl get pods -n envoy-gateway-system + ``` + +3. View controller logs: + + ```shell + kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller + ``` + +4. View External Processor Logs + + ```shell + kubectl logs -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic -c ai-gateway-extproc + ``` + +5. Common errors: + - 401: Invalid API key + - 429: Rate limit exceeded + - 503: TARS service unavailable diff --git a/site/versioned_docs/version-1.2/getting-started/index.md b/site/versioned_docs/version-1.2/getting-started/index.md new file mode 100644 index 0000000000..091e0c8b47 --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/index.md @@ -0,0 +1,69 @@ +--- +id: getting-started +title: Getting Started +sidebar_position: 2 +--- + +# Getting Started with Agent Router + +Agent Router runs two ways. The fastest is one command on your laptop; the same +configuration then ships to Kubernetes for production. + +## Run locally in 60 seconds + +The standalone CLI starts an OpenAI-compatible router on your machine — no +Kubernetes, no Docker required (Linux and macOS): + +```shell +OPENAI_API_KEY=sk-your-key aigw run +``` + +Then point any OpenAI-compatible client or SDK at `http://localhost:1975/v1`: + +```shell +curl http://localhost:1975/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model": "gpt-5", "messages": [{"role": "user", "content": "Say this is a test!"}]}' +``` + +`aigw` auto-configures from the same environment variables as the OpenAI SDK, +and can also front self-hosted models (Ollama, vLLM) and MCP servers. See +[Installation](../cli/installation.md) for how to get the `aigw` binary, and +[aigw run](../cli/run.md) for provider auto-configuration, MCP gateway mode, and +custom configuration files. + +## Deploy on Kubernetes + +For production, Agent Router runs as a control plane on Envoy Gateway in your +Kubernetes cluster. It uses the same configuration API you test locally with +`aigw run`, so what works on your laptop deploys unchanged. This guide walks +through the Kubernetes path: + +1. [Prerequisites](./prerequisites.md) + - Setting up your Kubernetes cluster + - Installing required tools + - Setting up Envoy Gateway + +2. [Installation](./installation.md) + - Installing Agent Router + - Configuring the gateway + - Verifying the installation + +3. [Basic Usage](./basic-usage.md) + - Deploying a basic configuration + - Making your first request + - Understanding the response format + +4. [Connect Providers](./connect-providers) + - Setting up OpenAI integration + - Configuring AWS Bedrock + - Managing credentials securely + +For local Docker or non-Kubernetes deployments, start with the [Agent Router CLI](../cli/) instead. + +## Need Help? + +If you run into any issues: + +- Join our [Community Discord](https://discord.gg/xuxtPq43gZ) +- File an issue on [GitHub](https://github.com/theagentrouter/agent-router/issues) diff --git a/site/versioned_docs/version-1.2/getting-started/installation.md b/site/versioned_docs/version-1.2/getting-started/installation.md new file mode 100644 index 0000000000..3c3c2ed3bf --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/installation.md @@ -0,0 +1,80 @@ +--- +id: installation +title: Installation +sidebar_position: 3 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; +import vars from '../\_vars.json'; + +This guide will walk you through installing Agent Router and its required components. + +## Installing Agent Router + +The easiest way to install Agent Router on Kubernetes is using the Helm charts. You need to install the CRDs first, followed by the AI Gateway controller. + +If you want to run Agent Router as a standalone proxy outside Kubernetes, use [`aigw run`](../cli/run.md). The CLI can run directly on Linux and macOS, or from the Docker Compose examples in [`cmd/aigw`](https://github.com/theagentrouter/agent-router/tree/main/cmd/aigw). Those examples include an Ollama-backed stack and an OpenTelemetry stack for local traces and metrics. + +### Step 1: Install AI Gateway CRDs + +First, install the CRD Helm chart (`ai-gateway-crds-helm`) which manages all Custom Resource Definitions: + + +{`helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \\ + --version v${vars.aigwVersion} \\ + --namespace envoy-ai-gateway-system \\ + --create-namespace`} + + +### Step 2: Install AI Gateway Resources + +After the CRDs are installed, install the AI Gateway Helm chart: + + +{`helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \\ + --version v${vars.aigwVersion} \\ + --namespace envoy-ai-gateway-system \\ + --create-namespace + +kubectl wait --timeout=2m -n envoy-ai-gateway-system deployment/ai-gateway-controller --for=condition=Available`} + + +:::tip +Note that you are browsing the documentation for the main branch version of Agent Router, which is not a stable release. +We highly recommend you replace `v0.0.0-latest` with `v0.0.0-${commit hash of https://github.com/theagentrouter/agent-router}` to pin to a specific version. +Otherwise, the controller will be installed with the latest version at the time of installation, which can be unstable over time due to ongoing development (the latest container tags are overwritten). +::: + +> If you are experiencing network issues with `docker.io`, you can install the helm charts from the code repo [ai-gateway-crds-helm](https://github.com/theagentrouter/agent-router/tree/{vars.aigwGitRef}/manifests/charts/ai-gateway-crds-helm) and [ai-gateway-helm](https://github.com/theagentrouter/agent-router/tree/{vars.aigwGitRef}/manifests/charts/ai-gateway-helm) instead. + +:::tip Verify Installation + +Check the status of the pods. All pods should be in the `Running` state with `Ready` status. + +Check AI Gateway pods: + +```shell +kubectl get pods -n envoy-ai-gateway-system +``` + +::: + +:::note Upgrading from Previous Versions + +If you installed AI Gateway with only `ai-gateway-helm` previously, first install the CRD chart with `--take-ownership` to transfer CRD ownership, then upgrade the main chart: + + +{`helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm --version v${vars.aigwVersion} --namespace envoy-ai-gateway-system --take-ownership +helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm --version v${vars.aigwVersion} --namespace envoy-ai-gateway-system`} + + +::: + +## Next Steps + +After completing the installation: + +- Continue to [Basic Usage](./basic-usage.md) to learn how to make your first request +- Or jump to [Connect Providers](./connect-providers) to set up OpenAI and AWS Bedrock integration diff --git a/site/versioned_docs/version-1.2/getting-started/prerequisites.md b/site/versioned_docs/version-1.2/getting-started/prerequisites.md new file mode 100644 index 0000000000..ec8e165cdc --- /dev/null +++ b/site/versioned_docs/version-1.2/getting-started/prerequisites.md @@ -0,0 +1,196 @@ +--- +id: prerequisites +title: Prerequisites +sidebar_position: 2 +--- + +import Link from '@docusaurus/Link'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; +import vars from '../\_vars.json'; + +Before you begin using Agent Router, you'll need to ensure you have the following prerequisites in place: + +## Required Tools + +Make sure you have the following tools installed: + +- `kubectl` - The Kubernetes command-line tool +- `helm` - The package manager for Kubernetes +- `curl` - For testing API endpoints (installed by default on most systems) + +:::tip Verify Installation +Run these commands to verify your tools are properly installed: + +Verify kubectl installation: + +```shell +kubectl version --client +``` + +Verify helm installation: + +```shell +helm version +``` + +Verify curl installation: + +```shell +curl --version +``` + +::: + +## Kubernetes Cluster + +:::info Version Requirements +Agent Router requires Kubernetes version {vars.k8sMinVersion} or higher. We recommend using a recent stable version of Kubernetes for the best experience. +::: + +You need a running Kubernetes cluster with your kubeconfig properly configured. You have several options: + + + + +If you already have a Kubernetes cluster, ensure your kubeconfig is properly configured to access it. + +Verify your cluster meets the version requirements by running: + +```shell +kubectl version --output=json +``` + +The server version in the output should show version {vars.k8sMinVersion} or higher: + +```json +{ + "serverVersion": { + "major": "1", + "minor": "32+", + ... + } +} +``` + +:::caution + +If your cluster is running a version lower than {vars.k8sMinVersion}, our recommendation is to upgrade your cluster to meet the minimum requirements. +The recommended Kubernetes versions are those that are actively tested and supported by the Agent Router team. + +::: + + + + +If you're using Docker Desktop, you can enable its built-in Kubernetes cluster: + +1. Open Docker Desktop +2. Click on the gear icon (Settings) +3. Select "Kubernetes" from the left sidebar +4. Check "Enable Kubernetes" +5. Click "Apply & Restart" + +Wait for Docker Desktop to restart and for the Kubernetes cluster to be ready. You can verify the setup by running: + +```shell +kubectl config use-context docker-desktop +kubectl cluster-info +``` + +The output should show that the Kubernetes control plane is running. + +:::tip + +Docker Desktop's Kubernetes is a great choice for local development as it: + +- Comes pre-installed with Docker Desktop +- Runs locally on your machine +- Integrates well with Docker Desktop's UI +- Requires minimal setup + +::: + + + + +If you don't have a Kubernetes cluster, you can quickly create a local one using [kind](https://kind.sigs.k8s.io/). + +First, install kind if you haven't already (on macOS with Homebrew): + +```shell +brew install kind +``` + +Then create a cluster: + +```shell +kind create cluster +``` + + + + +## Envoy Gateway + +:::warning Important + +Ensure you're using a clean Envoy Gateway deployment. If you have an existing Envoy Gateway installation with custom configurations, it may conflict with AI Gateway's requirements. We recommend: + +- Using a fresh Kubernetes cluster, or +- Uninstalling any existing Envoy Gateway deployments before proceeding: + ```shell + helm uninstall eg -n envoy-gateway-system + kubectl delete namespace envoy-gateway-system + ``` + +::: + +:::info Version Requirements + +Agent Router requires Envoy Gateway version {vars.egMinVersion} or higher. For the best experience while trying out AI Gateway, we recommend using the latest version as shown in the commands below. + +::: + +Agent Router is built on top of Envoy Gateway. Install it using Helm and wait for the deployment to be ready. + + +{`helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \\ + --version v${vars.egVersion} \\ + --namespace envoy-gateway-system \\ + --create-namespace \\ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/manifests/envoy-gateway-values.yaml + +kubectl wait --timeout=2m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available`} + + +> If you are experiencing network issues with `docker.io` , you can follow the [Install from Source Code](https://github.com/envoyproxy/gateway/tree/main/charts/gateway-helm#install-from-source-code) instructions. + +Note that we have included the AI Gateway-specific helm values file via the `-f` flag. This file contains the necessary configuration for AI Gateway integration. + +### Additional Features (Rate Limiting, InferencePool, etc.) + +Depending on the additional features you want (like rate limiting or InferencePool), you need to pass additional addon values files to modify the Envoy Gateway installation. +Currently, supported addons are: + +- [**Rate Limiting**](../capabilities/traffic/usage-based-ratelimiting.md): + + {`https://github.com/theagentrouter/agent-router/blob/${vars.aigwGitRef}/examples/token_ratelimit/envoy-gateway-values-addon.yaml`} + +- [**InferencePool**](../capabilities/inference/index.md): + + {`https://github.com/theagentrouter/agent-router/blob/${vars.aigwGitRef}/examples/inference-pool/envoy-gateway-values-addon.yaml`} + + +For example, to install with all addons enabled, run: + + +{`helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \\ + --version v${vars.egVersion} \\ + --namespace envoy-gateway-system \\ + --create-namespace \\ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/manifests/envoy-gateway-values.yaml \\ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/token_ratelimit/envoy-gateway-values-addon.yaml \\ + -f https://raw.githubusercontent.com/theagentrouter/agent-router/${vars.aigwGitRef}/examples/inference-pool/envoy-gateway-values-addon.yaml`} + diff --git a/site/versioned_docs/version-1.2/index.md b/site/versioned_docs/version-1.2/index.md new file mode 100644 index 0000000000..7fd3c05e09 --- /dev/null +++ b/site/versioned_docs/version-1.2/index.md @@ -0,0 +1,103 @@ +--- +id: home +title: Home +sidebar_position: 1 +--- + +# Agent Router Overview + +Welcome to the **Agent Router** documentation! This open-source project, built on **Envoy +Proxy**, aims to simplify how application clients interact with **Generative AI (GenAI)** services. +It provides a secure, scalable, and efficient way to manage LLM/AI traffic, with backend rate +limiting and policy control. + +:::info[Formerly Envoy AI Gateway] +Agent Router is the new name for **Envoy AI Gateway**, now an **Agentic AI Foundation** +project. Same code, same maintainers, new home — **no CRD, API, or CLI names have +changed** (`aigw`, `AIGatewayRoute`, and the `aigateway.envoyproxy.io` API group are all +unchanged), and there is nothing to migrate. +::: + +## Is Agent Router for you? + +Agent Router is worth exploring if: + +- Multiple applications or teams share access to models or tools. +- Application teams should not have to integrate with every provider separately. +- Provider credentials should not live inside application workloads. +- You need consistent permissions, quotas, usage attribution, or failover. +- You want to use hosted and self-managed models within a single platform boundary. +- You are exposing MCP servers and need to control what each identity can discover and invoke. + +Agent Router is not an agent framework and does not tell an agent how to reason. Its +value is clearest when AI traffic crosses provider, team, security, cost, or operational +boundaries — when a prototype becomes something the business depends on. + +The fastest way to find out is one command on your laptop: + +```shell +OPENAI_API_KEY=sk-your-key aigw run +``` + +That starts an OpenAI-compatible router at `localhost:1975` — see +[Run locally in 60 seconds](getting-started/index.md#run-locally-in-60-seconds). + +## **Project Overview** + +The project — created as **Envoy AI Gateway** — addresses the complexity of connecting applications to GenAI services by leveraging Envoy's flexibility and Kubernetes-native features. It has evolved through contributions from the Envoy community, fostering a collaborative approach to solving real-world challenges. + +### **Key Objectives** + +- Provide a unified layer for routing and managing LLM/AI traffic. +- Support automatic failover mechanisms to ensure service reliability. +- Ensure end-to-end security, including upstream authorization for LLM/AI traffic. +- Implement a policy framework to support usage limiting use cases. +- Foster an open-source community to address GenAI-specific routing and quality of service needs. + +## **Project Goals** + +The Agent Router project is designed to address the critical challenges of AI/LLM integration in enterprise environments through the following core goals: + +- **Resilient Connectivity Across Providers and Self-Hosted Models**: Create robust, fault-tolerant connections that integrate with LLM providers (such as OpenAI, Anthropic, AWS Bedrock, etc.) and self-hosted models, ensuring high availability through intelligent routing and automatic failover. + +- **Comprehensive Observability for Performance and Cost Management**: Provide visibility into traffic performance, usage patterns, and cost analytics, enabling organizations to optimize their GenAI usage and monitor service quality. + +- **Enterprise-Grade Security Features**: Implement security controls including fine-grained access control, authorization policies, rate limiting, to access services, as well as ensure secure egress connection to external providers from the gateway via [Upstream Authentication](capabilities/security/upstream-auth.mdx). + +- **Extensible Architecture**: Leverage Envoy's proven extensibility framework to enable rapid development of custom features, allowing organizations to adapt the gateway to their specific AI infrastructure needs. + +- **Reliable Foundation with Envoy Proxy**: Build upon Envoy's battle-tested proxy technology to provide a stable, performant, and production-ready foundation that enterprises can rely on for their traffic handling needs. + +These goals guide the development of features and capabilities that make AI/LLM integration secure, scalable, and operationally excellent for enterprise environments. + +Documentation for installation, setup, and contribution guidelines is included to help new users and contributors get started easily. + +## **Community Collaboration** + +[Weekly community meetings][meeting-notes] are held every Monday to discuss updates, address issues, and review contributions. + +## **Architecture Overview** + +Agent Router is the control plane; Envoy is the data plane that carries the traffic. See +the [architecture documentation](concepts/architecture/) for how the pieces fit together. + +## **Get Involved** + +We welcome community contributions! Here's how you can participate: + +- Attend the [weekly community meetings][meeting-notes] to stay updated and share ideas. +- Submit feature requests and pull requests via the GitHub repository. +- Join the conversation on the [community Discord][discord]. + +Refer to [this contributing guide][contributing.md] for detailed instructions on setting up your +environment and contributing. + +--- + +The **Agent Router** addresses the growing demand for secure, scalable, and efficient AI/LLM +traffic management. Your contributions and feedback are key to its success and to advancing the +future of AI service integration. + +[meeting-notes]: https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w +[discord]: https://discord.gg/xuxtPq43gZ +[contributing.md]: https://github.com/theagentrouter/agent-router/blob/main/CONTRIBUTING.md diff --git a/site/versioned_docs/version-1.2/terminology.md b/site/versioned_docs/version-1.2/terminology.md new file mode 100644 index 0000000000..249f4b48ee --- /dev/null +++ b/site/versioned_docs/version-1.2/terminology.md @@ -0,0 +1,160 @@ +--- +id: terminology +title: Terminology +sidebar_position: 6 +--- + +# AI Gateway Glossary + +This glossary provides definitions for key terms and concepts used in AI Gateway and GenAI traffic handling. + +## Quick Reference + +| Term | Quick Definition | +| ---------------- | -------------------------------------------------- | +| GenAI Gateway | Gateway for managing AI model traffic | +| Foundation Model | Base pre-trained AI model | +| Token | Basic unit of text in LLM processing | +| Token Usage | Monitoring and limiting model resource consumption | +| Model Routing | Directing requests to appropriate models | +| Prompt | Input text guiding AI model response | +| Temperature | Control for model output randomness | + +## Categories + +- [AI/ML Fundamentals](#ai-ml-fundamentals): Token, Prompt, Context Window, Temperature +- [Inference Infrastructure](#inference-infrastructure): Inference Instance, Service, Model Provider +- [Gateway Components](#gateway-components): GenAI Gateway, Gateway API Inference Extension +- [Usage & Analytics](#usage--analytics): Usage Monitoring, Token Usage Limiting +- [Model Types & Management](#model-types--management): Foundation Model, Fine-Tuned Model +- [Content & Safety](#content--safety): Content Filtering + +## AI/ML Fundamentals {#ai-ml-fundamentals} + +### Context Window + +The maximum amount of text (in tokens) that a model can process in a single request. + +**Related**: [Token](#token) + +### Prompt + +The input text that guides the AI model's response, including instructions, context, and specific queries. + +### Temperature + +A parameter that controls the randomness/creativity of model outputs, typically ranging from 0 (deterministic) to 1 (more creative). + +### Token + +The basic unit of text processing in LLMs, representing parts of words or characters. + +**Related**: [Context Window](#context-window) · [Token Cost](#token-cost) + +### Token Cost + +The financial or resource cost associated with token usage in model requests. + +**Related**: [Token](#token) · [Rate of LLM Token Consumption](#rate-of-llm-token-consumption) + +## Content & Safety {#content--safety} + +### Content Filtering + +A mechanism to screen and moderate AI-generated content to ensure compliance with ethical standards, company policies, or regulatory requirements. + +## Gateway Components {#gateway-components} + +### Gateway API Inference Extension + +A Kubernetes SIG Network extension for Gateway API that provides specialized routing and load balancing capabilities for AI/ML workloads, handling traffic management at the level of inference instances. + +**Related**: [Inference Instance](#inference-instance) + +### GenAI Gateway + +A specialized gateway solution designed to manage, monitor, and route traffic to Generative AI models. It provides capabilities such as load balancing, authorization, token usage monitoring, and integration with multiple model providers. + +**Related**: [Token](#token) · [Model Provider](#model-provider) + +### Hybrid GenAI Gateway + +A GenAI Gateway configuration that supports both local inference instances and external cloud-based AI models, providing flexibility in deployment and cost management. + +**Related**: [GenAI Gateway](#genai-gateway) · [Inference Instance](#inference-instance) · [Model Provider](#model-provider) + +## Inference Infrastructure {#inference-infrastructure} + +### Inference Instance + +An individual compute resource or container used to run a machine learning model for generating AI outputs (inference). + +### Inference Service + +A service that provides model inference capabilities, including model loading, input processing, inference execution, and output formatting. + +**Related**: [Inference Instance](#inference-instance) + +### Model Endpoint + +The API endpoint provided by a specific AI model, whether hosted by a cloud provider, open-source solution, or private deployment. + +### Model Provider + +Services providing AI model capabilities through APIs, which can be either first-party providers who develop their own models (like OpenAI, Anthropic) or third-party providers who host other companies' models (like AWS Bedrock, Azure OpenAI Service). + +## Model Types & Management {#model-types--management} + +### Fine-Tuned Model + +A version of a base Generative AI model that has been customized for specific tasks or domains using additional training data. + +**Related**: [Foundation Model](#foundation-model) + +### Foundation Model + +Foundation models are large-scale, pre-trained AI models designed to handle a broad range of tasks. They are trained on extensive datasets and can be fine-tuned or adapted to specific use cases. + +**Related**: [Fine-Tuned Model](#fine-tuned-model) + +### Model Routing + +A feature in GenAI Gateways that dynamically routes requests to specific models or model versions based on client configuration, use case requirements, or service level agreements. + +**Related**: [GenAI Gateway](#genai-gateway) + +## Usage & Analytics {#usage--analytics} + +### GenAI Usage Analytics + +The collection and analysis of data regarding how users interact with AI models via the GenAI Gateway, including token usage, request patterns, and latency metrics. + +**Related**: [GenAI Gateway](#genai-gateway) · [Token](#token) + +### GenAI Usage Monitoring + +The tracking of resource consumption across different types of models, including token-based monitoring for LLMs, image resolution and compute resources for LVMs, and combined metrics for multimodal models. + +**Related**: [Token](#token) + +### LLM Token Usage Limiting + +A mechanism to monitor and control the number of tokens processed by an LLM GenAI model, including input, output, and total token limits. + +**Related**: [Token](#token) · [GenAI Gateway](#genai-gateway) + +### Rate of LLM Token Consumption + +The speed at which tokens are consumed by an AI model during processing. This metric is crucial for cost estimation and performance optimization. + +**Related**: [Token](#token) + +:::note +This glossary is continuously evolving as the field of GenAI traffic handling develops. If you'd like to contribute or suggest changes, please visit our [GitHub repository](https://github.com/theagentrouter/agent-router). +::: + +:::tip See Also + +- Check our [Getting Started](./getting-started/index.md) guide for practical examples +- Join our [Community Discord](https://discord.gg/xuxtPq43gZ) for discussions + ::: diff --git a/site/versioned_sidebars/version-1.2-sidebars.json b/site/versioned_sidebars/version-1.2-sidebars.json new file mode 100644 index 0000000000..caea0c03ba --- /dev/null +++ b/site/versioned_sidebars/version-1.2-sidebars.json @@ -0,0 +1,8 @@ +{ + "tutorialSidebar": [ + { + "type": "autogenerated", + "dirName": "." + } + ] +} diff --git a/site/versions.json b/site/versions.json index 7410082773..39194d7b3b 100644 --- a/site/versions.json +++ b/site/versions.json @@ -1,4 +1,5 @@ [ + "1.2", "1.1", "1.0", "0.7", From b1ea1008ea2f41ec391bd38068a03f181711dfc0 Mon Sep 17 00:00:00 2001 From: Ignasi Barrera Date: Tue, 6 Oct 2026 10:58:28 +0200 Subject: [PATCH 4/5] format README Signed-off-by: Ignasi Barrera --- RELEASES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASES.md b/RELEASES.md index e39f7a2e9c..5757f6b8a2 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -77,7 +77,7 @@ Each non-patch release should start with Release Candidate (RC) phase as follows The release candidate should always be cut from the main branch. 2. Prepare the docs site with the new version: - + ``` cd site npm run docusaurus docs:version 0.50 From f848d386166c06e511142b3d03f15da4d867b35e Mon Sep 17 00:00:00 2001 From: Erica Hughberg Date: Tue, 6 Oct 2026 07:52:47 -0400 Subject: [PATCH 5/5] docs: set v1.2 release date to October 6 Co-Authored-By: Claude Opus 5.5 Signed-off-by: Erica Hughberg --- site/src/data/releases/index.json | 2 +- site/src/data/releases/v1.2.json | 2 +- site/src/pages/release-notes/index.mdx | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/site/src/data/releases/index.json b/site/src/data/releases/index.json index 43162afa4e..ce88f2620c 100644 --- a/site/src/data/releases/index.json +++ b/site/src/data/releases/index.json @@ -2,7 +2,7 @@ "metadata": { "title": "Release Notes", "description": "Release Notes for Agent Router", - "lastUpdated": "2026-10-05" + "lastUpdated": "2026-10-06" }, "community": { "discord": "https://discord.gg/xuxtPq43gZ", diff --git a/site/src/data/releases/v1.2.json b/site/src/data/releases/v1.2.json index 36fce6ebc3..401b557d61 100644 --- a/site/src/data/releases/v1.2.json +++ b/site/src/data/releases/v1.2.json @@ -9,7 +9,7 @@ "releases": [ { "version": "v1.2.0", - "date": "October 5, 2026", + "date": "October 6, 2026", "type": "minor", "tags": [ { "text": "Agent Router", "type": "feature" }, diff --git a/site/src/pages/release-notes/index.mdx b/site/src/pages/release-notes/index.mdx index f52d7c4b2f..c32d6b6469 100644 --- a/site/src/pages/release-notes/index.mdx +++ b/site/src/pages/release-notes/index.mdx @@ -49,7 +49,7 @@ Stay up-to-date with the latest improvements, features, and fixes in Agent Route