Skip to content

docs: add Vercel AI Gateway as a supported provider #2728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joemckenney wants to merge 1 commit into
base: main
Choose a base branch
from
+512 −0
Open

docs: add Vercel AI Gateway as a supported provider #2728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ Agent Router supports a wide range of AI providers, making it easy to integrate
<img src="site/static/img/providers/anthropic.svg" width="60" height="60" alt="Anthropic"/>
<br><sub><b>Anthropic</b></sub>
</td>
<td align="center" width="120">
<img src="site/static/img/providers/vercel.svg" width="60" height="60" alt="Vercel AI Gateway"/>
<br><sub><b>Vercel AI Gateway</b></sub>
</td>
</tr>
</table>
</div>
Expand Down
1 change: 1 addition & 0 deletions examples/basic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ traffic for various AI providers.
- `azure_openai.yaml` - Azure OpenAI integration
- `gcp_vertex.yaml` - GCP Vertex AI integration
- `tars.yaml` - TARS integration
- `vercel.yaml` - Vercel AI Gateway integration
- `cohere.yaml` - Cohere integration
- `typesafe.yaml` - TypeSafe AI (Jev) integration

Expand Down
88 changes: 88 additions & 0 deletions examples/basic/vercel.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Copyright Envoy AI Gateway Authors
# SPDX-License-Identifier: Apache-2.0
# The full text of the Apache license is available in the LICENSE file at
# the root of the repo.

apiVersion: aigateway.envoyproxy.io/v1beta1
kind: AIGatewayRoute
metadata:
name: envoy-ai-gateway-basic-vercel
namespace: default
spec:
parentRefs:
- name: envoy-ai-gateway-basic
kind: Gateway
group: gateway.networking.k8s.io
rules:
- matches:
- headers:
- type: RegularExpression
name: x-ai-eg-model
value: .*
backendRefs:
- name: envoy-ai-gateway-basic-vercel
timeouts:
request: 120s
---
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: AIServiceBackend
metadata:
name: envoy-ai-gateway-basic-vercel
namespace: default
spec:
schema:
name: OpenAI
backendRef:
name: envoy-ai-gateway-basic-vercel
kind: Backend
group: gateway.envoyproxy.io
---
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: BackendSecurityPolicy
metadata:
name: envoy-ai-gateway-basic-vercel-apikey
namespace: default
spec:
targetRefs:
- group: aigateway.envoyproxy.io
kind: AIServiceBackend
name: envoy-ai-gateway-basic-vercel
type: APIKey
apiKey:
secretRef:
name: envoy-ai-gateway-basic-vercel-apikey
namespace: default
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: envoy-ai-gateway-basic-vercel
namespace: default
spec:
endpoints:
- fqdn:
hostname: ai-gateway.vercel.sh
port: 443
---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
name: envoy-ai-gateway-basic-vercel-tls
namespace: default
spec:
targetRefs:
- group: "gateway.envoyproxy.io"
kind: Backend
name: envoy-ai-gateway-basic-vercel
validation:
wellKnownCACertificates: "System"
hostname: ai-gateway.vercel.sh
---
apiVersion: v1
kind: Secret
metadata:
name: envoy-ai-gateway-basic-vercel-apikey
namespace: default
type: Opaque
stringData:
apiKey: VERCEL_AI_GATEWAY_API_KEY # Replace with your Vercel AI Gateway API key.
23 changes: 23 additions & 0 deletions internal/autoconfig/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ var (
//go:embed testdata/tars.yaml
tarsYAML string

//go:embed testdata/vercel.yaml
vercelYAML string

//go:embed testdata/openrouter.yaml
openrouterYAML string

Expand Down Expand Up @@ -234,6 +237,26 @@ func TestWriteConfig(t *testing.T) {
},
expected: tarsYAML,
},
{
name: "Vercel AI Gateway (https host)",
input: ConfigData{
Backends: []Backend{
{
Name: "openai",
Hostname: "ai-gateway.vercel.sh",
Port: 443,
NeedsTLS: true,
},
},
OpenAI: &OpenAIConfig{
BackendName: "openai",
SchemaName: "OpenAI",
Version: "",
},
OTELLog: &otelLogConfig{Exporter: "console"},
},
expected: vercelYAML,
},
{
name: "OpenRouter (https path prefix)",
input: ConfigData{
Expand Down
234 changes: 234 additions & 0 deletions internal/autoconfig/testdata/vercel.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
# Copyright Envoy AI Gateway Authors
# SPDX-License-Identifier: Apache-2.0
# The full text of the Apache license is available in the LICENSE file at
# the root of the repo.

# Configuration for Envoy AI Gateway with OpenAI compatible endpoint
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: aigw-run
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: aigw-run
namespace: default
spec:
gatewayClassName: aigw-run
listeners:
- name: http
protocol: HTTP
port: 1975
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: envoy-ai-gateway
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: envoy-ai-gateway
namespace: default
spec:
logging:
level:
default: error
telemetry:
accessLog:
settings:
- matches:
# MCP metadata only exists on backend-listener requests, which do not carry /mcp paths.
# Match LLM by x-ai-eg-model and MCP by x-ai-eg-mcp-backend.
- "request.headers['x-ai-eg-model'] != ''"
sinks:
- type: File
file:
path: /dev/stdout
format:
type: JSON
json:
# LLM specific fields. Dynamic metadata expressions must match
# the ones defined in the AIGatewayRoute llmRequestCosts field or
# header-mapped attributes via OTEL_*_REQUEST_HEADER_ATTRIBUTES.
gen_ai.request.model: "%REQ(X-AI-EG-MODEL)%"
gen_ai.response.model: "%DYNAMIC_METADATA(io.envoy.ai_gateway:response_model)%"
gen_ai.provider.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:backend_name)%"
gen_ai.usage.input_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_input_token)%"
gen_ai.usage.output_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_output_token)%"
gen_ai.usage.reasoning_tokens: "%DYNAMIC_METADATA(io.envoy.ai_gateway:llm_reasoning_token)%"
session.id: "%DYNAMIC_METADATA(io.envoy.ai_gateway:session.id)%"
# Common fields
start_time: "%START_TIME%"
method: "%REQ(:METHOD)%"
request.path: "%REQ(:PATH)%"
x-envoy-origin-path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
response_code: "%RESPONSE_CODE%"
connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%"
upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%"
bytes_received: "%BYTES_RECEIVED%"
bytes_sent: "%BYTES_SENT%"
duration: "%DURATION%"
x-envoy-upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
x-forwarded-for: "%REQ(X-FORWARDED-FOR)%"
user-agent: "%REQ(USER-AGENT)%"
x-request-id: "%REQ(X-REQUEST-ID)%"
upstream_host: "%UPSTREAM_HOST%"
upstream_cluster: "%UPSTREAM_CLUSTER%"
upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%"
downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%"
downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%"
- matches:
- "request.headers['x-ai-eg-mcp-backend'] != ''"
sinks:
- type: File
file:
path: /dev/stdout
format:
type: JSON
json:
# MCP specific fields
jsonrpc.request.id: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_request_id)%"
mcp.session.id: "%REQ(MCP-SESSION-ID)%"
mcp.method.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_method)%"
mcp.tool.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_tool_name)%"
session.id: "%DYNAMIC_METADATA(io.envoy.ai_gateway:session.id)%"
mcp.provider.name: "%DYNAMIC_METADATA(io.envoy.ai_gateway:mcp_backend)%"
# Common fields
start_time: "%START_TIME%"
method: "%REQ(:METHOD)%"
request.path: "%REQ(:PATH)%"
x-envoy-origin-path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
response_code: "%RESPONSE_CODE%"
connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%"
upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%"
bytes_received: "%BYTES_RECEIVED%"
bytes_sent: "%BYTES_SENT%"
duration: "%DURATION%"
x-envoy-upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
x-forwarded-for: "%REQ(X-FORWARDED-FOR)%"
user-agent: "%REQ(USER-AGENT)%"
x-request-id: "%REQ(X-REQUEST-ID)%"
upstream_host: "%UPSTREAM_HOST%"
upstream_cluster: "%UPSTREAM_CLUSTER%"
upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%"
downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%"
downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%"

---
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: AIGatewayRoute
metadata:
name: aigw-run
namespace: default
spec:
parentRefs:
- name: aigw-run
kind: Gateway
group: gateway.networking.k8s.io
# Simple rule: route everything to OpenAI backend
rules:
- matches:
- headers:
- type: RegularExpression
name: x-ai-eg-model
value: .*
backendRefs:
- name: openai
namespace: default
timeouts:
request: 120s
# Configure the LLM request costs so they can be included in the Envoy access logs
llmRequestCosts:
- metadataKey: llm_input_token
type: InputToken
- metadataKey: llm_output_token
type: OutputToken
- metadataKey: llm_reasoning_token
type: ReasoningToken
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: openai
namespace: default
spec:
endpoints:
- fqdn:
hostname: ai-gateway.vercel.sh
port: 443
---
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: AIServiceBackend
metadata:
name: openai
namespace: default
spec:
timeouts:
request: 3m
schema:
name: OpenAI
backendRef:
name: openai
kind: Backend
group: gateway.envoyproxy.io
namespace: default
---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
name: openai-tls
namespace: default
spec:
targetRefs:
- group: 'gateway.envoyproxy.io'
kind: Backend
name: openai
validation:
wellKnownCACertificates: "System"
hostname: ai-gateway.vercel.sh
---
# By default, Envoy Gateway sets the buffer limit to 32kiB which is not
# sufficient for AI workloads. This ClientTrafficPolicy sets the buffer limit
# to 50MiB as an example.
# TODO: Remove after https://github.com/envoyproxy/ai-gateway/issues/1212
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: client-buffer-limit
namespace: default
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: aigw-run
connection:
bufferLimit: 50Mi
---
apiVersion: v1
kind: Secret
metadata:
name: openai-apikey
namespace: default
type: Opaque
stringData:
apiKey: ${OPENAI_API_KEY}
---
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: BackendSecurityPolicy
metadata:
name: openai-apikey
namespace: default
spec:
targetRefs:
- group: aigateway.envoyproxy.io
kind: AIServiceBackend
name: openai
type: APIKey
apiKey:
secretRef:
name: openai-apikey
---
1 change: 1 addition & 0 deletions site/docs/capabilities/llm-integrations/supported-endpoints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,7 @@ The following table summarizes which providers support which endpoints:
| [Hunyuan](https://cloud.tencent.com/document/product/1729/111007) | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API |
| [Tencent LLM Knowledge Engine](https://www.tencentcloud.com/document/product/1255/70381) | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API |
| [Tetrate Agent Router Service (TARS)](https://router.tetrate.ai/) | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Via OpenAI-compatible API |
| [Vercel AI Gateway](https://vercel.com/docs/ai-gateway) | ⚠️ | ❌ | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | Via OpenAI-compatible API; Anthropic endpoints via the `Anthropic` schema |
| [Google Vertex AI](https://cloud.google.com/vertex-ai/docs/reference/rest) | ✅ | 🚧 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | Via API translation |
| [Anthropic on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude) | ✅ | ❌ | 🚧 | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | Via API translation |
| [Anthropic on AWS Bedrock](https://aws.amazon.com/bedrock/anthropic/) | 🚧 | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | Native Anthropic API |
Expand Down
1 change: 1 addition & 0 deletions site/docs/capabilities/llm-integrations/supported-providers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Below is a table of currently supported providers and their respective configura
| [Hunyuan](https://cloud.tencent.com/document/product/1729/111007) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | |
| [Tencent LLM Knowledge Engine](https://www.tencentcloud.com/document/product/1255/70381?lang=en) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | |
| [Tetrate Agent Router Service (TARS)](https://router.tetrate.ai/) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | |
| [Vercel AI Gateway](https://vercel.com/docs/ai-gateway) | `{"name":"OpenAI","prefix":"/v1"}` or `{"name":"Anthropic"}` | [API Key] or [Anthropic API Key] | ✅ | Aggregating gateway. Model IDs are namespaced, e.g. `openai/gpt-4o-mini`. |
| [SambaNova](https://docs.sambanova.ai/sambastudio/latest/open-ai-api.html) | `{"name":"OpenAI","prefix":"/v1"}` | [API Key] | ✅ | |
| Self-hosted-models | `{"name":"OpenAI","prefix":"/v1"}` | N/A | ⚠️ | Depending on the API schema spoken by self-hosted servers. For example, [vLLM] speaks the OpenAI format. Also, API Key auth can be configured as well. |
| [Anthropic](https://docs.claude.com/en/home) | `{"name":"Anthropic"}` | [Anthropic API Key] | ✅ | Support only Native Anthropic messages endpoint |
Expand Down
Loading
Loading