diff --git a/.agents/skills/auditing-llm-gateway-parity/SKILL.md b/.agents/skills/auditing-llm-gateway-parity/SKILL.md new file mode 100644 index 000000000000..4b4e3a024312 --- /dev/null +++ b/.agents/skills/auditing-llm-gateway-parity/SKILL.md @@ -0,0 +1,74 @@ +--- +name: auditing-llm-gateway-parity +description: > + Audits services/llm-gateway against PostHog/ai-gateway and updates services/llm-gateway/PARITY.md from current implementation evidence. Use when either gateway changes auth, attribution, billing, endpoints, providers, models, routing, or metadata; when reviewing a Python gateway change; or when asked to refresh, verify, or report gateway parity. This skill updates the parity record but does not migrate callers. +--- + +# Auditing LLM gateway parity + +Compare the current implementations of both gateways and update [`services/llm-gateway/PARITY.md`](../../../services/llm-gateway/PARITY.md). Keep this audit separate from caller migration. + +## Record source revisions + +- Fetch `origin/master` and record its SHA as the PostHog baseline. +- Audit the current working tree, including committed and uncommitted changes relative to that baseline. Do not ignore an in-flight gateway change because it is not on `master` yet. +- Read the current `PostHog/ai-gateway` main SHA with `gh api repos/PostHog/ai-gateway/commits/main` and record it as the Go baseline. +- If the audit is for an in-flight Go change, inspect that PR branch or working tree relative to `main`. Treat its contracts as pending until the change merges; do not record them as currently supported. +- Otherwise, inspect an authenticated checkout of `PostHog/ai-gateway` at `main`. + +Audit implementation code. README files and the existing parity table are starting points, not proof. + +## Inspect each contract + +For the Python gateway, inspect: + +- `services/llm-gateway/src/llm_gateway/api/` for routes and wire behavior +- `services/llm-gateway/src/llm_gateway/auth/` for accepted credentials +- `services/llm-gateway/src/llm_gateway/products/config.py` for trusted product policy, models, and billing +- `services/llm-gateway/src/llm_gateway/rate_limiting/` for budgets and limits +- `services/llm-gateway/src/llm_gateway/callbacks/` for event attribution +- `posthog/llm/gateway_client.py` and real call sites for required contracts + +For the Go gateway, inspect: + +- `internal/httpapi/routes.go` and dispatch packages for API shapes +- `internal/auth/` and `internal/principal/` for credential and identity policy +- `internal/httpapi/admission.go`, `internal/ledger/`, and `internal/quota/` for billing and limits +- `internal/catalog/`, `internal/router/`, and `internal/dispatch/` for models, providers, translation, and failover +- `internal/emitter/` and request parsing for attribution and metadata +- `docs/product.md` for intended and deferred contracts, verified against code + +Check both request and response behavior. Matching route names do not prove parity for headers, streaming, errors, timeouts, retries, billing, or emitted events. + +## Classify the evidence + +- ✅ **Supported:** the Go gateway satisfies the contract. +- ⛔ **Blocking:** an active use case would lose auth, trusted attribution, billing policy, API, provider, or wire behavior. +- 🔎 **Verify:** support exists, but a caller must confirm configuration or behavior. + +Do not treat caller-supplied telemetry as trusted policy. An `ai_product` event property does not replace product authentication, authorization, or billing. + +Do not treat Python's unbilled flag as an automatic blocker. An internal workload can move to Go with a PostHog-owned team credential when debiting that wallet is the intended way to attribute PostHog spend. It is blocked only when it must preserve customer-specific billing policy or must debit no wallet. + +For each difference, identify at least one affected use-case class. Remove details that do not change a migration decision. + +## Update the parity record + +Update `services/llm-gateway/PARITY.md` with the evidence: + +- Move use cases between supported, blocked, and verify sections. +- Add or remove migration-relevant contracts in the parity map. +- Update the source SHAs and verification date. +- Keep detailed Go design in `PostHog/ai-gateway`. + +When a gap closes, name obvious caller classes that are newly eligible to migrate. Do not modify those callers unless the user also asks for migration work. Use `/migrating-llm-gateway-callers` for that separate job. + +## Validate + +Run: + +```sh +pnpm exec oxfmt services/llm-gateway/PARITY.md .agents/skills/auditing-llm-gateway-parity/SKILL.md +pnpm exec markdownlint-cli2 --config .config/.markdownlint-cli2.jsonc services/llm-gateway/PARITY.md .agents/skills/auditing-llm-gateway-parity/SKILL.md +git diff --check +``` diff --git a/.agents/skills/finding-llm-gateway-migration-candidates/SKILL.md b/.agents/skills/finding-llm-gateway-migration-candidates/SKILL.md new file mode 100644 index 000000000000..048b1d963b49 --- /dev/null +++ b/.agents/skills/finding-llm-gateway-migration-candidates/SKILL.md @@ -0,0 +1,94 @@ +--- +name: finding-llm-gateway-migration-candidates +description: > + Finds and ranks callers that could move from services/llm-gateway to PostHog/ai-gateway. Use when asked what to migrate next, to find low-risk gateway migration candidates, to audit remaining Python gateway callers, or to identify callers blocked by Go gateway parity. Searches code and deployment wiring, inventories each caller's required contract, filters out unsupported migrations, and returns an evidence-backed shortlist without changing callers. +--- + +# Finding LLM gateway migration candidates + +Use [`services/llm-gateway/PARITY.md`](../../../services/llm-gateway/PARITY.md) as the current contract decision record. If implementation evidence contradicts it, run `/auditing-llm-gateway-parity` before ranking candidates. + +This skill produces a point-in-time shortlist. Do not turn the parity document into a fleet-wide migration tracker. + +## Discover callers + +Search the current default branches for Python gateway use, including: + +- `LLM_GATEWAY_URL`, `LLM_GATEWAY_API_KEY`, and product-slug URLs +- `get_llm_client`, `get_async_llm_client`, and `get_async_anthropic_gateway_client` +- direct OpenAI, Anthropic, or agent SDK clients configured with a gateway base URL +- Python product names from `services/llm-gateway/src/llm_gateway/products/config.py` +- deployment, sandbox, workflow, and secret wiring in other PostHog repositories + +Trace each result to the production call site. Exclude tests, development-only tools, dead code, and callers already configured exclusively for Go. Treat shared worker configuration as evidence about availability, not proof that every call in that process uses Go. + +Search open and recently merged PRs across involved repositories before proposing work. A migration may already be underway even when the default branch still uses Python. + +## Prove the production path + +Before classifying a caller, follow it through every runtime boundary: + +1. Find the API, task, workflow, schedule, or command that invokes it. +2. For Temporal, map the workflow and activity to the exact task queue, then map that queue to its deployed worker. +3. Inspect every environment overlay that runs the caller. Do not borrow configuration from another worker in the same product. +4. Resolve the configured URL by host and path. Variable names are not proof: an `AI_GATEWAY_URL` can still contain a Python product-slug URL. +5. Confirm the current call can succeed. A broad exception handler with a deterministic fallback can hide missing credentials or a route that never runs. + +A missing `phs_` credential is deployment work, not a parity blocker. Once the intended paying team is known, create a project secret with `llm_gateway:read` in the PostHog dashboard and fund or configure that team wallet as part of the migration. A secret reference proves only that a credential is injected; verify ownership, scope, and funding during implementation or rollout. + +## Inventory each candidate + +Record only the contracts that affect the migration decision: + +- user-facing use case and production entry point +- credential source and trusted authorization policy +- spend owner, budget enforcement, current Python billing behavior, and the team wallet that should own Go spend +- API shape and model as one pair, provider, streaming, tools, and structured output +- distinct ID, trace, product, team, and custom attribution +- retry, timeout, fallback, and error behavior +- deployment and egress changes needed to activate and roll back the route + +Do not infer requirements from a product name. Read the call and its configuration. + +Check model and API shape together. Python can expose an Anthropic model through OpenAI Chat Completions, while Go's native routes do not generally cross-translate shapes. A migration may need both a gateway change and an SDK-shape change. + +Separate billing identity from event attribution. Go debits the credential's team. A caller-supplied `team_id` or `ai_product` property can preserve reporting context but cannot select the wallet. + +## Classify readiness + +Match the inventory to the parity record and current Go implementation: + +- ✅ **Ready:** required contracts are supported and the remaining work is client, test, or deployment wiring. +- 🔎 **Verify:** support probably exists, but model availability, wire behavior, billing, attribution, or deployment configuration needs a targeted check. +- ⛔ **Blocked:** migration would lose a required auth, policy, billing, provider, API, or attribution contract. +- 🚧 **In progress:** an open or recently merged PR already owns the migration. + +Name the exact evidence for every status. An existing Python helper is migration effort, not a blocker. An `ai_product` property is telemetry, not trusted identity or billing policy. + +Python's unbilled flag is not a blocker when an internal workload should debit a PostHog-owned team wallet for spend attribution. Verify that the Go credential resolves to that team. Treat billing as blocked only when migration would charge a customer incorrectly, lose required customer budget policy, or violate a requirement to debit no wallet. + +Use ✅ when the production entry point, worker or process, intended paying team, model and API pair, attribution conversion, and rollback path are supported by evidence. The token itself may still need to be created and wired. Use 🔎 when the spend owner, required policy, runtime behavior, or another migration contract is unknown. + +## Rank the shortlist + +Prioritize candidates that: + +1. Have no ⛔ contract. +2. Use a shared Go-capable builder or a stock SDK with a simple base URL change. +3. Already use a Go-supported model, provider, and API shape. +4. Need informational attribution rather than trusted product identity. +5. Can use a customer wallet intentionally or a PostHog-owned wallet for internal spend. +6. Have a narrow deployment boundary, explicit fallback, and cheap verification path. + +Lower the rank for broad shared-process switches, unverified billing changes, cross-repository deployment work, or missing production tests. Do not rank by code size alone. + +## Report candidates + +Return a small decision-oriented table with: + +| Candidate | Readiness | Why | Required work | Evidence gaps | +| --------- | --------- | --- | ------------- | ------------- | + +List the strongest ready candidate first. Name the exact production process and deployment for each candidate. For blocked callers, state the missing Go contract and where it was verified. For in-progress work, link the existing PR instead of proposing duplicate work. For 🔎 candidates, state the one check that would promote or reject them. + +Do not modify callers while running this skill. Once a candidate is selected, run `/migrating-llm-gateway-callers` for that caller. diff --git a/.agents/skills/migrating-llm-gateway-callers/SKILL.md b/.agents/skills/migrating-llm-gateway-callers/SKILL.md new file mode 100644 index 000000000000..7db74c864a84 --- /dev/null +++ b/.agents/skills/migrating-llm-gateway-callers/SKILL.md @@ -0,0 +1,81 @@ +--- +name: migrating-llm-gateway-callers +description: > + Migrates an LLM caller from services/llm-gateway to PostHog/ai-gateway. Use when adding a gateway caller, converting an existing Python gateway integration, adopting shared Go-capable client builders, changing gateway URLs or headers for a caller, or removing a Python fallback. Inventories the caller's contract, checks the parity record, implements the supported migration, updates tests, and stops with a documented blocker when Go parity is missing. +--- + +# Migrating LLM gateway callers + +Use [`services/llm-gateway/PARITY.md`](../../../services/llm-gateway/PARITY.md) as the current decision record. If it is stale or the migration reveals a new gap, run `/auditing-llm-gateway-parity` before continuing. + +## Inventory the caller + +Find the production call site, client construction, settings, deployment wiring, and tests. Record: + +- caller and user-facing use case +- credential source and required authorization policy +- spend owner, budgets, current Python billing behavior, and the team wallet that should own Go spend +- API shape, model, provider, streaming, and tool or structured-output requirements +- distinct ID, trace, product, team, feature flag, and custom property attribution +- timeout, retry, fallback, and error-handling behavior +- usage or quota APIs read by the caller + +Search for the product name, `LLM_GATEWAY`, `AI_GATEWAY`, gateway client helpers, and gateway headers. Follow configuration into sandbox or deployment code when the call does not run in Django. + +## Check whether migration is supported + +Match every required contract to the parity record and current code. + +- If all required contracts are supported, continue with the migration. +- If a 🔎 item applies, verify it against the caller's actual model, request, and configuration. +- If a ⛔ gap applies, stop the migration. Report the exact blocker and keep the Python path. + +An existing Python client or product route is not a blocker by itself. + +Python's unbilled flag is not a blocker when an internal workload should debit a PostHog-owned team wallet for spend attribution. Confirm that the Go credential resolves to that team. Stop only when migration would charge a customer incorrectly, lose required customer budget policy, or violate a requirement to debit no wallet. + +## Implement the migration + +Prefer the smallest existing pattern that matches the caller: + +Read [migration examples](references/migration-examples.md) for verified PRs covering Django clients, staged workload rollout, sandbox wiring, and attribution continuity. Follow the contract demonstrated by the relevant example rather than copying its code mechanically. + +1. Use `build_openai_client`, `build_async_openai_client`, or `build_async_anthropic_client` from `posthog/llm/gateway_client.py` for Django callers when possible. +2. Use the slugless Go base URL. Do not carry a Python `/{product}/` path into the Go URL. +3. Use a supported `phs_` or `pha_` credential with `llm_gateway:read`. Do not weaken auth or expose a shared secret to an untrusted runtime. +4. Send event labels in one `X-PostHog-Properties` JSON object. Use the dedicated distinct ID and trace ID headers where required. +5. Treat `ai_product` as telemetry only. Do not use it to replace trusted product auth or billing policy. +6. Confirm the canonical model and API shape against the Go model catalog. +7. Preserve the caller's provider and fallback requirements. Use provider pinning only when the caller requires a specific host. +8. Keep a Python fallback only when rollout needs it, and make the switch explicit in settings or the shared builder. + +For sandbox callers, follow the existing `SANDBOX_AI_GATEWAY_URL` and product rollout patterns rather than inventing another environment contract. + +If the target process does not have a Go credential, create a `phs_` project secret for the intended paying team in the PostHog dashboard, grant `llm_gateway:read`, and wire it through the existing deployment secret mechanism. Credential creation is normal migration work, not a parity exception. + +## Update tests and docs + +Invoke `/writing-tests` before changing tests. + +Cover the observable migration contract at the lowest useful level: + +- selected base URL and absence of the Python product slug +- credential and required headers without asserting secret values +- converted JSON properties and trace attribution +- model and API shape +- rollout fallback when one remains +- failure behavior the caller handles + +Update nearby docs and settings descriptions when the operator workflow changes. Do not update the parity record unless implementation evidence changed; use `/auditing-llm-gateway-parity` when it did. + +## Verify + +Run the narrow caller tests and the formatter or type checker for touched code. Exercise a real request when credentials and a safe development gateway are available. Confirm the event attribution and billing behavior when those contracts matter. + +Before handing off, summarize: + +- migrated caller and use case +- Go contracts relied on +- fallback left in place, if any +- parity blocker, if migration stopped +- checks run and any environment limitation diff --git a/.agents/skills/migrating-llm-gateway-callers/references/migration-examples.md b/.agents/skills/migrating-llm-gateway-callers/references/migration-examples.md new file mode 100644 index 000000000000..bc39bf1f5923 --- /dev/null +++ b/.agents/skills/migrating-llm-gateway-callers/references/migration-examples.md @@ -0,0 +1,56 @@ +# Migration examples + +These PRs show different migration shapes. Read the relevant diff before implementing a similar change because gateway contracts have continued to evolve. Use them for implementation decisions, not as an inventory of current rollout state. + +## Django and direct-provider callers + +- [#64448: make cluster labeling routable through the AI gateway](https://github.com/PostHog/posthog/pull/64448) introduces an environment-gated OpenAI client, validates paired settings, preserves direct-provider fallback, checks model availability before rollout, and tests both routes. +- [#65043: route the eval-report agent through the AI gateway](https://github.com/PostHog/posthog/pull/65043) moves the AI observability eval-report agent after the shared route exists and tests the selected client path. +- [#65044: route OpenAI summarization through the AI gateway](https://github.com/PostHog/posthog/pull/65044) extracts shared sync and async builders, uses the slugless Go URL and project secret, keeps `trust_env=False`, retains the Python fallback, and prevents duplicate `$ai_generation` capture. + +## Incremental product rollout + +- [#71947: add the Signals AI gateway client and per-call opt-in](https://github.com/PostHog/posthog/pull/71947) adds the Anthropic builder and an opt-in seam without moving traffic. It converts per-key metadata to `X-PostHog-Properties`, strips `/v1` for the Anthropic SDK, and preserves Python behavior for callers that have not opted in. +- [#71948: route Signals grouping through the AI gateway](https://github.com/PostHog/posthog/pull/71948) uses that seam to move one workload at a time. Reverting the small opt-in returns only grouping to Python. +- [#71949: route Signals emission stages through the AI gateway](https://github.com/PostHog/posthog/pull/71949) handles batch and per-call metadata explicitly. Its tests cover both the Go JSON properties blob and Python per-key fallback headers. +- [#71950: route Signals safety through the AI gateway](https://github.com/PostHog/posthog/pull/71950) opts the safety filter and report safety judge into the shared Go-capable client. +- [#71951: route Signals eval summarization through the AI gateway](https://github.com/PostHog/posthog/pull/71951) applies the same narrow per-call rollout to eval summaries. +- [#72769: tag eval-fixture generation as `signals_eval`](https://github.com/PostHog/posthog/pull/72769) preserves workload attribution for Signals eval generation. + +## Sandbox and deployment wiring + +- [#72770: route selected sandbox products to the AI gateway](https://github.com/PostHog/posthog/pull/72770) treats migration as more than a client change. It pairs URL and product rollout settings, reserves them against user overrides, updates both egress enforcement layers, validates configured hostnames, extends startup diagnostics, and keeps rollback to clearing either setting. + +## Cross-repository patterns + +### Standalone TypeScript service + +1. [PostHog/SherlockHog #104: route the agent through the slugless AI gateway](https://github.com/PostHog/SherlockHog/pull/104) adds the paired Go settings, base URL translation, project-secret auth, Python fallback, and route-selection tests for a TypeScript Claude Agent SDK service. +2. [PostHog/SherlockHog #111: boot with either complete gateway pair](https://github.com/PostHog/SherlockHog/pull/111) fixes the transitional startup contract so an AI-gateway-only deployment can boot while preserving the Python rollback route. +3. [PostHog/SherlockHog #112: tag the AI product through `X-PostHog-Properties`](https://github.com/PostHog/SherlockHog/pull/112) corrects attribution after the initial cutover reused Python's per-property headers. It keeps each gateway's metadata format separate and verifies that the Go route no longer emits the legacy form. +4. [PostHog/charts #12919: cut development over to the slugless AI gateway](https://github.com/PostHog/charts/pull/12919) wires the development URL and secret first, with explicit deployment rendering and rollback checks. +5. [PostHog/charts #12920: cut production over to the slugless AI gateway](https://github.com/PostHog/charts/pull/12920) applies the regional production configuration after the development stage. + +Together these show the client, transitional boot contract, attribution correction, staged deployment, and rollback boundaries. + +### Per-product sandbox routing + +1. [PostHog/code #3659: route selected products to the AI gateway](https://github.com/PostHog/code/pull/3659) selects a gateway per sandbox request. It requires both the Go URL and an allowlist, keeps unlisted products on Python, converts attribution to one bounded ASCII-safe JSON header, and gives each workload a distinct product tag. +2. [PostHog/posthog #72770: route selected sandbox products to the AI gateway](https://github.com/PostHog/posthog/pull/72770) passes the settings into sandboxes, reserves them against user overrides, updates both egress enforcement layers, validates configured hostnames, and extends startup diagnostics. +3. [PostHog/charts #13358: route Signals sandbox stages in development](https://github.com/PostHog/charts/pull/13358) activates the client and Django support for four workloads in development. + +These show how to select Go per request, pass configuration through a sandbox boundary, enforce egress, and activate a narrow set of workloads. + +### Standalone PR-review agent + +- [PostHog/posthog #68329: route the PR-approval agent through the AI gateway](https://github.com/PostHog/posthog/pull/68329) and [PostHog/code #3354: route PR review through the AI gateway](https://github.com/PostHog/code/pull/3354) validate paired settings, strip `/v1` before the SDK restores its messages path, set both Anthropic auth variables, avoid duplicate capture by bypassing the traced wrapper in gateway mode, and retain an explicit fallback. + +### Deployment activation + +- [PostHog/charts #13131: repoint worker deployments to the AI gateway](https://github.com/PostHog/charts/pull/13131) demonstrates that merged client support does not move traffic by itself. It wires the regional URL and app-specific secret into every deployment that runs an already-migrated caller, with removal of either setting as rollback. Use it as deployment guidance, not as a standalone caller migration. + +## Post-migration parity checks + +- [#74249: send a stable team trace ID to the Go gateway](https://github.com/PostHog/posthog/pull/74249) fixes trace fragmentation found after a migration. It matches Python's existing trace derivation, sends the dedicated Go header, and pins cross-gateway compatibility with fixed expected IDs. + +Use these examples for their decisions and verification boundaries. Re-check [`services/llm-gateway/PARITY.md`](../../../../services/llm-gateway/PARITY.md) before adopting an older pattern. diff --git a/AGENTS.md b/AGENTS.md index 065ab73f5164..33728c90187c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -169,6 +169,7 @@ See [.agents/security.md](.agents/security.md) for security guidelines — least - **Object storage is SeaweedFS — do not add new MinIO dependencies.** Both S3-compatible stores in the dev/CI stack are SeaweedFS: the `objectstorage` service (S3 API on `:19000`) backs general object storage (`OBJECT_STORAGE_*` settings — exports, media uploads, error-tracking source maps, query cache, tasks), and the `seaweedfs` service (S3 API on `:8333`) backs session replay v2 (`SESSION_RECORDING_V2_S3_*` settings). MinIO now survives only as migration tooling: `docker-compose.hobby.yml` keeps it as a source for `bin/migrate-storage-hobby`, and `bin/upgrade-objectstorage` starts a throwaway MinIO to salvage objects off the pre-swap volume. Outside that, don't add docker-compose services, scripts, tests, or docs that stand up a `minio/minio` container. Code that talks to object storage should go through the existing `OBJECT_STORAGE_*` / `SESSION_RECORDING_V2_S3_*` config and a standard S3 client rather than hardcoding an endpoint — that keeps backends swappable. Note the `objectstorage` service registers its credentials at runtime via a bootstrap loop and returns `InvalidAccessKeyId` until that completes, so anything depending on it must wait for its readiness sentinel rather than just for the container to start. - **Temporal activity payloads have a ~2 MiB hard limit — pass large data by reference, not by value.** Activity inputs and outputs are serialized across a gRPC boundary that Temporal caps at ~2 MiB per payload (the server rejects larger payloads via `blobSizeLimitError`). As a conservative field-level rule, if a field could exceed ~256 KB once serialized (serialized query results, exported file contents, LLM context, rendered HTML, image bytes, unbounded `list[dict[str, Any]]`), write it to Postgres / S3 / object storage from _inside_ the activity and return only the reference (row ID, S3 key). The workflow already has access to any row ID created earlier in the same run; it does not need the content to flow back through. Shuttling large data through the workflow on the way to persistence is a foreseeable failure mode that produces `PayloadSizeError` (`TMPRL1103`) the moment the underlying data crosses the limit. - **Outbound calls to a third-party API that need rate-limiting or egress telemetry belong in `posthog/egress/` — add a `/` incarnation (GitHub is the reference) and route callers through its gated, recorded transport, never hand-rolled `requests`. See `posthog/egress/README.md`.** +- **`services/llm-gateway` is under an unofficial code freeze while callers move to [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway).** New callers and features belong on the Go gateway by default. A Python gateway change needs a documented parity blocker for an active caller and must stay limited to that blocker. Read [`services/llm-gateway/PARITY.md`](services/llm-gateway/PARITY.md). Invoke `/auditing-llm-gateway-parity` for gateway contract changes and parity refreshes, `/finding-llm-gateway-migration-candidates` when deciding what to migrate next, and `/migrating-llm-gateway-callers` when moving a selected caller. ## Code Style @@ -243,3 +244,6 @@ ALWAYS invoke the matching skill **before** writing or reviewing code in these a - `/authoring-ci-workflows` — adding or editing any `.github/workflows` workflow, composite action, or reusable workflow - `/reviewing-personhog-protocol` — any personhog coordination-protocol change (leases, fencing, handoffs, supervisors, budgets, warming, changelog semantics), and any request for an exhaustive review of personhog code - `/gating-production-deploys` — any workflow that builds and pushes a production image or dispatches a deploy +- `/auditing-llm-gateway-parity` — changing either gateway's auth, attribution, billing, endpoints, providers, models, routing, or metadata contract; reviewing a `services/llm-gateway` change; or refreshing `services/llm-gateway/PARITY.md` +- `/finding-llm-gateway-migration-candidates` — finding, auditing, or ranking callers that could move from `services/llm-gateway` to `PostHog/ai-gateway`, including requests for the next or lowest-risk migration candidate +- `/migrating-llm-gateway-callers` — adding an LLM gateway caller or migrating an existing caller from `services/llm-gateway` to `PostHog/ai-gateway`, including shared client and gateway setting changes made for that migration diff --git a/services/llm-gateway/PARITY.md b/services/llm-gateway/PARITY.md new file mode 100644 index 000000000000..38d2a10efe63 --- /dev/null +++ b/services/llm-gateway/PARITY.md @@ -0,0 +1,97 @@ +# Choosing an AI gateway + +✅ **Default to [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway)** for new callers and features. + +⚠️ **Use `services/llm-gateway` temporarily** only when an active caller is blocked by a gap below. + +## Migration policy + +`services/llm-gateway` is under an unofficial code freeze. Everyone should move to the Go gateway unless they can name a contract it does not support. + +A PR that changes the Python gateway must: + +1. Name the active caller and its use case. +2. Name the exact Go gateway parity gap. +3. Explain why the change cannot wait for or land in the Go gateway. +4. Limit the Python change to the blocked caller's needs. +5. Update this document when it discovers or closes a gap. + +Bug, security, and reliability fixes for blocked callers are valid reasons. Convenience, an existing Python client, or an unfamiliar Go API are not parity gaps. + +When a gap closes, new work uses the Go gateway and affected callers should migrate. Do not add the same feature to both gateways by default. + +## Choose by use case + +### ✅ Use the Go gateway + +| Use case | Why it fits | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Customer LLM traffic | The caller uses a project secret or OAuth credential, the team wallet should pay, and standard OpenAI or Anthropic APIs are enough. | +| Server-to-server PostHog call | A `phs_` credential, team wallet billing, and informational event properties provide enough policy and attribution. Internal spend can use a PostHog-owned team wallet. | +| Stock SDK proxy | The caller needs OpenAI Chat Completions or Responses, Anthropic Messages or token counting, streaming, idempotency, or the model catalog. | +| Gateway-managed routing | The caller accepts Go host selection across OpenAI, Anthropic, Azure OpenAI, Bedrock, or selected Modal models. | + +Existing Django callers should use `build_openai_client`, `build_async_openai_client`, or `build_async_anthropic_client` from [`posthog/llm/gateway_client.py`](../../posthog/llm/gateway_client.py). These builders keep a temporary Python fallback during rollout. + +### ⛔ Stay on the Python gateway for now + +| Use case | Blocking contract | +| ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| PostHog Desktop, PostHog Code, cloud agents, onboarding, Wizard, and similar first-party products | Trusted product identity, OAuth application allowlists, OAuth user attribution, server-minted credential requirements, per-product model policy, and product billing or unbilled policy are not available together in Go. An `ai_product` event property is not an authorization or billing boundary. | +| Product or user budget enforcement | Python supports product and user cost limits, plan checks, quota buckets, and requests that debit no wallet. Go currently admits against the credential owner's team wallet and does not branch settlement on credential billing mode. | +| OpenRouter, Fireworks, Cloudflare Workers AI, or Baseten | These provider paths are not available in Go. | +| OpenAI audio transcription | Go has no transcription endpoint. | +| OpenAI models through Anthropic Messages | Go does not provide this reverse translation. | +| Python product usage status | The Python product usage and quota API is different from Go request usage and wallet APIs. | + +### 🔎 Verify before switching + +These are compatibility checks, not automatic blockers: + +- **Model:** it appears in `GET /v1/models` and supports the requested API shape and capabilities. +- **Credential:** Go accepts the credential type and projects the correct team, scope, and revocation state. +- **Billing:** identify which team owns the Go credential and should pay. Internal workloads can debit a PostHog-owned team wallet so spend stays attributable without charging a customer. +- **Attribution:** `X-PostHog-Distinct-Id`, `X-PostHog-Trace-Id`, and `X-PostHog-Properties` provide enough event context. Go does not derive the event distinct ID from OpenAI `user`, Anthropic `metadata.user_id`, or the OAuth user. Caller-supplied properties are not trusted policy. +- **Provider behavior:** health-based routing or strict `X-PostHog-Provider` pinning matches the caller's fallback requirements. +- **Wire behavior:** request fields, streaming chunks, errors, timeouts, and retries match what the caller handles. +- **Metadata:** Python per-key property and feature flag headers are converted to the Go JSON properties header. + +## Parity map + +| Contract | Go gateway today | Python-only behavior that can block migration | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Authentication | `phs_` project secrets and `pha_` OAuth credentials with the literal `llm_gateway:read` scope; credential resolves team and revocation state. | `phx_` personal keys, wildcard OAuth scope handling, and product-specific API key or OAuth application rules. | +| Trusted first-party identity | No trusted product identity or product-scoped authorization boundary. | Product route selects allowlisted OAuth apps, server credential requirements, and model policy. | +| Attribution | Distinct ID comes from `X-PostHog-Distinct-Id` or falls back to team ID. Supports trace ID, custom JSON properties, team project token, provider, usage, and cost. | OAuth user identity, body-derived OpenAI or Anthropic end-user identity, gateway-owned `$ai_product` and `$ai_billable`, per-key properties, and feature flag headers. | +| Billing and limits | Wallet reservation and settlement, idempotency, balance read, request usage read, and front-line rate limiting. | Product and user cost windows, plans, quota buckets, free tiers, unbilled products, and product usage status. | +| OpenAI APIs | Chat Completions, Responses, bare Responses alias, and normalized router chat. | Audio transcription and broader LiteLLM translation. | +| Anthropic APIs | Messages and token counting, including Bedrock-hosted models. | OpenAI models exposed through the Anthropic shape and Python-specific Bedrock opt-in behavior. | +| Providers | OpenAI, Anthropic, Azure OpenAI, Bedrock, and selected Modal models. | OpenRouter, Fireworks, Cloudflare Workers AI, and Baseten. | +| Models | Gateway-owned catalog, canonical IDs and aliases, capability checks, and router categories. | Broader LiteLLM model acceptance and Python product allowlists. | +| Routing and failure behavior | Health-based host choice, circuit breakers, hosted-provider failover, and strict provider pinning. | Caller opt-in Bedrock fallback and provider-specific Python routing. | +| Event metadata | One `X-PostHog-Properties` JSON object plus dedicated distinct ID, trace ID, and provider headers. | `X-POSTHOG-PROPERTY-*` and `X-POSTHOG-FLAG-*` headers. | + +## Migration checklist + +Run `/migrating-llm-gateway-callers` to inventory and convert a caller. + +1. Describe the caller and identify which identity controls its access and spend. +2. Check every relevant contract above against the real request and response. +3. Start with a shared Go-capable builder where one exists. +4. Test success, streaming if used, provider errors, attribution, and billing. +5. Keep the Python fallback only for staged rollout or a named blocker. Record the rollout plan or blocker in the PR. + +## Refreshing this document + +Run `/auditing-llm-gateway-parity` after either gateway changes auth, attribution, billing, endpoints, providers, models, routing, or event metadata. The skill audits implementation sources in both repositories and updates this file without migrating callers. + +Last verified on 2026-07-31 against: + +- `PostHog/posthog` master at `75c2b2ff6845097a3eb6dc5214e3f4f660395fb8` +- `PostHog/ai-gateway` main at `ea8230b6dbc4ebf6c4be83d359e561477a13b215` + +## References + +- Python gateway: [`services/llm-gateway`](./README.md) +- Shared clients: [`posthog/llm/gateway_client.py`](../../posthog/llm/gateway_client.py) +- Go gateway: [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway), especially `docs/product.md`