diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 2b8545629c..c9c3094097 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -154,6 +154,8 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | `unsafeAllowNativeLocalExec?` | `boolean` | Ancien booléen de Cursor, équivalent à `nativeLocalExec: "on"` uniquement lorsque le champ plus récent n'est pas défini. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Politique d'exécution locale de Cursor. `off` est la valeur par défaut ; actuellement, `codex-sandbox` échoue de manière sûre comme `off`. | +Pour les requêtes Responses traduites utilisant un pool de plusieurs clés, chaque invocation du fournisseur routé partage une limite de rotations de clés API entre la récupération initiale, les tours des ponts de recherche web, d’image et de vidéo, et les requêtes de continuation jusqu’à la fin de la réponse : au maximum `N - 1` rotations si la clé initiale appartient au pool, ou `N` si elle est extérieure au pool, où `N` est la taille du pool avant le premier envoi. L’expiration d’un délai de refroidissement ou l’agrandissement ultérieur du pool ne renouvelle pas cette limite. Les budgets d’envoi existants applicables peuvent arrêter les tentatives plus tôt ; les ponts sidecar conservent leurs propres limites de reprise après réinitialisation de connexion et de nouvelle tentative vers la même cible, sans que cette limite de rotation leur impose le budget d’envois physiques du core. Si la rotation est refusée, le dernier 429 enregistre toujours le délai de refroidissement de la clé en échec, mais aucune clé de remplacement n’est sélectionnée et la réponse suit le traitement d’erreur existant. + Les fournisseurs à clé API peuvent détenir une clé littérale ou une référence à une variable d'environnement. Les fournisseurs OAuth utilisent le magasin d'identifiants alimenté par `ocx login` ; le comportement de lancement de Claude Code avec abonnement est configuré sous [`claudeCode.authMode`](/fr/reference/configuration/server/#claude-code-claudecode). diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 9b5acfa274..ee9d15459a 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -146,6 +146,8 @@ account を削除しても mapping は保持され、同じ id を再追加す | `unsafeAllowNativeLocalExec?` | `boolean` |カーソルのレガシー ブール値。新しいフィールドが設定されていない場合のみ、`nativeLocalExec: "on"` と同等です。 | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` |カーソルのローカル実行ポリシー。 `off` がデフォルトです。 `codex-sandbox` は現在、`off` と同様にフェールクローズされます。 | +複数キーのプールを使用する変換済み Responses リクエストでは、ルーティング先プロバイダーの呼び出しごとに、初期リカバリー、ウェブ検索・画像・動画ブリッジの各ラウンド、応答完了のための後続リクエストで API キーの切り替え上限を共有し、最初の送信前のプールサイズを `N` として、最初のキーがそのプール内にある場合は最大 `N - 1` 回、プール外にある場合は最大 `N` 回に固定されます。クールダウンの満了や後からのプール拡張で、この上限が補充されることはありません。適用対象となる既存の送信予算によって、再試行がさらに早く停止する場合もあります。サイドカーブリッジは接続リセット時と同一対象への再試行にそれぞれ既存の上限を維持し、この切り替え上限によって core の物理送信予算がブリッジに新たに適用されることはありません。切り替えが拒否された場合も、最後の 429 に対する失敗したキーのクールダウンは記録されますが、代わりのキーは選択されず、応答には既存のエラー処理が適用されます。 + API キープロバイダーは、リテラルキーまたは環境参照を保持する場合があります。 OAuth プロバイダーは、`ocx login` によって設定された資格情報ストアを使用します。サブスクリプションに基づくクロード コードの起動動作は、[`claudeCode.authMode`](/reference/configuration/server/#claude-code) で構成されます。 ## プロバイダーによるアウトバウンドの安全性診断 diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index b6657ba58d..7cdd7d7280 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -146,6 +146,8 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 레거시 불리언입니다. 더 새로운 필드가 설정되지 않았을 때만 `nativeLocalExec: "on"`과 같습니다. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 로컬 실행 정책입니다. 기본값은 `off`입니다. `codex-sandbox`는 현재 `off`처럼 실패를 닫습니다. | +여러 키가 있는 풀을 사용하는 변환된 Responses 요청에서는 라우팅된 프로바이더 호출마다 최초 복구, 웹 검색·이미지·비디오 브리지의 각 라운드, 응답 완료를 위한 후속 요청이 API 키 회전 한도를 공유하며, 첫 전송 전의 풀 크기를 `N`으로 고정하고 최초 키가 해당 풀에 있으면 최대 `N - 1`회, 풀 밖에 있으면 최대 `N`회 회전할 수 있습니다. 쿨다운이 만료되거나 이후 풀이 커져도 이 한도는 충전되지 않습니다. 적용 대상인 기존 전송 예산에 따라 재시도가 더 일찍 멈출 수 있습니다. 사이드카 브리지는 자체 연결 재설정 및 동일 대상 재시도 한도를 유지하며, 이 회전 한도가 브리지에 core의 물리적 전송 예산을 새로 적용하지는 않습니다. 회전이 거부되면 마지막 429에 따른 실패 키의 쿨다운은 기록하지만 대체 키를 선택하지 않으며, 응답은 기존 오류 처리 방식으로 마무리합니다. + API 키 공급자는 리터럴 키나 환경 참조를 둘 수 있습니다. OAuth 공급자는 `ocx login`으로 채워지는 자격 증명 저장소를 사용합니다. 구독 기반 Claude Code 실행 동작은 [`claudeCode.authMode`](/reference/configuration/server/#claude-code)에서 설정합니다. ## 공급자 진단용 외부 요청 안전성 diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 9a0b822f52..428c5a3e9b 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -199,7 +199,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `noJsonSchemaModels?` | `string[]` | Exact model IDs whose `openai-chat` endpoint rejects a `json_schema` `response_format` but still accepts `json_object`. Such a request is downgraded to `json_object` instead of being dropped, so a caller asking for JSON still gets JSON. `noStructuredOutputModels` wins when a model is on both lists. The `opencode go`, `opencode zen`, and `opencode free` presets ship this for their DeepSeek routes. | | `omitReasoningEffortWithToolsModels?` | `string[]` | Exact `openai-chat` model IDs that accept a reasoning-effort field on an ordinary turn but reject it once function tools are present. The model keeps its advertised effort ladder; OpenCodex omits the wire field for tool-bearing requests only and the upstream default applies. Narrower than `noReasoningModels`, which strips reasoning from every request and costs the model its picker entirely. | | `parallelToolCalls?` | `boolean` | Toggle parallel tool calls. OpenAI Chat defaults on; non-chat adapters advertise only on explicit `true`. | -| `terminalContinuationGuard?` | `boolean` | Opt in an `openai-chat` provider to one bounded internal re-ask when an actionable turn announces work, then cleanly stops without a tool call. Defaults to `false`; explicit `false` behaves like omission. Combo attempts and routed compaction turns are excluded, and non-`openai-chat` adapters ignore this option. | +| `terminalContinuationGuard?` | `boolean` | Opt in an `openai-chat` provider to one bounded internal re-ask when an actionable turn announces work, then cleanly stops without a tool call. The re-ask shares the request's send budget, including its one available final-recovery send; it never creates a fresh allowance. Defaults to `false`; explicit `false` behaves like omission. Combo attempts and routed compaction turns are excluded, and non-`openai-chat` adapters ignore this option. | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | Disabled-by-default downstream SSE repair for exact placeholder ids, missing terminal ids, and (with `repairInvalidIds`) message/reasoning ids missing the canonical `msg_`/`rs_` prefix. Function-call ids are never rewritten. Built-in DeepSeek enables the last two by default. | | `responsesSnapshotRepair?` | `boolean` | Disabled-by-default client-facing repair for sparse Responses lifecycle snapshots in SSE and JSON. Fills missing canonical status, output, and tool metadata while raw inspection and persistence remain unchanged. | | `webSearchBridge?` | `{ enabled?: boolean; backend?: "ollama" \| "openai" \| "anthropic" \| "xai" \| "gemini" \| "exa"; maxSearches?: number; timeoutMs?: number; endpoint?: string }` | Key-auth `openai-responses` passthrough providers only. Off by default. Codex always declares the hosted `web_search` tool, and the passthrough relays it on the assumption the destination executes it. A gateway that does not run hosted search answers with a `function_call` named `web_search` that nothing runs, and the undeclared-tool guard ends the turn. With `enabled: true` and an explicit `backend` OpenCodex intercepts that call, runs the search itself, feeds the result back to the same upstream, and shows Codex a hosted `web_search_call` cell. Never armed for `authMode: "forward"` (ChatGPT already searches) or for a provider that executes hosted search upstream. `backend` is required; there is no implicit default and a missing credential for the named backend leaves the bridge disarmed rather than falling through to another paid search. `ollama` reuses this provider's own API key on `POST /api/web_search`, so the origin must be `https://ollama.com` unless the operator names `endpoint` explicitly. `openai` / `anthropic` / `xai` / `gemini` / `exa` reuse the matching sidecar executor and that executor's own credential (`webSearchSidecar.exaApiKey` for Exa). The search model comes from `webSearchSidecar.model` only when `webSearchSidecar.backend` resolves to the same backend this bridge names; otherwise the bridge runs that backend's own default, because a model chosen for one vendor is rejected by another. An unset `webSearchSidecar.backend` resolves to `openai`, so an unset-backend model reaches an `openai` bridge and no other. There is no per-provider bridge model override. Streaming turns only. A turn that mixes `web_search` with another client tool call still fails closed rather than dropping the client's call. Assistant text such as XML-like `` prose is not executed. Defaults: `maxSearches: 3` (1..10), `timeoutMs: 60000` (1000..600000). | @@ -225,6 +225,27 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor legacy boolean, equivalent to `nativeLocalExec: "on"` only when the newer field is unset. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor local-exec policy. `off` is default; `codex-sandbox` currently fails closed like `off`. | +For translated Responses requests using a multi-key pool with `authMode: "key"` (excluding OAuth and forward-auth account pools), each routed provider invocation shares its API-key rotation allowance between initial recovery, web-search/image/video bridge rounds, and terminal continuations: at most `N - 1` rotations if the initial key belongs to the pool, or `N` if it is outside the pool, where `N` is the pool size before the first send. Cooldown expiry or later pool growth does not replenish this allowance. Existing applicable send budgets may stop retries sooner; sidecar bridges retain their own connection-reset and same-target retry limits, and this rotation cap does not impose the core's physical-send budget on those bridges. Once rotation is refused, the last 429 still records the failed key's cooldown, but no replacement key is selected and the response follows the existing error handling. Same-key `retryOn429` replays use their separate request-wide retry counter; this allowance counts key rotations only. + +When `emptyCompletionRetry` is enabled, Kiro can use the request's remaining final-recovery send +after its three base sends finish without visible output. That recovery shares any existing +credential-hop reservation and is charged once. Command Code, MiMo and Google Vertex/Antigravity +also charge each inference attempt, including internal retries, to the shared request budget. +After base sends are exhausted, a key or OAuth recovery can still use the one available final +reserve. These transports reuse the server's pacing slot for their first attempt; each internal +retry takes one new slot, while a budget-refused internal retry takes none. +401 recovery reserves its next inference before refreshing a token or selecting another key. +If no send remains, the original 401 is returned and no replacement credential is selected. +Native Codex main/stored-account one-shot refreshes also honor the shared send ceiling. +When no retry fits, they retain the prior HTTP failure instead of sending again. JWT bootstrap and +model-catalog lookup are separate from inference accounting. Ordinary Google AI Studio continues +to use the server's existing retry helper. + +A combo child refused locally before any dispatch is recorded returns its unused send reservation, +so later targets keep their remaining allowance. Successful or already-dispatched children retain +their charge; cancellation or an unknown failure after child execution starts remains conservative. +This does not promise a refund for every failure that happens before network bytes are sent. + With `webSearchBridge` enabled, a search continuation stays bound to the API-key selection that served the first request. Changing the selected key, its reference or resolved value, authentication mode, or base URL during search or provider pacing ends the turn with a bridge error before another diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index ca389d9ffd..a0f5936360 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -159,6 +159,8 @@ cross-route credential fallback не существует. Строки API GPT- | `unsafeAllowNativeLocalExec?` | `boolean` | Legacy boolean Cursor, эквивалентен `nativeLocalExec: "on"` только если новое поле не задано. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Политика local-exec для Cursor. `off` — дефолт; `codex-sandbox` сейчас ведёт себя fail-closed как `off`. | +Для преобразованных запросов Responses с пулом из нескольких API-ключей каждый вызов выбранного маршрутизацией провайдера использует общий предел переключений для первоначального восстановления, раундов мостов веб-поиска, изображений и видео, а также последующих запросов, завершающих ответ: не более `N - 1`, если начальный ключ входит в пул, или `N`, если он находится вне пула; `N` фиксируется как размер пула до первой отправки. Истечение периода ожидания или последующее расширение пула не восстанавливает этот лимит. Применимые существующие бюджеты отправки могут остановить повторы раньше; мосты sidecar сохраняют собственные пределы повторов после сброса соединения и повторных запросов к той же цели, а этот предел переключений не распространяет на них бюджет физических отправок core. Если переключение запрещено, для последнего 429 всё равно записывается период ожидания отказавшего ключа, но другой ключ не выбирается, а ответ обрабатывается существующим механизмом обработки ошибок. + Провайдеры с API-key могут хранить literal key или environment-reference. OAuth-провайдеры используют credential store, заполняемый через `ocx login`; поведение subscription-backed launcher'а Claude Code настраивается через diff --git a/docs-site/src/content/docs/tr/reference/configuration/providers.md b/docs-site/src/content/docs/tr/reference/configuration/providers.md index 3126b9046c..778175d2fd 100644 --- a/docs-site/src/content/docs/tr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/tr/reference/configuration/providers.md @@ -160,6 +160,8 @@ alanlı seçilmiş kimlikleri yalın kimliklere yeniden yazar. | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor eski boolean değeri, yalnızca daha yeni alan ayarlanmadığında `nativeLocalExec: "on"` değerine eşdeğerdir. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor yerel yürütme politikası. `off` varsayılandır; `codex-sandbox` şu anda `off` gibi kapalı olarak başarısız olur. | +Birden çok anahtar içeren havuz kullanan dönüştürülmüş Responses isteklerinde, yönlendirilen sağlayıcının her çağrısı ilk kurtarma, web araması/görüntü/video köprüsü turları ve yanıtı tamamlayan devam istekleri arasında bir API anahtarı değiştirme sınırını paylaşır: başlangıç anahtarı havuzdaysa en fazla `N - 1`, havuz dışındaysa en fazla `N` değişim yapılabilir; `N`, ilk gönderimden önceki havuz boyutuna sabitlenir. Bekleme süresinin dolması veya havuzun sonradan büyümesi bu hakkı yenilemez. Geçerli mevcut gönderim bütçeleri yeniden denemeleri daha erken durdurabilir; sidecar köprüleri bağlantı sıfırlaması ve aynı hedefe yeniden deneme için kendi sınırlarını korur ve bu anahtar değiştirme sınırı, core'un fiziksel gönderim bütçesini bu köprülere uygulamaz. Anahtar değişimi reddedildiğinde son 429 için başarısız anahtarın bekleme süresi yine kaydedilir, ancak yerine başka bir anahtar seçilmez ve yanıt mevcut hata işleme yolunu izler. + API anahtarı sağlayıcıları değişmez bir anahtar veya bir ortam referansı tutabilir. OAuth sağlayıcıları `ocx login` tarafından doldurulan kimlik bilgisi deposunu kullanır; abonelik destekli Claude Code başlatma davranışı diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index b28a5a2af4..96f58746ea 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -146,6 +146,8 @@ selector，而不是分配一个新名称。 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 旧布尔值；仅当更新字段未设置时，等同于 `nativeLocalExec: "on"`。 | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 本地执行策略。`off` 是默认值；`codex-sandbox` 目前会像 `off` 一样失败关闭。 | +对于使用多密钥池的转换后 Responses 请求，每次路由到提供商的调用，其初始恢复、网页搜索/图像/视频桥接的各轮调用以及用于完成响应的后续请求共同使用 API 密钥轮换额度：初始密钥在池中时最多轮换 `N - 1` 次，不在池中时最多轮换 `N` 次，其中 `N` 固定为首次发送前的密钥池大小。冷却期结束或随后扩大密钥池都不会补充此额度。现有且适用的发送预算可能让重试更早停止；sidecar 桥接仍保留自身的连接重置重试和同目标重试限制，此轮换额度不会将 core 的实际发送预算新增应用于这些桥接。轮换被拒绝时，仍会根据最后一个 429 记录失败密钥的冷却期，但不会选择替代密钥，响应继续按现有错误处理方式处理。 + API key 提供者可以持有字面量 key，或环境引用。OAuth 提供者使用由 `ocx login` 填充的凭据存储；基于订阅的 Claude Code 启动行为在 [`claudeCode.authMode`](/reference/configuration/server/#claude-code) 下配置。 ## 提供者诊断出站安全性 diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 0680fa35aa..5626ef27c5 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -116,6 +116,8 @@ ocx models provider openrouter on | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 舊版布林值，僅在較新欄位未設定時等同於 `nativeLocalExec: "on"`。 | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 本機執行政策。`off` 為預設；`codex-sandbox` 目前像 `off` 般 fail closed。 | +對於使用多金鑰集區的轉換後 Responses 要求，每次路由至提供者的呼叫，其初始復原、網頁搜尋/影像/影片橋接的各輪呼叫以及用於完成回應的後續要求共同使用 API 金鑰輪替額度：初始金鑰在集區中時最多輪替 `N - 1` 次，不在集區中時最多輪替 `N` 次，其中 `N` 固定為首次傳送前的金鑰池大小。冷卻期結束或之後擴大金鑰池都不會補充此額度。現有且適用的傳送預算可能讓重試更早停止；sidecar 橋接仍保留自身的連線重設重試和同目標重試限制，此輪替額度不會將 core 的實際傳送預算新增套用至這些橋接。輪替遭拒時，仍會根據最後一個 429 記錄失敗金鑰的冷卻期，但不會選取替代金鑰，回應繼續依現有錯誤處理方式處理。 + API-key 供應商可持有字面值金鑰或環境參考。OAuth 供應商使用由 `ocx login` 填入的憑證存放；訂閱支援的 Claude Code 啟動行為在 [`claudeCode.authMode`](/zh-tw/reference/configuration/server/#claude-code) 下設定。 ## 供應商診斷對外安全 diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 5dce4f8b22..67fe2de361 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -167,6 +167,7 @@ } }, "explicit": { + "physical-send.test.ts": "adapters", "responses-core-modules.test.ts": "responses", "chat-responses-control-integration.test.ts": "responses", "coding-agent-tool-result-images.test.ts": "adapters", diff --git a/src/adapters/base.ts b/src/adapters/base.ts index f4cf7ab2ef..595c853f5d 100644 --- a/src/adapters/base.ts +++ b/src/adapters/base.ts @@ -65,6 +65,8 @@ export interface ProviderAdapter { localTerminal?(parsed: OcxParsedRequest): AdapterLocalTerminal | undefined; fetchResponse?(request: AdapterRequest, ctx?: AdapterFetchContext): Promise; + /** fetchResponse admits every physical send through ctx.sendBudget when supplied. */ + fetchResponseUsesSendBudget?: boolean; /** * Parse one upstream response. `tierMetadata` is the same live observer returned on the @@ -149,7 +151,9 @@ export interface AdapterFetchContext { returnRawErrors?: boolean; /** Whether the upstream response will be consumed as a stream; adapters may select low-latency transport settings. */ stream?: boolean; - /** Custom fetch executor to use for physical upstream network requests (defaults to globalThis.fetch). */ + /** Caller-owned fetchResponse transports must use this for inference and replays when supplied. + * Retains hop confirmation and pacing; absent uses the adapter's default fetch. + * Separate credential/catalog discovery does not count as inference. */ executor?: typeof globalThis.fetch; /** * The logical request's send budget (#4546). Optional and unlimited when absent, so an diff --git a/src/adapters/command-code.ts b/src/adapters/command-code.ts index 4b7c707d7e..c80eb94ebd 100644 --- a/src/adapters/command-code.ts +++ b/src/adapters/command-code.ts @@ -13,6 +13,8 @@ import { commandCodeReasoningEfforts, refreshCommandCodeReasoningEfforts } from import { identifyRoutedModel } from "./identity"; import { buildNonOpenAIToolCatalogNudgeForTools } from "./tool-catalog-nudge"; import { parseDataUrl } from "./image"; +import { createAdapterPhysicalSend } from "./physical-send"; +import { SendBudgetExhaustedError } from "../lib/upstream-retry"; // Retain the short ids emitted by the first local integration. New requests use the live catalog's // provider-native IDs directly; this map is compatibility-only and is not a model fallback list. @@ -555,8 +557,10 @@ export function createCommandCodeAdapter(provider: OcxProviderConfig): ProviderA ...(reasoningEffort ? { reasoningLog: { effectiveEffort: reasoningEffort, wireField: "reasoning_effort" as const, wireValue: reasoningEffort } } : {}), }; }, + fetchResponseUsesSendBudget: true, async fetchResponse(request: AdapterRequest, ctx?: AdapterFetchContext): Promise { - const response = await fetchCommandCode(request, ctx, executor); + const send = createAdapterPhysicalSend(ctx, executor); + const response = await send({ url: request.url, dispatch: physical => fetchCommandCode(request, ctx, physical) }); if (response.ok) return response; const currentEffort = (() => { try { return (JSON.parse(request.body) as { params?: { reasoning_effort?: unknown } }).params?.reasoning_effort; } catch { return undefined; } @@ -577,8 +581,14 @@ export function createCommandCodeAdapter(provider: OcxProviderConfig): ProviderA if (!refreshed || refreshed.includes(currentEffort)) return response; const retry = requestWithoutReasoningEffort(request); if (!retry) return response; - try { void response.body?.cancel(); } catch { /* already closed */ } - return fetchCommandCode(retry, ctx, executor); + try { + return await send({ url: retry.url, sendClass: "repair", recovery: "reasoning-effort-downgrade", + beforeDispatch: () => { try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ } }, + dispatch: physical => fetchCommandCode(retry, ctx, physical) }); + } catch (error) { + if (error instanceof SendBudgetExhaustedError) return response; + throw error; + } }, async *parseStream(response: Response, budget: TranslatorBudget): AsyncGenerator { let sawFinish = false; diff --git a/src/adapters/google-http.ts b/src/adapters/google-http.ts index f7b90de87e..de1332dae7 100644 --- a/src/adapters/google-http.ts +++ b/src/adapters/google-http.ts @@ -1,4 +1,7 @@ import type { AdapterFetchContext, AdapterRequest } from "./base"; +import { createAdapterPhysicalSend } from "./physical-send"; +import type { SendClass } from "../lib/request-execution-budget"; +import type { AttemptRecoveryKind } from "../usage/log"; import { isQuotaExhaustedBody, retryableGoogleStatus, safeGoogleHttpErrorMessage } from "./google-errors"; import { repairGoogleInvalidRequestBody } from "./google-wire-compiler"; import { normalizeUpstreamHttpErrorResponse, readDisplaySafeErrorPayloadText } from "./upstream-http-error"; @@ -8,6 +11,8 @@ import { fetchWithAttemptDeadline, retryBackoffDelayMs, sleepWithAbort, + SendBudgetExhaustedError, + isConnectionResetError, } from "../lib/upstream-retry"; const GOOGLE_RETRY_ATTEMPTS = 3; @@ -41,18 +46,30 @@ export async function fetchGoogleWithRetry( ): Promise { const repairInvalid400 = opts.repairInvalid400 ?? true; const timeoutMs = ctx.timeoutMs ?? 200_000; - const executor = ctx.executor ?? globalThis.fetch; + const send = createAdapterPhysicalSend(ctx); let lastError: unknown; let activeRequest = request; let compatibilityReplayUsed = false; + let pendingResponse: Response | undefined; + let retryDelayMs = 0; + let sendClass: SendClass = "transient"; + let recovery: AttemptRecoveryKind | undefined; for (let attempt = 0; attempt < GOOGLE_RETRY_ATTEMPTS; attempt++) { if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); try { - const res = await fetchWithAttemptDeadline(activeRequest.url, { - method: activeRequest.method, - headers: activeRequest.headers, - body: activeRequest.body, - }, timeoutMs, ctx.abortSignal, ctx.stream, executor); + const res = await send({ url: activeRequest.url, sendClass, recovery, + beforeDispatch: async () => { + if (retryDelayMs > 0) await sleepWithAbort(retryDelayMs, ctx.abortSignal); + if (pendingResponse) cancelResponseBodyBestEffort(pendingResponse); + pendingResponse = undefined; + }, + dispatch: executor => fetchWithAttemptDeadline(activeRequest.url, { + method: activeRequest.method, headers: activeRequest.headers, body: activeRequest.body, + }, timeoutMs, ctx.abortSignal, ctx.stream, executor), + }); + retryDelayMs = 0; + sendClass = "transient"; + recovery = undefined; if (res.status === 400 && repairInvalid400 && !compatibilityReplayUsed) { let payloadText = ""; try { @@ -64,7 +81,8 @@ export async function fetchGoogleWithRetry( if (repairedBody !== undefined) { compatibilityReplayUsed = true; activeRequest = { ...activeRequest, body: repairedBody }; - cancelResponseBodyBestEffort(res); + pendingResponse = res; + sendClass = "repair"; attempt--; // The changed-request replay is separate from transient retry accounting. continue; } @@ -75,7 +93,7 @@ export async function fetchGoogleWithRetry( // A 429 may be a transient rate limit (retry) or hard quota exhaustion (do NOT retry — // it won't recover for hours and burns retries). Peek the body to tell them apart. if (res.status === 429) { - const peekTarget = ctx.returnRawErrors ? res.clone() : res; + const peekTarget = res.clone(); const peek = await readDisplaySafeErrorPayloadText(peekTarget, ctx.abortSignal); if (isQuotaExhaustedBody(peek)) { return ctx.returnRawErrors ? res : normalizeUpstreamHttpErrorResponse(res, { @@ -84,20 +102,27 @@ export async function fetchGoogleWithRetry( }); } } - cancelResponseBodyBestEffort(res); - await sleepWithAbort(retryBackoffDelayMs(attempt, { + pendingResponse = res; + recovery = res.status === 429 ? "rate-limit-429" : "transient-5xx"; + retryDelayMs = retryBackoffDelayMs(attempt, { baseDelayMs: GOOGLE_RETRY_BASE_MS, maxDelayMs: GOOGLE_RETRY_MAX_MS, headers: res.headers, - }), ctx.abortSignal); + }); } catch (err) { if (ctx.abortSignal?.aborted) throw err; + if (err instanceof SendBudgetExhaustedError) { + if (pendingResponse) return ctx.returnRawErrors ? pendingResponse : normalizeFinalGoogleError(label, pendingResponse, ctx.abortSignal); + throw err; + } lastError = err; if (attempt === GOOGLE_RETRY_ATTEMPTS - 1) throw err; - await sleepWithAbort(retryBackoffDelayMs(attempt, { + sendClass = "transient"; + recovery = isConnectionResetError(err) ? "connection-reset" : undefined; + retryDelayMs = retryBackoffDelayMs(attempt, { baseDelayMs: GOOGLE_RETRY_BASE_MS, maxDelayMs: GOOGLE_RETRY_MAX_MS, - }), ctx.abortSignal); + }); } } throw lastError ?? new Error(`${label} fetch failed`); diff --git a/src/adapters/google.ts b/src/adapters/google.ts index 8829dc0784..65bcde84cb 100644 --- a/src/adapters/google.ts +++ b/src/adapters/google.ts @@ -781,6 +781,7 @@ export function createGoogleAdapter(provider: OcxProviderConfig): ProviderAdapte // 429 rotation and raw error formatting. ...(provider.googleMode === "vertex" || provider.googleMode === "cloud-code-assist" ? { + fetchResponseUsesSendBudget: true, fetchResponse: (request: AdapterRequest, ctx?: AdapterFetchContext): Promise => (provider.googleMode === "cloud-code-assist" ? fetchAntigravityWithRetry : fetchVertexWithRetry)(request, ctx), formatErrorBody: (status: number, _headers: Headers, payloadText: string): string => diff --git a/src/adapters/kiro/adapter.ts b/src/adapters/kiro/adapter.ts index b6a374cf4c..07922525a1 100644 --- a/src/adapters/kiro/adapter.ts +++ b/src/adapters/kiro/adapter.ts @@ -312,6 +312,7 @@ export function createKiroAdapter(provider: OcxProviderConfig): ProviderAdapter ); }, + fetchResponseUsesSendBudget: true, fetchResponse(request: AdapterRequest, ctx?: AdapterFetchContext): Promise { // The normal Responses path supplies cancellation at fetch time rather than build time. // Keep it for the adapter-owned bounded continuation so cancelling the client turn aborts diff --git a/src/adapters/mimo-free.ts b/src/adapters/mimo-free.ts index 6500e023b5..eb89452272 100644 --- a/src/adapters/mimo-free.ts +++ b/src/adapters/mimo-free.ts @@ -6,6 +6,8 @@ import { recordOwnedConfigPath } from "../lib/config-ownership"; import type { OcxProviderConfig, OcxParsedRequest } from "../types"; import { createOpenAIChatAdapter } from "./openai-chat"; import type { ProviderAdapter, AdapterRequest, IncomingMeta } from "./base"; +import { createAdapterPhysicalSend } from "./physical-send"; +import { SendBudgetExhaustedError } from "../lib/upstream-retry"; const BOOTSTRAP_URL = "https://api.xiaomimimo.com/api/free-ai/bootstrap"; export const MIMO_CHAT_URL = "https://api.xiaomimimo.com/api/free-ai/openai/chat"; @@ -247,34 +249,41 @@ export function createMimoFreeAdapter(provider: OcxProviderConfig): ProviderAdap }; }, + fetchResponseUsesSendBudget: true, async fetchResponse(request: AdapterRequest, ctx): Promise { - const response = await fetch(request.url, { + const send = createAdapterPhysicalSend(ctx); + const response = await send({ url: request.url, dispatch: executor => executor(request.url, { method: request.method, redirect: "manual", headers: request.headers as Record, body: request.body, signal: ctx?.abortSignal, - }); + }) }); // Retry predicate: 401 (expired/invalid JWT) retries ONCE with a fresh token. // 403 is NOT retried — Xiaomi uses it for anti-abuse "Illegal access" and there is // no documented token-expiry signature that would mark a 403 as retryable. if (response.status === 401) { - // Drain the first response body before issuing the retry. - try { await response.body?.cancel(); } catch { /* already consumed */ } - resetMimoJwtCache(); - const freshJwt = await getMimoJwt(ctx?.abortSignal); - const retryHeaders = { - ...(request.headers as Record), - "Authorization": `Bearer ${freshJwt}`, - }; - return fetch(request.url, { + let retryHeaders = request.headers; + try { + return await send({ url: request.url, sendClass: "auth-recovery", recovery: "oauth-401", + beforeDispatch: async () => { + resetMimoJwtCache(); + const freshJwt = await getMimoJwt(ctx?.abortSignal); + retryHeaders = { ...request.headers, "Authorization": `Bearer ${freshJwt}` }; + try { void response.body?.cancel().catch(() => {}); } catch { /* already consumed */ } + }, + dispatch: executor => executor(request.url, { method: request.method, redirect: "manual", headers: retryHeaders, body: request.body, signal: ctx?.abortSignal, - }); + }) }); + } catch (error) { + if (error instanceof SendBudgetExhaustedError) return response; + throw error; + } } return response; diff --git a/src/adapters/physical-send.ts b/src/adapters/physical-send.ts new file mode 100644 index 0000000000..9f7c2a9f93 --- /dev/null +++ b/src/adapters/physical-send.ts @@ -0,0 +1,50 @@ +import type { AdapterFetchContext } from "./base"; +import type { SendClass } from "../lib/request-execution-budget"; +import type { AttemptRecoveryKind } from "../usage/log"; +import { abortError, SendBudgetExhaustedError } from "../lib/upstream-retry"; + +type PacedFetch = typeof globalThis.fetch & { + waitForPacing?: (signal?: AbortSignal) => Promise; + unpacedFetch?: typeof globalThis.fetch; +}; + +/** One ordinal sequence per adapter fetchResponse call, across all of its inference retries. + * Consumption starts at underlying executor invocation; its own later preflight may still fail. */ +export function createAdapterPhysicalSend(ctx: AdapterFetchContext = {}, fallback = globalThis.fetch) { + const executor = (ctx.executor ?? fallback) as PacedFetch; + let ordinal = 0; + return async (options: { + url: string; + sendClass?: SendClass; + recovery?: AttemptRecoveryKind; + /** Runs only after admission, e.g. backoff and cancellation of a superseded response. */ + beforeDispatch?: () => void | Promise; + dispatch: (executor: typeof globalThis.fetch) => Promise; + }): Promise => { + if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); + const decision = ctx.sendBudget?.reserveDispatch({ + sendClass: options.sendClass ?? "transient", targetKey: options.url, + }); + if (decision && !decision.allowed) throw new SendBudgetExhaustedError(options.url); + const permit = decision?.allowed ? decision.permit : undefined; + let dispatched = false; + const physicalExecutor = (async (input, init) => { + if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); + if (init?.signal?.aborted) throw abortError(init.signal); + if (dispatched || (permit && !permit.use())) throw new SendBudgetExhaustedError(options.url); + dispatched = true; + ordinal += 1; + ctx.onPhysicalSend?.({ ordinal, ...(options.recovery ? { recovery: options.recovery } : {}) }); + return (executor.unpacedFetch ?? executor)(input, init); + }) as typeof globalThis.fetch; + try { + await executor.waitForPacing?.(ctx.abortSignal); + if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); + await options.beforeDispatch?.(); + if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); + return await options.dispatch(physicalExecutor); + } finally { + permit?.release(); + } + }; +} diff --git a/src/lib/request-execution-budget.ts b/src/lib/request-execution-budget.ts index 80654b0a94..a0b868ddd8 100644 --- a/src/lib/request-execution-budget.ts +++ b/src/lib/request-execution-budget.ts @@ -109,6 +109,8 @@ export interface RequestExecutionBudget extends TransientSendBudget { readonly logicalRequestId: string; readonly policyVersion: string; readonly policy: RequestExecutionBudgetPolicy; + /** Share charged/pending sends; an exact prepaid hop may fund the child's initial send. */ + deriveScope(policy: RequestExecutionBudgetPolicy, prepaid?: SingleUseDispatchPermit): RequestExecutionBudget; reserveDispatch(intent: DispatchIntent): DispatchDecision; /** * Sends still available from the base allowance, capped by a layer's own maximum. @@ -133,49 +135,81 @@ const RESERVE_FUNDED_CLASSES: ReadonlySet = new Set([ ]); let logicalRequestSeq = 0; +interface SharedSendLedger { spent: number; pendingExternalSends: Set; } +const permitBookings = new WeakMap(); export function createRequestExecutionBudget( policy: RequestExecutionBudgetPolicy = CODEX_TEXT_GUARDED_BUDGET_POLICY, logicalRequestId?: string, ): RequestExecutionBudget { - let spent = 0; + return createBudgetScope(policy, + logicalRequestId ?? `lr-${Date.now().toString(36)}-${(logicalRequestSeq += 1).toString(36)}`, + { spent: 0, pendingExternalSends: new Set() }); +} + +function createBudgetScope( + policy: RequestExecutionBudgetPolicy, + logicalRequestId: string, + ledger: SharedSendLedger, + prepaidBooking?: object, +): RequestExecutionBudget { // Reservations whose physical send is reported by a retry helper rather than by the permit. // They are already charged; the reporter's first send settles one instead of charging again. - let pendingExternalSends = 0; let reserveSpent = false; let alternateTargetSends = 0; let targetTransitions = 0; let lastTargetKey: string | undefined; const budget: RequestExecutionBudget = { - get used(): number { return spent; }, + get used(): number { return ledger.spent; }, set used(next: number) { // The retry helpers report their real send count by assigning through this field. A // reservation taken with `countedExternally` has already booked one of those sends, so // the report settles the pending booking first and only the surplus is charged. - const delta = next - spent; + const delta = next - ledger.spent; if (delta <= 0) { - spent = Math.max(0, next); + ledger.spent = Math.max(0, next); return; } - const settled = Math.min(delta, pendingExternalSends); - pendingExternalSends -= settled; - spent += delta - settled; + const settled = Math.min(delta, ledger.pendingExternalSends.size); + for (let i = 0; i < settled; i++) { + const ticket = ledger.pendingExternalSends.values().next().value; + if (ticket) ledger.pendingExternalSends.delete(ticket); + } + ledger.spent += delta - settled; }, - logicalRequestId: logicalRequestId ?? `lr-${Date.now().toString(36)}-${(logicalRequestSeq += 1).toString(36)}`, + logicalRequestId, policyVersion: REQUEST_BUDGET_POLICY_VERSION, policy, + deriveScope(scopePolicy, prepaid) { + const booking = prepaid && permitBookings.get(prepaid); + // Intermediate scopes retain the exact prepaid booking. A dispatched child consumes + // it from the shared ledger, so sibling scopes cannot turn it into a second send. + const inherited = prepaid === undefined ? prepaidBooking : booking?.ledger === ledger ? booking.ticket : undefined; + const inheritedPolicy = prepaid === undefined && inherited && ledger.pendingExternalSends.has(inherited) + ? { ...scopePolicy, maxTotalModelSends: Math.min(scopePolicy.maxTotalModelSends, policy.maxTotalModelSends), + baseSendAllowance: Math.min(scopePolicy.baseSendAllowance, policy.baseSendAllowance), + finalRecoveryAllowance: Math.min(scopePolicy.finalRecoveryAllowance, policy.finalRecoveryAllowance) } + : scopePolicy; + return createBudgetScope(inheritedPolicy, logicalRequestId, ledger, inherited); + }, get reserveSpent() { return reserveSpent; }, get alternateTargetSends() { return alternateTargetSends; }, get targetTransitions() { return targetTransitions; }, get lastTargetKey() { return lastTargetKey; }, remainingBaseSends(cap: number): number { const capped = Number.isFinite(cap) ? Math.trunc(cap) : 0; - return Math.max(0, Math.min(capped, policy.baseSendAllowance - spent)); + const prepaid = prepaidBooking && ledger.pendingExternalSends.has(prepaidBooking) ? 1 : 0; + return Math.max(0, Math.min(capped, Math.max(prepaid, policy.baseSendAllowance - ledger.spent + prepaid))); }, reserveDispatch(intent: DispatchIntent): DispatchDecision { if (intent.replaySafe === false) return { allowed: false, reason: "not-replay-safe" }; - if (spent >= policy.maxTotalModelSends) return { allowed: false, reason: "total-exhausted" }; + // Inner adapters call even their first physical attempt "transient". The explicit hop + // permit identifies the prepaid send; the recovery class does not identify its owner. + const inherited = prepaidBooking + && ledger.pendingExternalSends.has(prepaidBooking) ? prepaidBooking : undefined; + const spentBeforeSend = ledger.spent - (inherited ? 1 : 0); + if (spentBeforeSend >= policy.maxTotalModelSends) return { allowed: false, reason: "total-exhausted" }; const changesTarget = lastTargetKey !== undefined && lastTargetKey !== intent.targetKey; const isAlternateTarget = changesTarget || intent.sendClass === "account-failover" @@ -190,7 +224,7 @@ export function createRequestExecutionBudget( // The base allowance is spent first. Only once it is gone does a recovery class reach // for the single shared reserve -- an account move and a validated rebuild cannot each // take one. - const drawsReserve = policy.baseSendAllowance - spent <= 0; + const drawsReserve = !inherited && policy.baseSendAllowance - spentBeforeSend <= 0; if (drawsReserve) { if (!RESERVE_FUNDED_CLASSES.has(intent.sendClass)) { return { allowed: false, reason: "base-allowance-exhausted" }; @@ -205,17 +239,19 @@ export function createRequestExecutionBudget( // one remaining send admitted two physical sends, which is the per-request multiplication // this budget exists to stop. Everything is booked now; `release()` is the way back. const previousTargetKey = lastTargetKey; - spent += 1; - if (intent.countedExternally === true) pendingExternalSends += 1; + const ticket = {}; + if (inherited) { + prepaidBooking = undefined; + ledger.pendingExternalSends.delete(inherited); + } else ledger.spent += 1; + if (intent.countedExternally === true) ledger.pendingExternalSends.add(ticket); if (drawsReserve) reserveSpent = true; if (isAlternateTarget) alternateTargetSends += 1; if (changesTarget) targetTransitions += 1; lastTargetKey = intent.targetKey; let settled: "open" | "used" | "released" = "open"; - return { - allowed: true, - permit: { + const permit: SingleUseDispatchPermit = { sendClass: intent.sendClass, use(): boolean { if (settled !== "open") return false; @@ -228,17 +264,21 @@ export function createRequestExecutionBudget( // An externally counted reservation the reporter already settled paid for a send // that physically happened. Refunding it would hand the request a free send back. if (intent.countedExternally === true) { - if (pendingExternalSends === 0) return; - pendingExternalSends -= 1; + if (!ledger.pendingExternalSends.has(ticket)) return; + ledger.pendingExternalSends.delete(ticket); } - spent -= 1; + if (inherited) { + ledger.pendingExternalSends.add(inherited); + prepaidBooking = inherited; + } else ledger.spent -= 1; if (drawsReserve) reserveSpent = false; if (isAlternateTarget) alternateTargetSends -= 1; if (changesTarget) targetTransitions -= 1; lastTargetKey = previousTargetKey; }, - }, }; + if (intent.countedExternally === true) permitBookings.set(permit, { ledger, ticket }); + return { allowed: true, permit }; }, }; return budget; diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index d5ccd759e3..ef35b2c505 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -363,6 +363,7 @@ function rotateKeyAfterFailure( now = Date.now(), attemptedKey?: string, attemptedSelection?: ProviderApiKeySelection, + allowRotation = true, ): OcxProviderConfig | null { const provider = config.providers[providerName]; if (!provider) return null; @@ -382,6 +383,10 @@ function rotateKeyAfterFailure( ? pool.find(entry => entry.id === attemptedSelection.entryId && entry.key === failedKey) : pool.find(entry => entry.key === failedKey); + // A spent request still records the failed key, but must not select or persist an + // unattempted replacement. Keep the fresh identity check and the changed:false path. + if (!allowRotation) return { changed: false, value: { failedId: failedEntry?.id } }; + if (freshProvider.apiKey !== failedKey) { const activeEntry = pool.find(entry => entry.key === freshProvider.apiKey); if (activeEntry && !isKeyInCooldown(providerName, activeEntry.id, now)) { @@ -410,6 +415,7 @@ function rotateKeyAfterFailure( }, attemptedSelection); if (outcome.status === "unavailable") return null; if (outcome.status === "superseded") { + if (!allowRotation) return null; // A newer manual selection (including A→B→A) owns subsequent dispatch. Reusing the // same failed key here would loop forever; preserve its original failure instead. return outcome.provider.apiKey !== failedKey ? structuredClone(outcome.provider) : null; @@ -425,6 +431,7 @@ function rotateKeyAfterFailure( keyCooldowns.set(cooldownKey(providerName, outcome.value.failedId), { cooldownUntil: now + cooldownMs }); sweepExpiredOnWrite(now); } + if (!allowRotation) return null; if ("exhaustedCount" in outcome.value) { console.warn(`[key-failover] ${providerName}: all ${outcome.value.exhaustedCount} keys in cooldown after ${failureStatus}; returning the upstream status to the client`); return null; @@ -448,8 +455,9 @@ export function rotateKeyOn429( now = Date.now(), attemptedKey?: string, attemptedSelection?: ProviderApiKeySelection, + allowRotation = true, ): OcxProviderConfig | null { - return rotateKeyAfterFailure(config, providerName, 429, retryAfterHeader, now, attemptedKey, attemptedSelection); + return rotateKeyAfterFailure(config, providerName, 429, retryAfterHeader, now, attemptedKey, attemptedSelection, allowRotation); } /** @@ -486,6 +494,8 @@ interface RotateProviderTransportOptions { attemptedKey?: string; attemptedSelection?: ProviderApiKeySelection; promptCacheKey?: string; + /** False records a proven 429 cooldown without changing the selected key or returning a retry. */ + allowRotation?: boolean; } /** @@ -507,6 +517,7 @@ export function rotateProviderTransportOn429( options.now, options.attemptedKey, options.attemptedSelection ?? routedProvider._apiKeyAttempt, + options.allowRotation, ); if (!rotated) return null; return applyRotatedTransport(providerName, routedProvider, rotated, options.promptCacheKey); @@ -517,7 +528,7 @@ export function rotateProviderTransportOn401( config: OcxConfig, providerName: string, routedProvider: OcxProviderTransport, - options: Omit = {}, + options: Omit = {}, ): OcxProviderTransport | null { const rotated = rotateKeyOn401(config, providerName, options.now, options.attemptedKey, options.attemptedSelection ?? routedProvider._apiKeyAttempt); diff --git a/src/server/responses/adapter-continuation.ts b/src/server/responses/adapter-continuation.ts index a1db9398d5..b16af1f1d2 100644 --- a/src/server/responses/adapter-continuation.ts +++ b/src/server/responses/adapter-continuation.ts @@ -15,7 +15,7 @@ import { recordAttemptCredentialSource, } from "../request-log"; import { waitForProviderRequestSlot } from "../../providers/request-pacing"; -import { providerFetch, fetchWithHeaderTimeout, safeHostLabel } from "./fetch-helpers"; +import { providerFetch, fetchWithHeaderTimeout, safeHostLabel, storedPoolReplayDispatchNotifier } from "./fetch-helpers"; import { transientRetryPolicyFor, rateLimitRetryDelayMs, @@ -24,6 +24,8 @@ import { } from "../../providers/key-failover"; import { fetchWithTransientRetry, + TRANSIENT_RETRY_MAX_ATTEMPTS, + SendBudgetExhaustedError, fetchWithResetRetry, applyUpstreamRecoveryInit, prepareSameTarget429Wait, @@ -84,9 +86,13 @@ export function createAdapterContinuations( ResponsesSendBudget, | "adapterSendBudget" | "noteAdapterPhysicalSend" - | "remainingTransientSendBudget" | "noteTransientSends" | "reserveCredentialHop" + | "pendingHopPermit" + | "keyPoolFailovers" + | "recoverySendAllowance" + | "recoveryClassFor" + | "sendBudgetExhausted" >, adapterExchange: Pick< AdapterExchange, @@ -95,6 +101,7 @@ export function createAdapterContinuations( | "rateLimitPolicy" | "rateLimitRetries" | "stallTimeoutMs" + | "keyPool429RetryAllowed" >, ) { const { options, logCtx, config } = requestContext; @@ -112,9 +119,11 @@ export function createAdapterContinuations( const { adapterSendBudget, noteAdapterPhysicalSend, - remainingTransientSendBudget, noteTransientSends, reserveCredentialHop, + recoverySendAllowance, + recoveryClassFor, + sendBudgetExhausted, } = sendBudgetState; @@ -184,18 +193,36 @@ export function createAdapterContinuations( if (transportState.activeAdapter.fetchResponse) { noteAttemptSend(logCtx.activeAttempt, continuationEstimate, replayKind); await waitForProviderRequestSlot(route.providerName, route.provider, nextParsed.modelId, upstream.signal); - return await transportState.activeAdapter.fetchResponse(builtContinuationRequest, { - abortSignal: upstream.signal, - timeoutMs: connectMs, - sendBudget: adapterSendBudget, - onPhysicalSend: send => noteAdapterPhysicalSend(continuationEstimate, send), - stream: nextParsed.stream, - executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { - dispatchOverride: oauthDispatch(builtContinuationRequest, nextParsed), - providerName: route.providerName, - modelId: nextParsed.modelId, - }), - }); + // An adapter-owned repair needs the same prepaid reserve as a credential hop. + // Acquire after shaping/pacing, reuse an existing hop, and refund only if unused. + const ownedPermit = transportState.activeAdapter.fetchResponseUsesSendBudget && adapterSendBudget + ? sendBudgetState.pendingHopPermit ?? recoverySendAllowance(adapterSendBudget.policy.baseSendAllowance, + replayKind ? recoveryClassFor(replayKind) : "repair", builtContinuationRequest.url).permit + : undefined; + const callerHop = transportState.activeAdapter.fetchResponseUsesSendBudget + ? undefined : sendBudgetState.pendingHopPermit; + try { + const replaySendBudget = adapterSendBudget && ownedPermit + ? adapterSendBudget.deriveScope({ ...adapterSendBudget.policy, finalRecoveryAllowance: 0 }, ownedPermit) + : adapterSendBudget; + return await transportState.activeAdapter.fetchResponse(builtContinuationRequest, { + abortSignal: upstream.signal, + timeoutMs: connectMs, + sendBudget: replaySendBudget, + onPhysicalSend: send => noteAdapterPhysicalSend(continuationEstimate, send), + stream: nextParsed.stream, + executor: storedPoolReplayDispatchNotifier(providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, + dispatchOverride: oauthDispatch(builtContinuationRequest, nextParsed), + providerName: route.providerName, + modelId: nextParsed.modelId, + }), callerHop ? () => { + if (!callerHop.use()) throw new SendBudgetExhaustedError(safeHostLabel(builtContinuationRequest.url)); + } : undefined), + }); + } finally { + ownedPermit?.release(); + } } // Same #1851 scope guard as the initial send: transient-5xx retry only for direct // Google AI Studio; every other adapter keeps reset-only semantics here. @@ -203,8 +230,21 @@ export function createAdapterContinuations( const fetchContinuationWithRetryPolicy = (route.provider.adapter === "google" || continuationTransientPolicy) ? fetchWithTransientRetry : fetchWithResetRetry; - return await fetchContinuationWithRetryPolicy( + const continuationCap = continuationTransientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS; + // The first terminal-guard re-ask repairs a no-tool completion even without a + // recovery log label. It shares the one final reserve with subsequent recovery. + const allowance = recoverySendAllowance(continuationCap, + replayKind ? recoveryClassFor(replayKind) : "repair", + `${route.providerName}|${route.modelId}|${replayKind ?? "terminal-continuation"}`); + let firstPermit = allowance.permit; + try { + return await fetchContinuationWithRetryPolicy( recovery => { + const permit = firstPermit; + firstPermit = undefined; + if (permit && !permit.use()) { + throw new SendBudgetExhaustedError(safeHostLabel(builtContinuationRequest.url)); + } noteAttemptSend(logCtx.activeAttempt, continuationEstimate, recovery ?? replayKind); return fetchWithHeaderTimeout( builtContinuationRequest.url, @@ -229,19 +269,19 @@ export function createAdapterContinuations( // Same request-scoped budget as the initial send and the 429/rotation refetches: // a terminal-guard continuation is another leg of ONE request, so handing it a // fresh `attempts` would let one request exceed the configured total-send ceiling. - ...(continuationTransientPolicy - ? { - attempts: remainingTransientSendBudget(continuationTransientPolicy.attempts), - onSendsConsumed: noteTransientSends, - } - : {}), + attempts: allowance.attempts, + onSendsConsumed: noteTransientSends, }, ); + } finally { + allowance.permit?.release(); + } } finally { builtContinuationRequest.releaseBodyObservation?.(); } }; while (true) { + const continuationHop = sendBudgetState.pendingHopPermit; try { const recoveryKind = nextContinuationRecoveryKind; nextContinuationRecoveryKind = undefined; @@ -253,6 +293,9 @@ export function createAdapterContinuations( yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}` }; } return; + } finally { + sendBudgetState.pendingHopPermit = undefined; + continuationHop?.release(); } // Same-target 429 wait-and-retry (opt-in `retryOn429`) before key/account failover: @@ -262,6 +305,7 @@ export function createAdapterContinuations( response.status === 429 && rateLimitPolicy !== null && adapterExchange.rateLimitRetries < rateLimitPolicy.attempts + && !sendBudgetExhausted() ) { adapterExchange.rateLimitRetries += 1; // Release unread body + heartbeat-fed wait via the shared same-target helper. @@ -308,8 +352,10 @@ export function createAdapterContinuations( now: Date.now(), attemptedKey: route.provider.apiKey, promptCacheKey: nextParsed.options.promptCacheKey, + allowRotation: adapterExchange.keyPool429RetryAllowed(), }); if (rotated) { + sendBudgetState.keyPoolFailovers += 1; try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ } route.provider = rotated; invalidateSameTargetRequest(); @@ -388,6 +434,7 @@ export function createAdapterContinuations( const hop = reserveCredentialHop( "auth-recovery", `${route.providerName}|${route.modelId}|continuation-oauth-429`, + !transportState.activeAdapter.fetchResponse || transportState.activeAdapter.fetchResponseUsesSendBudget === true, ); const nextAccountId = hop.allowed ? rotateGenericOAuthAccountOn429( @@ -418,6 +465,7 @@ export function createAdapterContinuations( sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, transportState.activeAdapter.name, logCtx.accountLogLabel); recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, transportState.activeAdapter.name); nextContinuationRecoveryKind = "oauth-account-429"; + sendBudgetState.pendingHopPermit = hop.permit; continue; } } catch { diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index 57eadf6a28..2b339bc7fe 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -29,6 +29,7 @@ import { } from "../../providers/key-failover"; import { fetchWithTransientRetry, + TRANSIENT_RETRY_MAX_ATTEMPTS, fetchWithResetRetry, applyUpstreamRecoveryInit, SendBudgetExhaustedError, @@ -127,6 +128,9 @@ export async function prepareAdapterExchange( | "recoveryClassFor" | "sendBudgetExhausted" | "reserveCredentialHop" + | "pendingHopPermit" + | "keyPoolFailovers" + | "keyPool429RetryAllowed" >, ) { const { options, config, logCtx, req } = requestContext; @@ -171,6 +175,7 @@ export async function prepareAdapterExchange( ? Math.floor(config.stallTimeoutSec * 1000) : 300_000; transportState.activeAdapter = transportState.adapter; + const keyPool429RetryAllowed = () => sendBudgetState.keyPool429RetryAllowed(transportState.activeAdapter.fetchResponseUsesSendBudget === true); // One immutable, body-safe outbound request per same-target sequence (URL, serialized body, // auth headers, generated compat headers). Same-target 429 replays reuse it verbatim; the @@ -281,7 +286,8 @@ export async function prepareAdapterExchange( onPhysicalSend: send => noteAdapterPhysicalSend(inputTokenEstimate, send), stream: parsed.stream, executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { - dispatchOverride: oauthDispatch(builtInitialRequest), + pacingSlotAcquired: true, + dispatchOverride: oauthDispatch(builtInitialRequest), providerName: route.providerName, modelId: route.modelId, }), @@ -315,15 +321,9 @@ export async function prepareAdapterExchange( { abortSignal: upstream.signal, label: safeHostLabel(builtInitialRequest.url), - ...(transientPolicy - // Draws the remainder, not the raw policy. A combo child inherits the parent's - // holder but used to take a fresh full allowance on its own first send, so the - // shared counter was inherited without ever being read as a limit. - ? { - attempts: remainingTransientSendBudget(transientPolicy.attempts), - onSendsConsumed: noteTransientSends, - } - : {}), + // Count both retry policies; reset-only still hops immediately on HTTP 5xx. + attempts: remainingTransientSendBudget(transientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS), + onSendsConsumed: noteTransientSends, }, ); } @@ -376,6 +376,7 @@ export async function prepareAdapterExchange( * a permit confirmed earlier would keep the charge for a send that never happened. */ onDispatch?: () => void, + replaySendBudget = adapterSendBudget, ): Promise => { let retryRequest: AdapterRequest; if (transportState.sameTargetRequest !== undefined && transportState.sameTargetParsed === parsed && transportState.sameTargetToken === transportState.transportToken) { @@ -416,24 +417,33 @@ export async function prepareAdapterExchange( try { try { if (transportState.activeAdapter.fetchResponse) { - await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal); - // The dispatch boundary is HERE, not before the pacing wait: that wait can reject for - // an abort, a saturated queue, an expired slot or a removed provider, and none of - // those reach the wire. Confirming earlier would hold the charge for a send that the - // pacer refused. - onDispatch?.(); - return await transportState.activeAdapter.fetchResponse(retryRequest, { - abortSignal: upstream.signal, - timeoutMs: connectMs, - sendBudget: adapterSendBudget, - onPhysicalSend: send => noteAdapterPhysicalSend(retryEstimate, send), - stream: parsed.stream, - executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { - dispatchOverride: oauthDispatch(retryRequest), - providerName: route.providerName, - modelId: route.modelId, - }), - }); + // Existing derived OAuth scopes already own their booking. Otherwise an adapter + // recovery can fund its first send from the shared reserve, like a continuation. + const ownedPermit = transportState.activeAdapter.fetchResponseUsesSendBudget && adapterSendBudget + && replaySendBudget === adapterSendBudget + ? recoverySendAllowance(adapterSendBudget.policy.baseSendAllowance, recoveryClassFor(recovery), retryRequest.url).permit + : undefined; + try { + await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal); + // Confirm only after pacing: an abort, rejected queue or removed provider + // has not reached the executor and must not consume the reservation. + onDispatch?.(); + return await transportState.activeAdapter.fetchResponse(retryRequest, { + abortSignal: upstream.signal, + timeoutMs: connectMs, + sendBudget: ownedPermit && adapterSendBudget + ? adapterSendBudget.deriveScope({ ...adapterSendBudget.policy, finalRecoveryAllowance: 0 }, ownedPermit) + : replaySendBudget, + onPhysicalSend: send => noteAdapterPhysicalSend(retryEstimate, send), + stream: parsed.stream, + executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, + dispatchOverride: oauthDispatch(retryRequest), + providerName: route.providerName, + modelId: route.modelId, + }), + }); + } finally { ownedPermit?.release(); } } // #2643 review: this leg used to call fetchWithHeaderTimeout directly, so an // opted-in provider's transient-5xx policy applied to the initial send and to @@ -448,17 +458,18 @@ export async function prepareAdapterExchange( // Same rule as the passthrough rebuild: spend the base allowance first, then the one // shared final-recovery reserve, so a recovery that follows a spent streak still gets // its single send instead of dying at three. - const refetchAllowance = refetchTransientPolicy - ? recoverySendAllowance( - refetchTransientPolicy.attempts, - recoveryClassFor(recovery), - `${route.providerName}|${route.modelId}|${recovery}`, - ) - : undefined; + const refetchAllowance = recoverySendAllowance( + refetchTransientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS, + recoveryClassFor(recovery), + `${route.providerName}|${route.modelId}|${recovery}`, + ); + let firstPermit = refetchAllowance.permit; try { return await refetchWithPolicy( recoveryKind => { - if (refetchAllowance?.permit && !refetchAllowance.permit.use()) { + const permit = firstPermit; + firstPermit = undefined; + if (permit && !permit.use()) { throw new SendBudgetExhaustedError(safeHostLabel(retryRequest.url)); } // Same boundary on the helper path: the thunk is what reaches the wire, and it @@ -503,6 +514,16 @@ export async function prepareAdapterExchange( return { failed: formatErrorResponse(502, "upstream_error", msg) }; } }; + const rebuildWithCredentialHop = (recovery: AttemptRecoveryKind, permit: ReturnType["permit"]) => { + const adapterSettlesHop = transportState.activeAdapter.fetchResponseUsesSendBudget === true; + sendBudgetState.pendingHopPermit = transportState.activeAdapter.fetchResponse ? undefined : permit; + const replayBudget = adapterSettlesHop && adapterSendBudget && permit + ? adapterSendBudget.deriveScope({ ...adapterSendBudget.policy, finalRecoveryAllowance: 0 }, permit) + : adapterSendBudget; + return rebuildAndRefetch(recovery, + transportState.activeAdapter.fetchResponse && !adapterSettlesHop ? () => { permit?.use(); } : undefined, + replayBudget); + }; // Keep recovery kinds in sync with the native Responses `passthroughRecovery:` loop above. recovery: for (;;) { if ( @@ -510,58 +531,65 @@ export async function prepareAdapterExchange( && isOAuth401ReplayProvider && transportState.sentOAuthSnapshot && !oauth401ReplayAttempted - && !sendBudgetExhausted() ) { - oauth401ReplayAttempted = true; - try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } - let refreshed: OAuthAccessSnapshot; + const hop = reserveCredentialHop("auth-recovery", transportState.sameTargetRequest?.url ?? builtInitialRequest.url, + !transportState.activeAdapter.fetchResponse || transportState.activeAdapter.fetchResponseUsesSendBudget === true); + if (!hop.allowed || (!hop.permit && sendBudgetExhausted())) break recovery; try { - refreshed = await refreshResolvedOAuthSelection(transportState.sentOAuthSnapshot); - } catch (err) { - cleanupUpstreamAbort(); - return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(err)); - } - if (route.provider.googleMode === "cloud-code-assist" && !refreshed.projectId) { - cleanupUpstreamAbort(); - return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(new Error("Cloud Code Assist project is required"))); - } - transportState.sentOAuthSnapshot = refreshed; - transportState.replayOAuthCredentialSnapshot = { - accountId: refreshed.accountId, - generation: refreshed.generation, - }; - if (route.providerName === "kiro") { - parsed._kiroAuthContext = { ...(refreshed.kiro ?? {}) }; + oauth401ReplayAttempted = true; + try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } + let refreshed: OAuthAccessSnapshot; + try { + refreshed = await refreshResolvedOAuthSelection(transportState.sentOAuthSnapshot); + } catch (err) { + cleanupUpstreamAbort(); + return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(err)); + } + if (route.provider.googleMode === "cloud-code-assist" && !refreshed.projectId) { + cleanupUpstreamAbort(); + return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(new Error("Cloud Code Assist project is required"))); + } + transportState.sentOAuthSnapshot = refreshed; + transportState.replayOAuthCredentialSnapshot = { + accountId: refreshed.accountId, + generation: refreshed.generation, + }; + if (route.providerName === "kiro") { + parsed._kiroAuthContext = { ...(refreshed.kiro ?? {}) }; + } + const refreshedProvider = resolveProviderTransport( + route.providerName, + { + ...route.provider, + apiKey: refreshed.accessToken, + ...(refreshed.projectId ? { project: refreshed.projectId } : {}), + }, + parsed.options.promptCacheKey, + route.providerName === "github-copilot" + ? resolveCopilotApiBaseUrl(refreshed.apiBaseUrl) + : undefined, + ); + route.provider = refreshedProvider; + invalidateSameTargetRequest(); + transportState.activeAdapter = resolveSelectionAdapter( + resolveWireProtocolOverride(route.providerName, route.modelId, refreshedProvider, inboundWire), + config.cacheRetention, + ); + bindRouteReasoningReplayScope({ + parsed, + providerName: route.providerName, + provider: refreshedProvider, + adapterName: transportState.activeAdapter.name, + oauthCredentialSnapshot: transportState.replayOAuthCredentialSnapshot, + }); + const result = await rebuildWithCredentialHop("oauth-401", hop.permit); + if ("failed" in result) return result.failed; + upstreamResponse = result; + continue recovery; + } finally { + sendBudgetState.pendingHopPermit = undefined; + hop.permit?.release(); } - const refreshedProvider = resolveProviderTransport( - route.providerName, - { - ...route.provider, - apiKey: refreshed.accessToken, - ...(refreshed.projectId ? { project: refreshed.projectId } : {}), - }, - parsed.options.promptCacheKey, - route.providerName === "github-copilot" - ? resolveCopilotApiBaseUrl(refreshed.apiBaseUrl) - : undefined, - ); - route.provider = refreshedProvider; - invalidateSameTargetRequest(); - transportState.activeAdapter = resolveSelectionAdapter( - resolveWireProtocolOverride(route.providerName, route.modelId, refreshedProvider, inboundWire), - config.cacheRetention, - ); - bindRouteReasoningReplayScope({ - parsed, - providerName: route.providerName, - provider: refreshedProvider, - adapterName: transportState.activeAdapter.name, - oauthCredentialSnapshot: transportState.replayOAuthCredentialSnapshot, - }); - const result = await rebuildAndRefetch("oauth-401"); - if ("failed" in result) return result.failed; - upstreamResponse = result; - continue recovery; } // Static API-key pools can recover a credential-scoped 401 without abandoning the @@ -569,30 +597,38 @@ export async function prepareAdapterExchange( // refresh above and never enter here — `hasKeyPoolFailover` rejects oauth/forward modes. // Runs after the OAuth replay so a refreshable token is never treated as a dead key. while (upstreamResponse.status === 401 && hasKeyPoolFailover(route.provider)) { - const rotated = rotateProviderTransportOn401(config, route.providerName, route.provider, { - now: Date.now(), - attemptedKey: route.provider.apiKey, - promptCacheKey: parsed.options.promptCacheKey, - }); - if (!rotated) break; - // Release the failed response's socket before retrying; unread bodies otherwise linger - // until runtime cleanup (one per rotated key). - try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } - route.provider = rotated; - invalidateSameTargetRequest(); - transportState.activeAdapter = resolveSelectionAdapter( - resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire), - config.cacheRetention, - ); - bindRouteReasoningReplayScope({ - parsed, - providerName: route.providerName, - provider: route.provider, - adapterName: transportState.activeAdapter.name, - }); - const result = await rebuildAndRefetch("key-401"); - if ("failed" in result) return result.failed; - upstreamResponse = result; + const hop = reserveCredentialHop("auth-recovery", transportState.sameTargetRequest?.url ?? builtInitialRequest.url, + !transportState.activeAdapter.fetchResponse || transportState.activeAdapter.fetchResponseUsesSendBudget === true); + if (!hop.allowed || (!hop.permit && sendBudgetExhausted())) break; + try { + const rotated = rotateProviderTransportOn401(config, route.providerName, route.provider, { + now: Date.now(), + attemptedKey: route.provider.apiKey, + promptCacheKey: parsed.options.promptCacheKey, + }); + if (!rotated) break; + // Release the failed response's socket before retrying; unread bodies otherwise linger + // until runtime cleanup (one per rotated key). + try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } + route.provider = rotated; + invalidateSameTargetRequest(); + transportState.activeAdapter = resolveSelectionAdapter( + resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire), + config.cacheRetention, + ); + bindRouteReasoningReplayScope({ + parsed, + providerName: route.providerName, + provider: route.provider, + adapterName: transportState.activeAdapter.name, + }); + const result = await rebuildWithCredentialHop("key-401", hop.permit); + if ("failed" in result) return result.failed; + upstreamResponse = result; + } finally { + sendBudgetState.pendingHopPermit = undefined; + hop.permit?.release(); + } } // Same-target 429 wait-and-retry (opt-in `retryOn429`, issue #487). Codex never retries @@ -644,8 +680,10 @@ export async function prepareAdapterExchange( now: Date.now(), attemptedKey: route.provider.apiKey, promptCacheKey: parsed.options.promptCacheKey, + allowRotation: keyPool429RetryAllowed(), }); if (!rotated) break; + sendBudgetState.keyPoolFailovers += 1; // Release the failed response's socket before retrying; unread bodies otherwise linger // until runtime cleanup (one per rotated key under a rate-limit storm). try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } @@ -725,6 +763,7 @@ export async function prepareAdapterExchange( const hop = reserveCredentialHop( "auth-recovery", `${route.providerName}|${route.modelId}|adapter-recovery-oauth-429`, + !transportState.activeAdapter.fetchResponse || transportState.activeAdapter.fetchResponseUsesSendBudget === true, ); if (!hop.allowed) break; const nextAccountId = rotateGenericOAuthAccountOn429( @@ -755,22 +794,27 @@ export async function prepareAdapterExchange( ); sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, transportState.activeAdapter.name, logCtx.accountLogLabel); recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, transportState.activeAdapter.name); - // Confirm at the dispatch boundary, not here: a rebuild can fail while shaping the - // request and return `{ failed }` without reaching the wire, and a permit confirmed - // before that would hold the charge for a send that never happened. - const result = await rebuildAndRefetch("oauth-account-429", () => { hop.permit?.use(); }); - if ("failed" in result) { - // A no-op if the boundary was reached; a refund if the rebuild died before it. + // Budget-aware adapters spend this same booking at their physical-send boundary. + // Other fetchResponse implementations retain caller-owned confirmation after pacing. + sendBudgetState.pendingHopPermit = transportState.activeAdapter.fetchResponse ? undefined : hop.permit; + const adapterSettlesHop = transportState.activeAdapter.fetchResponseUsesSendBudget === true; + const replaySendBudget = adapterSettlesHop && adapterSendBudget && hop.permit + ? adapterSendBudget.deriveScope({ ...adapterSendBudget.policy, finalRecoveryAllowance: 0 }, hop.permit) + : adapterSendBudget; + try { + const result = await rebuildAndRefetch("oauth-account-429", + transportState.activeAdapter.fetchResponse && !adapterSettlesHop ? () => { hop.permit?.use(); } : undefined, + replaySendBudget); + if ("failed" in result) return result.failed; + upstreamResponse = result; + } finally { + sendBudgetState.pendingHopPermit = undefined; + // Idempotent: refund only an unused reservation, never a dispatched send. hop.permit?.release(); - return result.failed; } - upstreamResponse = result; } catch { - // A throw before the send — snapshot fetch, credential application, adapter - // resolution — must hand the reservation back. Without this the ladder charges the - // request for a send it never made, and a later recovery in the same request is - // refused on an allowance nothing spent. release() is idempotent and a no-op once - // used, so a throw from the rebuild keeps its charge. + // Snapshot, credential and adapter-resolution failures may precede dispatch. + // Both adapter and retry-helper paths must refund an unused reservation. hop.permit?.release(); break; } @@ -954,6 +998,7 @@ export async function prepareAdapterExchange( return { upstream, + keyPool429RetryAllowed, cleanupUpstreamAbort, connectMs, stallTimeoutMs, diff --git a/src/server/responses/combo-send-budget.ts b/src/server/responses/combo-send-budget.ts new file mode 100644 index 0000000000..7ec4b94539 --- /dev/null +++ b/src/server/responses/combo-send-budget.ts @@ -0,0 +1,81 @@ +import { CODEX_TEXT_GUARDED_BUDGET_POLICY, type RequestExecutionBudget, type RequestExecutionBudgetPolicy, type SingleUseDispatchPermit } from "../../lib/request-execution-budget"; + +/** + * Sends one combo target may run on its own before the ladder moves on. A target is a whole + * request as far as its own provider is concerned, so this is the guarded profile's base + * allowance rather than a separate number to keep in sync. + */ +export const COMBO_TARGET_BASE_SENDS = CODEX_TEXT_GUARDED_BUDGET_POLICY.baseSendAllowance; + +/** + * A combo's execution policy is DECLARED by the combo, not inherited from the single-target + * profile. + * + * `maxTargetTransitions: 1` and `maxAlternateTargetSends: 1` describe an account move, and + * applying them to a combo would refuse the second hop of a three-target combo -- which is why + * combo was left off `reserveDispatch` when the per-request split landed. The transitions a + * combo may make are exactly the targets it declares minus the one it starts on. What stays + * capped is the TOTAL: the first target's full ladder, one send for every further declared + * target, and the one shared final-recovery reserve. A one-target combo reduces to the guarded + * profile exactly, and a three-target combo whose every target fails hard reaches upstream six + * times instead of the twelve #4546 measured. + */ +export function comboExecutionBudgetPolicy(declaredTargets: number): RequestExecutionBudgetPolicy { + const targets = Math.max(1, Math.trunc(declaredTargets)); + const hops = targets - 1; + const reserve = CODEX_TEXT_GUARDED_BUDGET_POLICY.finalRecoveryAllowance; + const total = COMBO_TARGET_BASE_SENDS + hops + reserve; + return { + maxTotalModelSends: total, + baseSendAllowance: total - reserve, + finalRecoveryAllowance: reserve, + maxAlternateTargetSends: Math.max(1, hops), + maxTargetTransitions: Math.max(1, hops), + }; +} + +/** + * A budget scope that keeps its own recovery ledgers but spends the SAME request-wide counter. + * + * The factory shares both charged sends and pending external reports. Forwarding `used` alone + * cannot share reservation checks held in the factory closure, and would let each target + * refill its allowance. The reserve, alternate-target and transition ledgers stay + * per-scope on purpose: a combo target's account failover is its own recovery decision, while + * the request total still bounds every target together. + */ +export function deriveSendBudgetScope( + parent: RequestExecutionBudget, + policy: RequestExecutionBudgetPolicy, + prepaid?: SingleUseDispatchPermit, +): RequestExecutionBudget { + return parent.deriveScope(policy, prepaid); +} + +/** + * The ladder one combo target may run, expressed as an allowance on the request-wide counter. + * + * `used + COMBO_TARGET_BASE_SENDS` gives this target its own ladder from wherever the request + * already stands, and the clamp holds back one send for each target still declared after it: a + * first target that 5xx-streaks must not eat the send the last declared target is entitled to. + * That guarantee is the difference between a per-target policy and a shared pool the first + * target drains. + */ +export function comboTargetSendBudget( + comboScope: RequestExecutionBudget, + targetsDeclaredAfterThisOne: number, + prepaid?: SingleUseDispatchPermit, +): RequestExecutionBudget { + const policy = comboScope.policy; + const heldForLaterTargets = Math.max(0, targetsDeclaredAfterThisOne); + const ceiling = Math.min(policy.maxTotalModelSends, + Math.max(comboScope.used, 1, policy.maxTotalModelSends - heldForLaterTargets)); + return deriveSendBudgetScope(comboScope, { + maxTotalModelSends: ceiling, + baseSendAllowance: Math.min(ceiling, comboScope.used - (prepaid ? 1 : 0) + COMBO_TARGET_BASE_SENDS), + finalRecoveryAllowance: policy.finalRecoveryAllowance, + // Within one target the account-move shape is unchanged: three same-account sends plus one + // alternate is the recovery live traffic depends on, and a combo does not widen it. + maxAlternateTargetSends: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxAlternateTargetSends, + maxTargetTransitions: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxTargetTransitions, + }, prepaid); +} diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index c4ba12f5ef..15909b9645 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1235,6 +1235,20 @@ export async function handleResponsesCompact( } else if (quotaFailure && !storedPool401ReplayAttempted) { const fallbackModel = compactHandoffRoute(req, raw.model); if (fallbackModel && !req.signal.aborted) { + const hop = sendBudget.reserveDispatch({ + sendClass: "account-failover", + targetKey: `compact-handoff:${fallbackModel}`, + countedExternally: true, + }); + if (!hop.allowed) return buffered; + // The routed child's initial send spends this exact recovery booking. Its base + // exposes the prepaid send even after native compact used the three-send ladder; + // it shares the total ledger and receives no additional recovery reserve. + const handoffBudget = sendBudget.deriveScope({ + ...sendBudget.policy, + baseSendAllowance: Math.max(sendBudget.policy.baseSendAllowance, sendBudget.used), + finalRecoveryAllowance: 0, + }, hop.permit); const fallbackReq = new Request(req.url, { method: "POST", headers: req.headers, @@ -1250,13 +1264,15 @@ export async function handleResponsesCompact( admission, // The handoff child is the same logical compact on a second model, so it inherits // the holder. Forwarding `options` alone was not enough: the child minted its own. - { ...options, sendBudget }, + { ...options, sendBudget: handoffBudget }, ); if (fallback.ok || fallback.status === 499) return fallback; await fallback.body?.cancel().catch(() => undefined); } catch { // The previous-model rejection is the authoritative failure when the // remembered handoff route can no longer compact this thread. + } finally { + hop.permit.release(); } } } diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 1db4b9d0bf..84f687a117 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -1,14 +1,6 @@ +import { isRequestExecutionBudget } from "../../lib/request-execution-budget"; import { isDeclaredReasoningEffort } from "../../reasoning-effort"; import { recordAttemptRequestedEffort } from "../request-log"; -import { - CODEX_TEXT_GUARDED_BUDGET_POLICY, - createRequestExecutionBudget, - isRequestExecutionBudget, -} from "../../lib/request-execution-budget"; -import type { - RequestExecutionBudgetPolicy, - RequestExecutionBudget, -} from "../../lib/request-execution-budget"; import type { OcxConfig } from "../../types"; import type { RequestLogContext } from "../request-log"; import type { HandleResponsesOptions, ResponsesDispatchers, ConsumedComboFailure } from "./core-options"; @@ -68,93 +60,8 @@ import { import { preflightComboStreamResponse } from "./combo-stream-preflight"; import { streamingContextOverflowResponse, jsonContextOverflowResponse } from "./context-overflow"; -/** - * Sends one combo target may run on its own before the ladder moves on. A target is a whole - * request as far as its own provider is concerned, so this is the guarded profile's base - * allowance rather than a separate number to keep in sync. - */ -export const COMBO_TARGET_BASE_SENDS = CODEX_TEXT_GUARDED_BUDGET_POLICY.baseSendAllowance; - - -/** - * A combo's execution policy is DECLARED by the combo, not inherited from the single-target - * profile. - * - * `maxTargetTransitions: 1` and `maxAlternateTargetSends: 1` describe an account move, and - * applying them to a combo would refuse the second hop of a three-target combo -- which is why - * combo was left off `reserveDispatch` when the per-request split landed. The transitions a - * combo may make are exactly the targets it declares minus the one it starts on. What stays - * capped is the TOTAL: the first target's full ladder, one send for every further declared - * target, and the one shared final-recovery reserve. A one-target combo reduces to the guarded - * profile exactly, and a three-target combo whose every target fails hard reaches upstream six - * times instead of the twelve #4546 measured. - */ -export function comboExecutionBudgetPolicy(declaredTargets: number): RequestExecutionBudgetPolicy { - const targets = Math.max(1, Math.trunc(declaredTargets)); - const hops = targets - 1; - const reserve = CODEX_TEXT_GUARDED_BUDGET_POLICY.finalRecoveryAllowance; - const total = COMBO_TARGET_BASE_SENDS + hops + reserve; - return { - maxTotalModelSends: total, - baseSendAllowance: total - reserve, - finalRecoveryAllowance: reserve, - maxAlternateTargetSends: Math.max(1, hops), - maxTargetTransitions: Math.max(1, hops), - }; -} - - -/** - * A budget scope that keeps its own recovery ledgers but spends the SAME request-wide counter. - * - * `used` is redefined as an accessor onto the parent because the factory reads it back off this - * object -- `remainingBaseSends` and the total check both do -- so a copied number would let a - * combo target run its ladder against a stale total, which is precisely the per-layer counting - * this work exists to remove. The reserve, alternate-target and transition ledgers stay - * per-scope on purpose: a combo target's account failover is its own recovery decision, while - * the request total still bounds every target together. - */ -export function deriveSendBudgetScope( - parent: RequestExecutionBudget, - policy: RequestExecutionBudgetPolicy, -): RequestExecutionBudget { - const scope = createRequestExecutionBudget(policy, parent.logicalRequestId); - Object.defineProperty(scope, "used", { - get: () => parent.used, - set: (value: number) => { parent.used = value; }, - enumerable: true, - configurable: true, - }); - return scope; -} - - -/** - * The ladder one combo target may run, expressed as an allowance on the request-wide counter. - * - * `used + COMBO_TARGET_BASE_SENDS` gives this target its own ladder from wherever the request - * already stands, and the clamp holds back one send for each target still declared after it: a - * first target that 5xx-streaks must not eat the send the last declared target is entitled to. - * That guarantee is the difference between a per-target policy and a shared pool the first - * target drains. - */ -export function comboTargetSendBudget( - comboScope: RequestExecutionBudget, - targetsDeclaredAfterThisOne: number, -): RequestExecutionBudget { - const policy = comboScope.policy; - const heldForLaterTargets = Math.max(0, targetsDeclaredAfterThisOne); - const ceiling = Math.max(1, policy.maxTotalModelSends - heldForLaterTargets); - return deriveSendBudgetScope(comboScope, { - maxTotalModelSends: policy.maxTotalModelSends, - baseSendAllowance: Math.min(ceiling, comboScope.used + COMBO_TARGET_BASE_SENDS), - finalRecoveryAllowance: policy.finalRecoveryAllowance, - // Within one target the account-move shape is unchanged: three same-account sends plus one - // alternate is the recovery live traffic depends on, and a combo does not widen it. - maxAlternateTargetSends: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxAlternateTargetSends, - maxTargetTransitions: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxTargetTransitions, - }); -} +import { comboExecutionBudgetPolicy, deriveSendBudgetScope, comboTargetSendBudget } from "./combo-send-budget"; +export { COMBO_TARGET_BASE_SENDS, comboExecutionBudgetPolicy, deriveSendBudgetScope, comboTargetSendBudget } from "./combo-send-budget"; export async function executeComboResponses( @@ -394,17 +301,14 @@ export async function executeComboResponses( while (pick) { if (options.abortSignal?.aborted) return clientCancelledResponse(); const firstComboTarget = comboTargetsDispatched === 0; - // The first target seeds the ledger's target identity and charges nothing; every later one - // is a real transition, refused once the declared hops, the alternate-target ledger or the - // request total are spent. `countedExternally` is required: the child charges its own - // physical sends, and charging here as well would halve the cap without saying so. + // Reserve the first send without confirming dispatch. Children normally settle the booking + // through their physical-send owner; a local refusal must be able to return an unused one. const hopDecision = comboSendScope?.reserveDispatch({ sendClass: firstComboTarget ? "initial" : "combo-failover", targetKey: `${pick.target.provider}/${pick.target.model}`, countedExternally: true, }); - if (hopDecision && hopDecision.allowed) hopDecision.permit.use(); - else if (hopDecision && !firstComboTarget) { + if (hopDecision && !hopDecision.allowed && !firstComboTarget) { // Out of budget is not this target's failure. The established exhaustion contract is to // return the last real upstream answer with its status, headers and any quota body // intact rather than to mint a synthetic error, and a later target only exists because @@ -412,315 +316,344 @@ export async function executeComboResponses( if (lastFailedChildLog) adoptFailedChildLog(lastFailedChildLog); break; } - const targetSendBudget = comboSendScope - ? comboTargetSendBudget(comboSendScope, combo.targets.length - 1 - comboTargetsDispatched) - : options.sendBudget; - comboTargetsDispatched += 1; - const childLog: RequestLogContext = { - model: pick.target.model, - provider: pick.target.provider, - ...(logCtx.conversationId ? { conversationId: logCtx.conversationId } : {}), - ...(logCtx.surface ? { surface: logCtx.surface } : {}), - }; - const targetRoute = routeConcreteModel(config, `${pick.target.provider}/${pick.target.model}`); - const childBody = concreteComboRequestBody( - body, - pick.target, - comboDefaultEffort(config, comboId), - supportedLadderFor({ provider: targetRoute.provider, modelId: targetRoute.modelId }), - combo.reasoningEffortMode, - combo.defaultEffortMode, - ); - const childHeaders = buildComboChildHeaders(req.headers); - const childRequest = new Request(req.url, { - method: req.method, - headers: childHeaders, - body: JSON.stringify(childBody), - }); - linkRequestSessionLane(req, childRequest); - let resolvedAuth: CodexAuthContext | undefined; - let terminalRecorder: ((status: ResponsesTerminalStatus, httpStatusOverride?: number) => void) | undefined; - const started = Date.now(); - const attempt = beginRequestAttempt( - (logCtx.attempts?.length ?? 0) + 1, - pick.target.provider, - pick.target.model, - config.providers[pick.target.provider]!.adapter, - ); - childLog.activeAttempt = attempt; - if (originalRequestedEffort !== undefined) { - childLog.requestedEffort = originalRequestedEffort; - recordAttemptRequestedEffort(childLog); - } - let attemptRetained = false; - const retainCancelledAttempt = (): void => { - if (attemptRetained) return; - sealRequestAttemptIdentity( - attempt, - childLog.provider, - childLog.providerAdapter ?? attempt.adapter, - childLog.accountLogLabel, - ); - finishRequestAttempt(attempt, 499, Date.now() - started, childLog.usage); - (logCtx.attempts ??= []).push(attempt); - attemptRetained = true; - }; - const completedTarget = { provider: pick.target.provider, model: pick.target.model }; - const writerGeneration = pick.writerGeneration; - let consumedChildFailure: ConsumedComboFailure | undefined; - const callbackGate = createChildPassthroughCallbackGate({ - ...options, - onResponseComplete: model => { - // The live config can change while the child is streaming. Never retain credentials. - const currentCombo = getCombo(config, comboId); - const provider = config.providers[completedTarget.provider]; - if (Object.hasOwn(config.providers, completedTarget.provider) - && provider && provider.disabled !== true - && currentCombo?.targets.some(target => targetKey(target) === targetKey(completedTarget))) { - rememberComboForLane(sessionLaneIdFromRequest(req.headers), comboId, completedTarget, model, writerGeneration); - } - options.onResponseComplete?.(model); - }, - onNativePassthroughTerminal: status => { - // A committed stream can acquire terminal metadata after preflight copied - // the child log. Publish it before the outer logger finalizes, but only - // through the gate: discarded attempts must never affect the parent. - // Undefined child fields must preserve metadata already inspected by WS. - if (childLog.terminalHttpStatus !== undefined) logCtx.terminalHttpStatus = childLog.terminalHttpStatus; - if (childLog.terminalIncompleteReason !== undefined) logCtx.terminalIncompleteReason = childLog.terminalIncompleteReason; - if (childLog.terminalErrorCode !== undefined) logCtx.terminalErrorCode = childLog.terminalErrorCode; - if (childLog.upstreamError !== undefined) logCtx.upstreamError = childLog.upstreamError; - options.onNativePassthroughTerminal?.(status); - }, - }); - let response: Response; + let observedAttempt: ReturnType | undefined; + let observedChildLog: RequestLogContext | undefined; + let childEntered = false; + let returnedStatus: number | undefined; try { - const currentTargetProvider = pick.target.provider; - const deferCodexResetDerivedCooldown = combo.strategy === "failover" - && combo.targets.slice(pick.targetIndex + 1).some(target => - target.provider === currentTargetProvider - && payloadEligible(target) - && !isComboTargetInCooldown(comboId, target), + const targetSendBudget = comboSendScope + ? comboTargetSendBudget(comboSendScope, combo.targets.length - 1 - comboTargetsDispatched, hopDecision?.allowed ? hopDecision.permit : undefined) + : options.sendBudget; + comboTargetsDispatched += 1; + const childLog: RequestLogContext = { + model: pick.target.model, + provider: pick.target.provider, + ...(logCtx.conversationId ? { conversationId: logCtx.conversationId } : {}), + ...(logCtx.surface ? { surface: logCtx.surface } : {}), + }; + observedChildLog = childLog; + const targetRoute = routeConcreteModel(config, `${pick.target.provider}/${pick.target.model}`); + const childBody = concreteComboRequestBody( + body, + pick.target, + comboDefaultEffort(config, comboId), + supportedLadderFor({ provider: targetRoute.provider, modelId: targetRoute.modelId }), + combo.reasoningEffortMode, + combo.defaultEffortMode, + ); + const childHeaders = buildComboChildHeaders(req.headers); + const childRequest = new Request(req.url, { + method: req.method, + headers: childHeaders, + body: JSON.stringify(childBody), + }); + linkRequestSessionLane(req, childRequest); + let resolvedAuth: CodexAuthContext | undefined; + let terminalRecorder: ((status: ResponsesTerminalStatus, httpStatusOverride?: number) => void) | undefined; + const started = Date.now(); + const attempt = beginRequestAttempt( + (logCtx.attempts?.length ?? 0) + 1, + pick.target.provider, + pick.target.model, + config.providers[pick.target.provider]!.adapter, + ); + observedAttempt = attempt; + childLog.activeAttempt = attempt; + if (originalRequestedEffort !== undefined) { + childLog.requestedEffort = originalRequestedEffort; + recordAttemptRequestedEffort(childLog); + } + let attemptRetained = false; + const retainCancelledAttempt = (): void => { + if (attemptRetained) return; + sealRequestAttemptIdentity( + attempt, + childLog.provider, + childLog.providerAdapter ?? attempt.adapter, + childLog.accountLogLabel, ); - response = await requestDispatchers.handleResponses(childRequest, config, childLog, { + finishRequestAttempt(attempt, 499, Date.now() - started, childLog.usage); + (logCtx.attempts ??= []).push(attempt); + attemptRetained = true; + }; + const completedTarget = { provider: pick.target.provider, model: pick.target.model }; + const writerGeneration = pick.writerGeneration; + let consumedChildFailure: ConsumedComboFailure | undefined; + const callbackGate = createChildPassthroughCallbackGate({ ...options, - // After the spread: the child must run on THIS target's ladder, not on the holder the - // parent arrived with. - sendBudget: targetSendBudget, - comboAttempt: true, - comboReplaySnapshot, - deferCodexResetDerivedCooldown, - // Attempt-relative TTFT is recorded HERE (not via childLog.firstOutputMs — a later - // Object.assign(logCtx, childLog) would overwrite the request-relative value). - onFirstOutput: () => { - if (attempt.firstOutputMs === undefined) { - attempt.firstOutputMs = Math.max(0, Date.now() - started); + onResponseComplete: model => { + // The live config can change while the child is streaming. Never retain credentials. + const currentCombo = getCombo(config, comboId); + const provider = config.providers[completedTarget.provider]; + if (Object.hasOwn(config.providers, completedTarget.provider) + && provider && provider.disabled !== true + && currentCombo?.targets.some(target => targetKey(target) === targetKey(completedTarget))) { + rememberComboForLane(sessionLaneIdFromRequest(req.headers), comboId, completedTarget, model, writerGeneration); } - options.onFirstOutput?.(); + options.onResponseComplete?.(model); + }, + onNativePassthroughTerminal: status => { + // A committed stream can acquire terminal metadata after preflight copied + // the child log. Publish it before the outer logger finalizes, but only + // through the gate: discarded attempts must never affect the parent. + // Undefined child fields must preserve metadata already inspected by WS. + if (childLog.terminalHttpStatus !== undefined) logCtx.terminalHttpStatus = childLog.terminalHttpStatus; + if (childLog.terminalIncompleteReason !== undefined) logCtx.terminalIncompleteReason = childLog.terminalIncompleteReason; + if (childLog.terminalErrorCode !== undefined) logCtx.terminalErrorCode = childLog.terminalErrorCode; + if (childLog.upstreamError !== undefined) logCtx.upstreamError = childLog.upstreamError; + options.onNativePassthroughTerminal?.(status); }, - onCodexAuthContextResolved: value => { resolvedAuth = value; }, - setTerminalOutcomeRecorder: value => { terminalRecorder = value; }, - onConsumedComboFailure: value => { consumedChildFailure = value; }, - onStoredPool401ReplayDispatched: () => { storedPool401ReplayDispatched = true; }, - onNativePassthroughTerminal: callbackGate.onTerminal, - onNativePassthroughCancel: callbackGate.onCancel, - onResponseComplete: callbackGate.onResponseComplete, }); - restoreOriginalRequestedEffort(childLog); - } catch (error) { - callbackGate.discard(); + let response: Response; + try { + if (options.abortSignal?.aborted) { + callbackGate.discard(); + retainCancelledAttempt(); + return clientCancelledResponse(); + } + const currentTargetProvider = pick.target.provider; + const deferCodexResetDerivedCooldown = combo.strategy === "failover" + && combo.targets.slice(pick.targetIndex + 1).some(target => + target.provider === currentTargetProvider + && payloadEligible(target) + && !isComboTargetInCooldown(comboId, target), + ); + childEntered = true; + response = await requestDispatchers.handleResponses(childRequest, config, childLog, { + ...options, + // After the spread: the child must run on THIS target's ladder, not on the holder the + // parent arrived with. + sendBudget: targetSendBudget, + comboAttempt: true, + comboReplaySnapshot, + deferCodexResetDerivedCooldown, + // Attempt-relative TTFT is recorded HERE (not via childLog.firstOutputMs — a later + // Object.assign(logCtx, childLog) would overwrite the request-relative value). + onFirstOutput: () => { + if (attempt.firstOutputMs === undefined) { + attempt.firstOutputMs = Math.max(0, Date.now() - started); + } + options.onFirstOutput?.(); + }, + onCodexAuthContextResolved: value => { resolvedAuth = value; }, + setTerminalOutcomeRecorder: value => { terminalRecorder = value; }, + onConsumedComboFailure: value => { consumedChildFailure = value; }, + onStoredPool401ReplayDispatched: () => { storedPool401ReplayDispatched = true; }, + onNativePassthroughTerminal: callbackGate.onTerminal, + onNativePassthroughCancel: callbackGate.onCancel, + onResponseComplete: callbackGate.onResponseComplete, + }); + returnedStatus = response.status; + restoreOriginalRequestedEffort(childLog); + } catch (error) { + callbackGate.discard(); + if (options.abortSignal?.aborted) { + retainCancelledAttempt(); + return clientCancelledResponse(); + } + throw error; + } + if (options.abortSignal?.aborted) { + callbackGate.discard(); retainCancelledAttempt(); return clientCancelledResponse(); } - throw error; - } - if (options.abortSignal?.aborted) { - callbackGate.discard(); - retainCancelledAttempt(); - return clientCancelledResponse(); - } + if (response.ok && !runTurnAdapterSseResponses.has(response)) { + const nativePassthrough = isNativePassthroughSseResponse(response); + const eagerRelay = isEagerRelaySseResponse(response); + let preflight; + try { + preflight = await preflightComboStreamResponse(response, childLog); + } catch (error) { + callbackGate.discard(); + if (options.abortSignal?.aborted) { + retainCancelledAttempt(); + return clientCancelledResponse(); + } + throw error; + } + if (preflight.kind === "failed") { + callbackGate.discard(); + terminalRecorder?.("failed", preflight.response.status); + response = preflight.response; + } else { + response = preflight.response; + if (nativePassthrough) markNativePassthroughSseResponse(response); + if (eagerRelay) markEagerRelaySseResponse(response); + } + } - if (response.ok && !runTurnAdapterSseResponses.has(response)) { - const nativePassthrough = isNativePassthroughSseResponse(response); - const eagerRelay = isEagerRelaySseResponse(response); - let preflight; + if (response.ok) { + sealRequestAttemptIdentity( + attempt, + childLog.provider, + childLog.providerAdapter ?? attempt.adapter, + childLog.accountLogLabel, + ); + (logCtx.attempts ??= []).push(attempt); + attemptRetained = true; + noteComboSuccess(comboId, combo, pick.target, pick.writerGeneration); + Object.assign(logCtx, childLog, { + requestedModel, + model: requestedModel, + provider: "combo", + comboId, + routeDecision: logCtx.routeDecision, + attempts: logCtx.attempts, + activeAttempt: attempt, + activeAttemptStartedAt: started, + resolvedModel: childLog.resolvedModel ?? childLog.model, + }); + options.onCodexAuthContextResolved?.(resolvedAuth); + options.setTerminalOutcomeRecorder?.(terminalRecorder); + callbackGate.commit(); + return response; + } + + callbackGate.discard(); + if (response.status === 499) { + retainCancelledAttempt(); + return clientCancelledResponse(); + } + let failure: ConsumedComboFailure; try { - preflight = await preflightComboStreamResponse(response, childLog); + failure = consumedChildFailure + ?? await consumeComboFailure(response, options.abortSignal); } catch (error) { - callbackGate.discard(); if (options.abortSignal?.aborted) { retainCancelledAttempt(); return clientCancelledResponse(); } throw error; } - if (preflight.kind === "failed") { - callbackGate.discard(); - terminalRecorder?.("failed", preflight.response.status); - response = preflight.response; - } else { - response = preflight.response; - if (nativePassthrough) markNativePassthroughSseResponse(response); - if (eagerRelay) markEagerRelaySseResponse(response); + if (options.abortSignal?.aborted) { + retainCancelledAttempt(); + return clientCancelledResponse(); } - } - - if (response.ok) { sealRequestAttemptIdentity( attempt, childLog.provider, childLog.providerAdapter ?? attempt.adapter, childLog.accountLogLabel, ); + finishRequestAttempt( + attempt, + failure.response.status, + Date.now() - started, + failure.usage, + ); (logCtx.attempts ??= []).push(attempt); attemptRetained = true; - noteComboSuccess(comboId, combo, pick.target, pick.writerGeneration); - Object.assign(logCtx, childLog, { - requestedModel, - model: requestedModel, - provider: "combo", - comboId, - routeDecision: logCtx.routeDecision, - attempts: logCtx.attempts, - activeAttempt: attempt, - activeAttemptStartedAt: started, - resolvedModel: childLog.resolvedModel ?? childLog.model, + lastFailure = failure.response; + lastFailedChildLog = childLog; + const failureDecision = comboFailureDecision(failure.response.status, failure.classificationText, { + code: failure.upstreamCode, }); - options.onCodexAuthContextResolved?.(resolvedAuth); - options.setTerminalOutcomeRecorder?.(terminalRecorder); - callbackGate.commit(); - return response; - } - - callbackGate.discard(); - if (response.status === 499) { - retainCancelledAttempt(); - return clientCancelledResponse(); - } - let failure: ConsumedComboFailure; - try { - failure = consumedChildFailure - ?? await consumeComboFailure(response, options.abortSignal); - } catch (error) { - if (options.abortSignal?.aborted) { - retainCancelledAttempt(); - return clientCancelledResponse(); - } - throw error; - } - if (options.abortSignal?.aborted) { - retainCancelledAttempt(); - return clientCancelledResponse(); - } - sealRequestAttemptIdentity( - attempt, - childLog.provider, - childLog.providerAdapter ?? attempt.adapter, - childLog.accountLogLabel, - ); - finishRequestAttempt( - attempt, - failure.response.status, - Date.now() - started, - failure.usage, - ); - (logCtx.attempts ??= []).push(attempt); - attemptRetained = true; - lastFailure = failure.response; - lastFailedChildLog = childLog; - const failureDecision = comboFailureDecision(failure.response.status, failure.classificationText, { - code: failure.upstreamCode, - }); - const wantsStream = (rawBody as { stream?: unknown } | null)?.stream === true; - // Local byte admission has its own diagnostic; do not relabel it as an upstream refusal. - const classifyOverflow = failure.response.status === 413 - && (wantsStream || (failure.upstreamCode !== "outbound_body_too_large" - && failure.upstreamCode !== "translation_buffer_limit")); - lastFailureClassifiesOverflow = classifyOverflow; - if (storedPool401ReplayDispatched) { - if (failureDecision === "hop" && unreadableEncryptedAgentTask && !comboPayloadReadable) { - const recoveredTarget = await pickWithWait({ - exclude: pick.attempted, - eligible: target => { - try { - const route = routeConcreteModel(config, `${target.provider}/${target.model}`); - return route.codexAccountMode === undefined - && !isCanonicalOpenAiForwardProvider(route.provider); - } catch { - return false; - } - }, - }); - if (options.abortSignal?.aborted) return clientCancelledResponse(); - if (recoveredTarget && await recoverUnreadableEncryptedTask()) { - pick = recoveredTarget; - continue; + const wantsStream = (rawBody as { stream?: unknown } | null)?.stream === true; + // Local byte admission has its own diagnostic; do not relabel it as an upstream refusal. + const classifyOverflow = failure.response.status === 413 + && (wantsStream || (failure.upstreamCode !== "outbound_body_too_large" + && failure.upstreamCode !== "translation_buffer_limit")); + lastFailureClassifiesOverflow = classifyOverflow; + if (storedPool401ReplayDispatched) { + if (failureDecision === "hop" && unreadableEncryptedAgentTask && !comboPayloadReadable) { + const recoveredTarget = await pickWithWait({ + exclude: pick.attempted, + eligible: target => { + try { + const route = routeConcreteModel(config, `${target.provider}/${target.model}`); + return route.codexAccountMode === undefined + && !isCanonicalOpenAiForwardProvider(route.provider); + } catch { + return false; + } + }, + }); + if (options.abortSignal?.aborted) return clientCancelledResponse(); + if (recoveredTarget && await recoverUnreadableEncryptedTask()) { + pick = recoveredTarget; + continue; + } + if (options.abortSignal?.aborted) return clientCancelledResponse(); } - if (options.abortSignal?.aborted) return clientCancelledResponse(); - } - // Keep the spent Pool budget sticky even after a recovered routed child: - // no later failure may reopen ordinary combo/native account hopping. - adoptFailedChildLog(childLog); - if (classifyOverflow && failureDecision === "stop") { - return wantsStream - ? streamingContextOverflowResponse(requestedModel, options.translatorBudget) - : jsonContextOverflowResponse(); + // Keep the spent Pool budget sticky even after a recovered routed child: + // no later failure may reopen ordinary combo/native account hopping. + adoptFailedChildLog(childLog); + if (classifyOverflow && failureDecision === "stop") { + return wantsStream + ? streamingContextOverflowResponse(requestedModel, options.translatorBudget) + : jsonContextOverflowResponse(); + } + return lastFailure; } - return lastFailure; - } - if (failureDecision === "stop") { - adoptFailedChildLog(childLog); - if (classifyOverflow) { - return wantsStream - ? streamingContextOverflowResponse(requestedModel, options.translatorBudget) - : jsonContextOverflowResponse(); + if (failureDecision === "stop") { + adoptFailedChildLog(childLog); + if (classifyOverflow) { + return wantsStream + ? streamingContextOverflowResponse(requestedModel, options.translatorBudget) + : jsonContextOverflowResponse(); + } + return lastFailure; } - return lastFailure; - } - console.warn( - `[combo] ${comboId}: ${targetKey(pick.target)} failed with ${failure.response.status} after ${Date.now() - started}ms`, - ); - const failureNow = Date.now(); - const attemptedTargets = pick.attempted; - const nextPick = advanceComboAfterFailure(config, pick, { - retryAfter: failure.retryAfter, - resetAt: failure.resetAt, - cooldownMs: combo.cooldownMs, - now: failureNow, - cooldownScope: comboFailureCooldownScope(failure.response.status, failure.classificationText, { - code: failure.upstreamCode, - }), - eligible: payloadEligible, - status: failure.response.status, - code: failure.upstreamCode, - message: failure.classificationText, - }); - if (nextPick) { - pick = nextPick; - } else { - pick = await pickWithWait({ - exclude: pick.attempted, - eligible: payloadEligible, + console.warn( + `[combo] ${comboId}: ${targetKey(pick.target)} failed with ${failure.response.status} after ${Date.now() - started}ms`, + ); + const failureNow = Date.now(); + const attemptedTargets = pick.attempted; + const nextPick = advanceComboAfterFailure(config, pick, { + retryAfter: failure.retryAfter, + resetAt: failure.resetAt, + cooldownMs: combo.cooldownMs, now: failureNow, + cooldownScope: comboFailureCooldownScope(failure.response.status, failure.classificationText, { + code: failure.upstreamCode, + }), + eligible: payloadEligible, + status: failure.response.status, + code: failure.upstreamCode, + message: failure.classificationText, }); - } - if (!pick) { - if (options.abortSignal?.aborted) return clientCancelledResponse(); - if (unreadableEncryptedAgentTask && !comboPayloadReadable) { - const recoveredTarget = await pickWithWait({ - exclude: attemptedTargets, + if (nextPick) { + pick = nextPick; + } else { + pick = await pickWithWait({ + exclude: pick.attempted, + eligible: payloadEligible, now: failureNow, }); - if (recoveredTarget && await recoverUnreadableEncryptedTask()) { - pick = recoveredTarget; - continue; + } + if (!pick) { + if (options.abortSignal?.aborted) return clientCancelledResponse(); + if (unreadableEncryptedAgentTask && !comboPayloadReadable) { + const recoveredTarget = await pickWithWait({ + exclude: attemptedTargets, + now: failureNow, + }); + if (recoveredTarget && await recoverUnreadableEncryptedTask()) { + pick = recoveredTarget; + continue; + } + } + // Waiting or recovery may have observed cancellation after the check above. + if (options.abortSignal?.aborted) return clientCancelledResponse(); + adoptFailedChildLog(childLog); + } + } finally { + if (hopDecision?.allowed) { + // Some runTurn/sidecar transports only report an attempt or return a live stream. + // Preserve their existing conservative charge, and an entered child's ambiguous + // throw/abort. Refund preparation failures and failed children with no observed send. + if ((observedAttempt?.sendCount ?? 0) > 0 + || (observedChildLog?.activeAttempt?.sendCount ?? 0) > 0 + || observedChildLog?.attempts?.some(childAttempt => childAttempt.sendCount > 0) + || (returnedStatus !== undefined && returnedStatus >= 200 && returnedStatus < 300) + || (childEntered && (returnedStatus === undefined || returnedStatus === 499))) { + hopDecision.permit.use(); } + hopDecision.permit.release(); } - // Waiting or recovery may have observed cancellation after the check above. - if (options.abortSignal?.aborted) return clientCancelledResponse(); - adoptFailedChildLog(childLog); } } if ( diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 1bef92779a..342eec4e63 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -111,7 +111,7 @@ async function handleResponsesInner( requestState, sidecarState, ); - const sendBudgetState = createResponsesSendBudget(requestContext); + const sendBudgetState = createResponsesSendBudget(requestContext, requestState); if (sendBudgetState instanceof Response) return sendBudgetState; if ("passthrough" in transportState.adapter && transportState.adapter.passthrough && !sidecarState.routedCompaction) { return await executePassthroughResponse( diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index f5c5b94694..655a8dc6f8 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -840,12 +840,14 @@ export async function preparePassthroughExchange( recoveryClassFor(recovery), `${route.providerName}|${route.modelId}|${recovery}`, ); + let firstPermit = allowance.permit; try { return await fetchWithTransientRetry( innerRecovery => { - // Gated on the return, not fire-and-forget: a consumed permit means this leg - // already sent once, and letting the second call through would be a free send. - if (allowance.permit && !allowance.permit.use()) { + // The prepaid hop funds the first send; later attempts use the remaining base. + const permit = firstPermit; + firstPermit = undefined; + if (permit && !permit.use()) { throw new SendBudgetExhaustedError(safeHostLabel(request.url)); } noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, innerRecovery ?? recovery); @@ -886,98 +888,106 @@ export async function preparePassthroughExchange( && usesCodexForwardPoolAuth(admissionState.authCtx, route.provider) && codex401ReplayKind === null ) { - codex401ReplayKind = admissionState.authCtx.kind === "pool" ? "stored" : "main"; - try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed */ } - const poolAuthCtx = admissionState.authCtx.kind === "pool" ? admissionState.authCtx : undefined; - const poolReplay = poolAuthCtx - ? await refreshPoolForwardAuth({ req, config, route, authCtx: poolAuthCtx, substituteMainCredential, options, logCtx }) - : undefined; - const replay = poolReplay - ?? await refreshNativeMainForwardAuth({ req, config, route, authCtx: admissionState.authCtx, substituteMainCredential, options }); - if (!replay.ok) { - // Compact already records this; core historically returned without recording, - // so a dead grant stayed selectable and every request repeated the same doomed - // refresh. Fenced by the generation the 401 belongs to (#2887). - if (poolAuthCtx && poolReplay && !poolReplay.ok && poolReplay.quarantine) { - recordCodexUpstreamOutcome(config, poolAuthCtx.accountId, 401, { - threadId: poolAuthCtx.affinityKey, - fixedAccount: poolAuthCtx.fixedAccount, - modelId: route.modelId, - writerGeneration: poolAuthCtx.writerGeneration, - credentialGeneration: poolReplay.quarantineGeneration ?? poolAuthCtx.generation, - }); - } - upstream.abort(); - releaseCodexAuthContextProbeLease(admissionState.authCtx); - return replay.response; - } - admissionState.authCtx = replay.authCtx; - route.provider = replay.provider; - requestState.selectedForwardHeaders = withClaudeNativeSession(replay.headers, replay.provider, options.claudeNativeSessionId); - const replayAdapter = resolveSelectionAdapter( - resolveWireProtocolOverride(route.providerName, route.modelId, replay.provider, inboundWire), - config.cacheRetention, - ); - if (!("passthrough" in replayAdapter) || !replayAdapter.passthrough) { - upstream.abort(); - return formatErrorResponse(502, "upstream_error", "Native main refresh changed the provider wire unexpectedly"); - } - bindRouteReasoningReplayScope({ - parsed, - providerName: route.providerName, - provider: replay.provider, - adapterName: replayAdapter.name, - codexAuthContext: admissionState.authCtx, - forwardHeaders: requestState.selectedForwardHeaders, - }); - logCtx.providerAdapter = replayAdapter.name; - sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, replayAdapter.name, logCtx.accountLogLabel); - recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, replayAdapter.name); + const nativeHop = reserveCredentialHop("auth-recovery", request.url); + if (!nativeHop.allowed || (!nativeHop.permit && sendBudgetExhausted())) break passthroughRecovery; try { - request = await replayAdapter.buildRequest(parsed, { - headers: requestState.selectedForwardHeaders, - translatorBudget, - }); - refreshRequestToolAliases(request); - recordAdapterReasoning(logCtx, request); - recordAdapterTier(logCtx, request); - refreshUndeclaredToolGuard(request); - // The 401 replay rebuilds the body before sending, so it needs the same ceiling as - // every other build site; a replay is exactly when a grown payload reappears. - const replayBodyRefusal = refuseOversizedOutboundBody(request); - if (replayBodyRefusal) return replayBodyRefusal; - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, "oauth-401"); - upstreamResponse = await fetchWithHeaderTimeout( - request.url, - { method: request.method, headers: request.headers, body: request.body }, - upstream.signal, - connectMs, - parsed.stream, - // The replay-dispatched signal is what bounds the rest of this logical request, so it - // has to describe a send that actually happened. fetchWithHeaderTimeout awaits pacing - // admission BEFORE calling the executor, so signalling at the call site would spend the - // budget even when a rejected pacing wait means nothing reaches the network. Wrapping - // the executor moves the signal to the last moment before the send, where a throw from - // here on is a genuine transport attempt. - storedPoolReplayDispatchNotifier( - providerFetch(route.provider, options.codexWsRuntimeIdentity, { - dispatchOverride: oauthDispatch(request), - providerName: route.providerName, + codex401ReplayKind = admissionState.authCtx.kind === "pool" ? "stored" : "main"; + try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed */ } + const poolAuthCtx = admissionState.authCtx.kind === "pool" ? admissionState.authCtx : undefined; + const poolReplay = poolAuthCtx + ? await refreshPoolForwardAuth({ req, config, route, authCtx: poolAuthCtx, substituteMainCredential, options, logCtx }) + : undefined; + const replay = poolReplay + ?? await refreshNativeMainForwardAuth({ req, config, route, authCtx: admissionState.authCtx, substituteMainCredential, options }); + if (!replay.ok) { + // Compact already records this; core historically returned without recording, + // so a dead grant stayed selectable and every request repeated the same doomed + // refresh. Fenced by the generation the 401 belongs to (#2887). + if (poolAuthCtx && poolReplay && !poolReplay.ok && poolReplay.quarantine) { + recordCodexUpstreamOutcome(config, poolAuthCtx.accountId, 401, { + threadId: poolAuthCtx.affinityKey, + fixedAccount: poolAuthCtx.fixedAccount, modelId: route.modelId, - onCodexWsQuota: codexWsQuotaObserver(admissionState.authCtx, route.provider, route.modelId), - beforeDispatch: isCanonicalOpenAiForwardProvider(route.provider) - ? createCodexReserveDispatchGuard(admissionState.authCtx, options.codexAuthPolicy ?? config, route.modelId, options.admission, options.visionDescribeTerminal === true) : undefined, - }), - codex401ReplayKind === "stored" ? options.onStoredPool401ReplayDispatched : undefined, - ), - route.provider.authMode === "forward", - ).then(adoptObservedResponse); - } catch (err) { - return transportFailureResponse(err); - } finally { - request.releaseBodyObservation?.(); - } - continue passthroughRecovery; + writerGeneration: poolAuthCtx.writerGeneration, + credentialGeneration: poolReplay.quarantineGeneration ?? poolAuthCtx.generation, + }); + } + upstream.abort(); + releaseCodexAuthContextProbeLease(admissionState.authCtx); + return replay.response; + } + admissionState.authCtx = replay.authCtx; + route.provider = replay.provider; + requestState.selectedForwardHeaders = withClaudeNativeSession(replay.headers, replay.provider, options.claudeNativeSessionId); + const replayAdapter = resolveSelectionAdapter( + resolveWireProtocolOverride(route.providerName, route.modelId, replay.provider, inboundWire), + config.cacheRetention, + ); + if (!("passthrough" in replayAdapter) || !replayAdapter.passthrough) { + upstream.abort(); + return formatErrorResponse(502, "upstream_error", "Native main refresh changed the provider wire unexpectedly"); + } + bindRouteReasoningReplayScope({ + parsed, + providerName: route.providerName, + provider: replay.provider, + adapterName: replayAdapter.name, + codexAuthContext: admissionState.authCtx, + forwardHeaders: requestState.selectedForwardHeaders, + }); + logCtx.providerAdapter = replayAdapter.name; + sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, replayAdapter.name, logCtx.accountLogLabel); + recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, replayAdapter.name); + try { + request = await replayAdapter.buildRequest(parsed, { + headers: requestState.selectedForwardHeaders, + translatorBudget, + }); + refreshRequestToolAliases(request); + recordAdapterReasoning(logCtx, request); + recordAdapterTier(logCtx, request); + refreshUndeclaredToolGuard(request); + // The 401 replay rebuilds the body before sending, so it needs the same ceiling as + // every other build site; a replay is exactly when a grown payload reappears. + const replayBodyRefusal = refuseOversizedOutboundBody(request); + if (replayBodyRefusal) return replayBodyRefusal; + noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, "oauth-401"); + upstreamResponse = await fetchWithHeaderTimeout( + request.url, + { method: request.method, headers: request.headers, body: request.body }, + upstream.signal, + connectMs, + parsed.stream, + // The replay-dispatched signal is what bounds the rest of this logical request, so it + // has to describe a send that actually happened. fetchWithHeaderTimeout awaits pacing + // admission BEFORE calling the executor, so signalling at the call site would spend the + // budget even when a rejected pacing wait means nothing reaches the network. Wrapping + // the executor moves the signal to the last moment before the send, where a throw from + // here on is a genuine transport attempt. + storedPoolReplayDispatchNotifier( + providerFetch(route.provider, options.codexWsRuntimeIdentity, { + dispatchOverride: oauthDispatch(request), + providerName: route.providerName, + modelId: route.modelId, + onCodexWsQuota: codexWsQuotaObserver(admissionState.authCtx, route.provider, route.modelId), + beforeDispatch: isCanonicalOpenAiForwardProvider(route.provider) + ? createCodexReserveDispatchGuard(admissionState.authCtx, options.codexAuthPolicy ?? config, route.modelId, options.admission, options.visionDescribeTerminal === true) : undefined, + }), + () => { + if (nativeHop.permit && !nativeHop.permit.use()) throw new SendBudgetExhaustedError(safeHostLabel(request.url)); + if (!nativeHop.permit) noteTransientSends(1); // Legacy holders did not pre-charge a permit. + if (codex401ReplayKind === "stored") options.onStoredPool401ReplayDispatched?.(); + }, + ), + route.provider.authMode === "forward", + ).then(adoptObservedResponse); + } catch (err) { + return transportFailureResponse(err); + } finally { + request.releaseBodyObservation?.(); + } + continue passthroughRecovery; + } finally { nativeHop.permit?.release(); } } if (codex401ReplayKind !== null && upstreamResponse.status === 401) break; @@ -991,113 +1001,76 @@ export async function preparePassthroughExchange( && isOAuth401ReplayProvider && transportState.sentOAuthSnapshot && !oauth401ReplayAttempted - // Refused here, before the 401 body is cancelled: once it is gone the request can only - // answer with a synthetic 502, which would report a proxy budget decision as an upstream - // fault and throw away the credential evidence the client needs. - && !sendBudgetExhausted() ) { - oauth401ReplayAttempted = true; - try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } - let refreshed: OAuthAccessSnapshot; - try { - refreshed = await refreshResolvedOAuthSelection(transportState.sentOAuthSnapshot); - } catch (err) { - upstream.abort(); - releaseCodexAuthContextProbeLease(admissionState.authCtx); - return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(err)); - } - if (route.provider.googleMode === "cloud-code-assist" && !refreshed.projectId) { - upstream.abort(); - releaseCodexAuthContextProbeLease(admissionState.authCtx); - return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(new Error("Cloud Code Assist project is required"))); - } - transportState.sentOAuthSnapshot = refreshed; - transportState.replayOAuthCredentialSnapshot = { - accountId: refreshed.accountId, - generation: refreshed.generation, - }; - if (route.providerName === "kiro") { - parsed._kiroAuthContext = { ...(refreshed.kiro ?? {}) }; - } - const refreshedProvider = resolveProviderTransport( - route.providerName, - { - ...route.provider, - apiKey: refreshed.accessToken, - ...(refreshed.projectId ? { project: refreshed.projectId } : {}), - }, - parsed.options.promptCacheKey, - route.providerName === "github-copilot" - ? resolveCopilotApiBaseUrl(refreshed.apiBaseUrl) - : undefined, - ); - route.provider = refreshedProvider; - const refreshedAdapter = resolveSelectionAdapter( - resolveWireProtocolOverride(route.providerName, route.modelId, refreshedProvider, inboundWire), - config.cacheRetention, - ); - if (!("passthrough" in refreshedAdapter) || !refreshedAdapter.passthrough) { - upstream.abort(); - return formatErrorResponse(502, "upstream_error", "OAuth refresh changed the provider wire unexpectedly"); - } - bindRouteReasoningReplayScope({ - parsed, - providerName: route.providerName, - provider: refreshedProvider, - adapterName: refreshedAdapter.name, - oauthCredentialSnapshot: transportState.replayOAuthCredentialSnapshot, - }); - logCtx.providerAdapter = refreshedAdapter.name; - sealRequestAttemptIdentity( - logCtx.activeAttempt, - logCtx.provider, - refreshedAdapter.name, - logCtx.accountLogLabel, - ); - recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, refreshedAdapter.name); + const hop = reserveCredentialHop("auth-recovery", request.url, true); + if (!hop.allowed || (!hop.permit && sendBudgetExhausted())) break passthroughRecovery; try { - request = await refreshedAdapter.buildRequest(parsed, { - headers: requestState.selectedForwardHeaders, - translatorBudget, - }); - refreshRequestToolAliases(request); - recordAdapterReasoning(logCtx, request); - recordAdapterTier(logCtx, request); - } catch (err) { - upstream.abort(); - if (options.abortSignal?.aborted) return clientCancelledResponse(); - const msg = err instanceof Error ? err.message : String(err); - return formatErrorResponse(400, "invalid_request_error", redactSecretString(msg)); - } - refreshUndeclaredToolGuard(request); - const refreshedBodyRefusal = refuseOversizedOutboundBody(request); - if (refreshedBodyRefusal) return refreshedBodyRefusal; - try { - upstreamResponse = await fetchWithTransientRetry( - recovery => { - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, recovery ?? "oauth-401"); - return fetchWithHeaderTimeout(request.url, applyUpstreamRecoveryInit({ - method: request.method, - headers: request.headers, - body: request.body, - }, recovery), upstream.signal, connectMs, parsed.stream, - providerFetch(route.provider, options.codexWsRuntimeIdentity, { - dispatchOverride: oauthDispatch(request), - providerName: route.providerName, - modelId: route.modelId, - onCodexWsQuota: codexWsQuotaObserver(admissionState.authCtx, route.provider, route.modelId), - beforeDispatch: isCanonicalOpenAiForwardProvider(route.provider) - ? createCodexReserveDispatchGuard(admissionState.authCtx, options.codexAuthPolicy ?? config, route.modelId, options.admission, options.visionDescribeTerminal === true) : undefined, - }), - route.provider.authMode === "forward") - .then(adoptObservedResponse); + oauth401ReplayAttempted = true; + try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } + let refreshed: OAuthAccessSnapshot; + try { + refreshed = await refreshResolvedOAuthSelection(transportState.sentOAuthSnapshot); + } catch (err) { + upstream.abort(); + releaseCodexAuthContextProbeLease(admissionState.authCtx); + return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(err)); + } + if (route.provider.googleMode === "cloud-code-assist" && !refreshed.projectId) { + upstream.abort(); + releaseCodexAuthContextProbeLease(admissionState.authCtx); + return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(new Error("Cloud Code Assist project is required"))); + } + transportState.sentOAuthSnapshot = refreshed; + transportState.replayOAuthCredentialSnapshot = { + accountId: refreshed.accountId, + generation: refreshed.generation, + }; + if (route.providerName === "kiro") { + parsed._kiroAuthContext = { ...(refreshed.kiro ?? {}) }; + } + const refreshedProvider = resolveProviderTransport( + route.providerName, + { + ...route.provider, + apiKey: refreshed.accessToken, + ...(refreshed.projectId ? { project: refreshed.projectId } : {}), }, - { abortSignal: upstream.signal, label: safeHostLabel(request.url), attempts: remainingTransientSendBudget(TRANSIENT_RETRY_MAX_ATTEMPTS), onSendsConsumed: noteTransientSends }, + parsed.options.promptCacheKey, + route.providerName === "github-copilot" + ? resolveCopilotApiBaseUrl(refreshed.apiBaseUrl) + : undefined, ); - } catch (err) { - return transportFailureResponse(err); + route.provider = refreshedProvider; + const refreshedAdapter = resolveSelectionAdapter( + resolveWireProtocolOverride(route.providerName, route.modelId, refreshedProvider, inboundWire), + config.cacheRetention, + ); + if (!("passthrough" in refreshedAdapter) || !refreshedAdapter.passthrough) { + upstream.abort(); + return formatErrorResponse(502, "upstream_error", "OAuth refresh changed the provider wire unexpectedly"); + } + bindRouteReasoningReplayScope({ + parsed, + providerName: route.providerName, + provider: refreshedProvider, + adapterName: refreshedAdapter.name, + oauthCredentialSnapshot: transportState.replayOAuthCredentialSnapshot, + }); + logCtx.providerAdapter = refreshedAdapter.name; + sealRequestAttemptIdentity( + logCtx.activeAttempt, + logCtx.provider, + refreshedAdapter.name, + logCtx.accountLogLabel, + ); + recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, refreshedAdapter.name); + sendBudgetState.pendingHopPermit = hop.permit; + const result = await rebuildAndRefetch("oauth-401"); + if ("failed" in result) return result.failed; + upstreamResponse = result; } finally { - request.releaseBodyObservation?.(); + sendBudgetState.pendingHopPermit = undefined; + hop.permit?.release(); } } diff --git a/src/server/responses/request-send-budget.ts b/src/server/responses/request-send-budget.ts index c5879e106f..82c2226c31 100644 --- a/src/server/responses/request-send-budget.ts +++ b/src/server/responses/request-send-budget.ts @@ -1,4 +1,6 @@ import type { ResponsesRequestContext } from "./core-options"; +import type { PreparedResponsesRequest } from "./request-prepare"; +import { transientRetryPolicyFor } from "../../providers/key-failover"; import { createRequestExecutionBudget, isRequestExecutionBudget } from "../../lib/request-execution-budget"; import { chargeWorkflowSends, workflowSendCeilingReached } from "../../lib/workflow-budget"; import { workflowRefusalResponse } from "../workflow-refusal"; @@ -10,8 +12,16 @@ import type { SingleUseDispatchPermit, SendClass } from "../../lib/request-execu /** Owns the shared request send counter and recovery permits. */ export function createResponsesSendBudget( requestContext: Pick, + requestState: Pick, ) { const { options, req, logCtx } = requestContext; + const { route } = requestState; + // Capture before sidecar dispatch; every phase shares this rotation count. + const initialKeyPool = route.provider.apiKeyPool ?? []; + const initialKeyReference = route.provider._apiKeyAttempt?.reference ?? route.provider.apiKey; + const initialKeyIsPooled = initialKeyPool.some(entry => entry.key === initialKeyReference); + const maxKeyPoolFailovers = Math.max(0, initialKeyPool.length - (initialKeyIsPooled ? 1 : 0)); + let keyPoolFailovers = 0; // One transient-retry budget for the whole LOGICAL request, read ABOVE the passthrough branch @@ -90,12 +100,13 @@ export function createResponsesSendBudget( targetKey: string, ): { attempts: number; permit?: SingleUseDispatchPermit } => { const base = remainingTransientSendBudget(cap); - if (base > 0) return { attempts: base }; if (pendingHopPermit) { const hopPermit = pendingHopPermit; pendingHopPermit = undefined; - return { attempts: 1, permit: hopPermit }; + // The prepaid first send already reduced base; retain the unspent ordinary retries. + return { attempts: Math.min(cap, base + 1), permit: hopPermit }; } + if (base > 0) return { attempts: base }; if (!isRequestExecutionBudget(sendBudget)) return { attempts: 0 }; const decision = sendBudget.reserveDispatch({ sendClass, targetKey, countedExternally: true }); return decision.allowed ? { attempts: 1, permit: decision.permit } : { attempts: 0 }; @@ -142,7 +153,30 @@ export function createResponsesSendBudget( const recoveryClassFor = (recovery: AttemptRecoveryKind): SendClass => /401|429|oauth|rate-limit|key/.test(recovery) ? "auth-recovery" : "repair"; + const keyPool429RetryAllowed = (adapterOwnsSends: boolean): boolean => { + if (keyPoolFailovers >= maxKeyPoolFailovers) return false; + if (adapterOwnsSends && !adapterSendBudget) return true; + const policy = transientRetryPolicyFor(route.provider); + const attempts = adapterOwnsSends ? adapterSendBudget!.policy.baseSendAllowance + : policy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS; + if (!Number.isInteger(attempts) || attempts <= 0) return false; + if (remainingTransientSendBudget(attempts) > 0) return true; + if (!isRequestExecutionBudget(sendBudget)) return false; + const decision = sendBudget.reserveDispatch({ + sendClass: "auth-recovery", targetKey: adapterOwnsSends && adapterSendBudget?.lastTargetKey + ? adapterSendBudget.lastTargetKey : `${route.providerName}|${route.modelId}|key-429`, + countedExternally: true, + }); + if (!decision.allowed) return false; + decision.permit.release(); + return true; + }; + return { + maxKeyPoolFailovers, + get keyPoolFailovers() { return keyPoolFailovers; }, + set keyPoolFailovers(value: number) { keyPoolFailovers = value; }, + keyPool429RetryAllowed, workflowRootId, noteTransientSends, remainingTransientSendBudget, diff --git a/src/server/responses/sidecar-execution.ts b/src/server/responses/sidecar-execution.ts index 7987d5ca93..36af7605e6 100644 --- a/src/server/responses/sidecar-execution.ts +++ b/src/server/responses/sidecar-execution.ts @@ -74,7 +74,7 @@ export async function executeResponsesSidecars( | "notifyResponseComplete" | "cancelResponseCompletion" >, - sendBudgetState: Pick, + sendBudgetState: Pick, ) { const { config, options, logCtx } = requestContext; const { @@ -159,8 +159,10 @@ export async function executeResponsesSidecars( now: Date.now(), attemptedKey: route.provider.apiKey, promptCacheKey: parsed.options.promptCacheKey, + allowRotation: sendBudgetState.keyPoolFailovers < sendBudgetState.maxKeyPoolFailovers, }); if (rotated) { + sendBudgetState.keyPoolFailovers += 1; route.provider = rotated; } else if ( // A POSITIVE gate, not an early return. An early `return null` here made every later arm diff --git a/structure/adapters/registry.md b/structure/adapters/registry.md index 7a4d23583e..45fadd75f1 100644 --- a/structure/adapters/registry.md +++ b/structure/adapters/registry.md @@ -1,10 +1,18 @@ # Adapter Registry Authority +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](../transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + +Command Code, MiMo and Google Vertex/Antigravity admit each inference attempt against the shared budget under the [hop confirmation contract](../transports/responses.md#credential-hop-reservations); credential and catalog discovery remain separate. + Request-local adapter bindings are separate from registry authority in the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. + +Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. Response-attached WebSocket telemetry follows the [stage record identity contract](../transports/responses.md#passthrough-sse-stream-shapes-314). diff --git a/structure/catalog.md b/structure/catalog.md index e368479497..8394916233 100644 --- a/structure/catalog.md +++ b/structure/catalog.md @@ -1,10 +1,16 @@ # Model Catalog +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](transports/responses.md#credential-hop-reservations). + Catalog discovery remains separate from the Responses final-route [core module ownership](transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. CLI installation inspection reason codes, including Windows deferral, follow the [runtime inspection contract](runtime.md#lifecycle). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. + +CLI installation inspection reason codes, including Windows deferral, follow the [runtime inspection contract](runtime.md#lifecycle). Shared parsing and streaming follow the [request-copy](transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](transports/byte-accounting.md#stream-buffer-accounting) contracts. Response-attached WebSocket telemetry follows the [stage record identity contract](transports/responses.md#passthrough-sse-stream-shapes-314). diff --git a/structure/clients/claude-desktop.md b/structure/clients/claude-desktop.md index ec86ef8027..63502e306b 100644 --- a/structure/clients/claude-desktop.md +++ b/structure/clients/claude-desktop.md @@ -1,10 +1,14 @@ # Claude Desktop Integration +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](../transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + Desktop callers retain their existing ingress through the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. Codex-native model discovery follows the [shared retirement policy](../catalog.md#shared-catalog). That projection does not migrate existing user-selected Desktop configuration or usage history. diff --git a/structure/clients/integrations.md b/structure/clients/integrations.md index 3f2d3bf103..83cc894491 100644 --- a/structure/clients/integrations.md +++ b/structure/clients/integrations.md @@ -1,6 +1,6 @@ # Client Integrations -Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. +Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local. Nested compact/combo recovery follows the [prepaid send-budget contract](../transports/responses.md#bounded-api-key-429-rotation). The client-integration subsystem writes one generated OpenCodex provider contribution into a third-party client's existing config without taking ownership of the rest of that file. Its core diff --git a/structure/data-planes/images.md b/structure/data-planes/images.md index a7420072bf..1eb2b29fb1 100644 --- a/structure/data-planes/images.md +++ b/structure/data-planes/images.md @@ -1,10 +1,14 @@ # Images Data Plane +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](../transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + Vision preprocessing and image/video/search execution use the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. Hosted Responses image-tool eligibility uses the shared compatibility policy without a Codex Spark exception; standalone Images retain the separate relay contract below. See diff --git a/structure/data-planes/inbound-compat.md b/structure/data-planes/inbound-compat.md index 503a8bea80..297057b816 100644 --- a/structure/data-planes/inbound-compat.md +++ b/structure/data-planes/inbound-compat.md @@ -1,10 +1,18 @@ # Inbound Compatibility Surfaces +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](../transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + +Command Code, MiMo and Google Vertex/Antigravity admit each inference attempt against the shared budget under the [hop confirmation contract](../transports/responses.md#credential-hop-reservations); credential and catalog discovery remain separate. + Compatibility callers retain the public Responses ingress described by the [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. + +Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). ## Standalone file transcription diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 0502f6d7c4..5370d98046 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -1,10 +1,16 @@ # GUI And Management API +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](transports/responses.md#credential-hop-reservations). + The shared server request path follows the Responses [core module ownership](transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Response-attached WebSocket telemetry follows the [stage record identity contract](transports/responses.md#passthrough-sse-stream-shapes-314). Catalog HTTP acquisition follows the [proxy-routing contract](catalog.md#remote-catalog-http-proxy-routing). CLI installation inspection reason codes, including Windows deferral, follow the [runtime inspection contract](runtime.md#lifecycle). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Response-attached WebSocket telemetry follows the [stage record identity contract](transports/responses.md#passthrough-sse-stream-shapes-314). Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. + +Catalog HTTP acquisition follows the [proxy-routing contract](catalog.md#remote-catalog-http-proxy-routing). CLI installation inspection reason codes, including Windows deferral, follow the [runtime inspection contract](runtime.md#lifecycle). ## Dashboard serving diff --git a/structure/ops/docs-and-release.md b/structure/ops/docs-and-release.md index 0fbdcefc3e..f495f2b88f 100644 --- a/structure/ops/docs-and-release.md +++ b/structure/ops/docs-and-release.md @@ -5,7 +5,9 @@ Catalog HTTP acquisition follows the [proxy-routing contract](../catalog.md#remo Refresh-lock validation covers fresh unreadable locks, descriptor-matched release, path-probe failures preserving callback outcomes, and confirmed-owner unlink error handling in `tests/codex-integration/codex-account-store.test.ts`; the [catalog contract](../catalog.md#accounts-namespaces-and-pool-rotation) explicitly does not promise atomic compare-and-delete. Cooperating lock metadata changes serialize through the existing SQLite mutation transaction; release keeps the descriptor open through identity comparison and any unlink, then closes it. Failed metadata writes remove only a matching owned path after successful coordination; unknown identity, failed probes or unavailable coordination retain the path for stale recovery. Async refresh work holds no metadata transaction. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. CLI installation inspection reason codes, including Windows deferral, follow the [runtime inspection contract](../runtime.md#lifecycle). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local. Nested compact/combo recovery follows the [prepaid send-budget contract](../transports/responses.md#bounded-api-key-429-rotation). + +CLI installation inspection reason codes, including Windows deferral, follow the [runtime inspection contract](../runtime.md#lifecycle). Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. diff --git a/structure/ops/service-and-sidecars.md b/structure/ops/service-and-sidecars.md index e28575b432..69a47e80b7 100644 --- a/structure/ops/service-and-sidecars.md +++ b/structure/ops/service-and-sidecars.md @@ -1,10 +1,14 @@ # Background Service And Sidecars +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](../transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + Service endpoints are unchanged by the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. Service startup and restore use the [catalog retirement policy](../catalog.md#shared-catalog); retirement does not itself change service registration or user-selected model configuration. diff --git a/structure/overview.md b/structure/overview.md index 0151115adc..d80872ccc8 100644 --- a/structure/overview.md +++ b/structure/overview.md @@ -1,7 +1,7 @@ # Overview The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local. Nested compact/combo recovery follows the [prepaid send-budget contract](transports/responses.md#bounded-api-key-429-rotation). Shared parsing and streaming follow the [request-copy](transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](transports/byte-accounting.md#stream-buffer-accounting) contracts. diff --git a/structure/providers/chat-compat.md b/structure/providers/chat-compat.md index ea9630cb3b..8595d74d77 100644 --- a/structure/providers/chat-compat.md +++ b/structure/providers/chat-compat.md @@ -1,5 +1,11 @@ # Chat Provider Compatibility +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + +Command Code, MiMo and Google Vertex/Antigravity admit each inference attempt against the shared budget under the [hop confirmation contract](../transports/responses.md#credential-hop-reservations); credential and catalog discovery remain separate. + +HTTP adapters declare their own send-budget consumption under the [credential-hop contract](../transports/responses.md#credential-hop-reservations); ordinary Chat retry helpers retain external-send settlement. + The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Cursor's localized native-shell names follow the [routing-commentary guard contract](cursor.md#cursor-native-exec). diff --git a/structure/providers/cursor.md b/structure/providers/cursor.md index ee5356c228..f236c8d750 100644 --- a/structure/providers/cursor.md +++ b/structure/providers/cursor.md @@ -1,5 +1,11 @@ # Cursor Provider +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + +Command Code, MiMo and Google Vertex/Antigravity admit each inference attempt against the shared budget under the [hop confirmation contract](../transports/responses.md#credential-hop-reservations); credential and catalog discovery remain separate. + +HTTP adapters declare their own send-budget consumption under the [credential-hop contract](../transports/responses.md#credential-hop-reservations); Cursor's run-turn transport retains its existing ownership. + The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. diff --git a/structure/providers/xai-grok.md b/structure/providers/xai-grok.md index 2d98e7860d..55bb7d763f 100644 --- a/structure/providers/xai-grok.md +++ b/structure/providers/xai-grok.md @@ -1,10 +1,14 @@ # xAI Grok Provider +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](../transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](../transports/responses.md#credential-hop-reservations). + xAI uses the same shared credential and delivery policies through the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. Codex-native retirement is scoped to OpenAI catalog/quota evidence. Shared Responses handling retains xAI provider behavior; see diff --git a/structure/runtime.md b/structure/runtime.md index 40fe7301ae..8770ee1150 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -1,5 +1,11 @@ # Runtime +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](transports/responses.md#credential-hop-reservations). + +Command Code, MiMo and Google Vertex/Antigravity admit each inference attempt against the shared budget under the [hop confirmation contract](transports/responses.md#credential-hop-reservations); credential and catalog discovery remain separate. + Responses admission and finalization are composed through the [core module ownership](transports/responses.md#core-module-ownership). This surface retains its existing behavior. @@ -8,7 +14,9 @@ Catalog HTTP acquisition follows the [proxy-routing contract](catalog.md#remote- OAuth refresh coordination follows the [refresh-lock identity contract](catalog.md#accounts-namespaces-and-pool-rotation): a fresh unreadable lock remains held, and release requires matching descriptor identity. A failed path-identity probe preserves the refresh callback outcome. Cooperating lock metadata changes serialize through the existing SQLite mutation transaction; release keeps the descriptor open through identity comparison and any unlink, then closes it. Failed metadata writes remove only a matching owned path after successful coordination; unknown identity, failed probes or unavailable coordination retain the path for stale recovery. Async refresh work holds no metadata transaction. The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Cursor's localized native-shell names follow the [routing-commentary guard contract](providers/cursor.md#cursor-native-exec). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. + +Cursor's localized native-shell names follow the [routing-commentary guard contract](providers/cursor.md#cursor-native-exec). Chat request serialization owns the destination-scoped [OpenCode Go instruction ordering](providers/chat-compat.md#opencode-go-chronological-instructions); diff --git a/structure/subagents.md b/structure/subagents.md index 8e6e96aed9..f0eb08dc6c 100644 --- a/structure/subagents.md +++ b/structure/subagents.md @@ -1,5 +1,9 @@ # Subagents And Multi-Agent Surface +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](transports/responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](transports/responses.md#credential-hop-reservations). + Encrypted-task and fallback request handling follow the Responses [core module ownership](transports/responses.md#core-module-ownership). This surface retains its existing behavior. @@ -16,7 +20,7 @@ CLI installation inspection reason codes, including Windows deferral, follow the only explicit true on Responses ingress to the final canonical ChatGPT forward route activates it. A default top-level collaboration catalog is required. The compiler preserves caller objects, aliases the namespace and three message functions, and removes only their true encryption marker. -Declaration/reference collisions refuse the whole rewrite without changing the request. +Declaration/reference collisions refuse the whole rewrite without changing the request. Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. `src/adapters/openai-responses.ts` returns request-local alias capabilities. The Responses core refreshes them after every request rebuild and restores JSON, SSE and WebSocket identities after diff --git a/structure/transports/byte-accounting.md b/structure/transports/byte-accounting.md index 8f7967db01..98b96a9237 100644 --- a/structure/transports/byte-accounting.md +++ b/structure/transports/byte-accounting.md @@ -1,11 +1,19 @@ # Byte Accounting +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](responses.md#credential-hop-reservations). + +Command Code, MiMo and Google Vertex/Antigravity admit each inference attempt against the shared budget under the [hop confirmation contract](responses.md#credential-hop-reservations); credential and catalog discovery remain separate. + Responses body-reader limits and lifetime handling follow the [core module ownership](responses.md#core-module-ownership). This surface retains its existing behavior. How opencodex measures request and stream bytes without allocating copies solely to count them. These contracts are shared by request parsing, SSE rewriting, the provider adapters and -the translator budget, which is why so many documents link here rather than restating them. Response-attached WebSocket telemetry follows the [stage record identity contract](responses.md#passthrough-sse-stream-shapes-314). Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). +the translator budget, which is why so many documents link here rather than restating them. Response-attached WebSocket telemetry follows the [stage record identity contract](responses.md#passthrough-sse-stream-shapes-314). Generic Responses API-key failover follows the [bounded rotation contract](responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. + +Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). ## Request-copy accounting diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index d2b8553c97..492a40d0d1 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -1,10 +1,18 @@ # Transport Inventory +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](responses.md#credential-hop-reservations). + +Command Code, MiMo and Google Vertex/Antigravity admit each inference attempt against the shared budget under the [hop confirmation contract](responses.md#credential-hop-reservations); credential and catalog discovery remain separate. + The existing Responses transport is divided by responsibility in the [core module ownership](responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. + +Cursor's localized native-shell names follow the [routing-commentary guard contract](../providers/cursor.md#cursor-native-exec). The Chat adapter's [OpenCode Go instruction ordering](../providers/chat-compat.md#opencode-go-chronological-instructions) changes translated message placement only; endpoint selection and transport stay with their existing owners. diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 971695ee4e..594b322763 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -118,13 +118,7 @@ within their route; neither route falls through to the other. See ### Pre-dispatch API-key pool pick -Key-auth routes with a configured `apiKeyPoolStrategy` and two or more pool entries pick a -warm key before the first send (`selectProactiveApiKeyTransport` in -`src/providers/key-failover.ts`). The pick is inert unless that strategy is set and the -committed key is already cooling or missing from the pool: a healthy committed key, including -a manual selection, is left alone and the common path returns null without a config write. -`forgetApiKeyRotationCursor` drops the process-local round-robin cursor when the operator -edits the pool, so a later pick cannot second-guess that choice. +Key-auth routes with a configured `apiKeyPoolStrategy` and two or more pool entries pick a warm key before the first send (`selectProactiveApiKeyTransport` in `src/providers/key-failover.ts`). The pick is inert unless that strategy is set and the committed key is already cooling or missing from the pool: a healthy committed key, including a manual selection, is left alone and the common path returns null without a config write. `forgetApiKeyRotationCursor` drops the process-local round-robin cursor when the operator edits the pool, so a later pick cannot second-guess that choice. On the shared Responses path the assignment lands in `src/server/responses/core.ts` immediately before `resolveProviderTransport`. `route.provider` is copied into @@ -147,6 +141,11 @@ are not at risk on a stored row, because the config schema requires both. Reactive 429 rotation (`rotateProviderTransportOn429`) remains the recovery path after a send has already earned a throttle. +### Bounded API-key 429 rotation + +The request-owned budget in `src/server/responses/request-send-budget.ts` captures a failover allowance before sidecar or adapter dispatch: the initial pool size minus one when the initially selected key reference belongs to that pool, or the full initial pool size when it is outside the pool and shares that failover count across ordinary recovery, web-search/image/video bridge rounds and terminal continuations. Initialization precedes sidecar dispatch. Sidecars enforce this count only at their API-key rotation callback, retaining their own reset and same-target retry policies; the ordinary core physical-send budget is not newly applied to them, and OAuth/Anthropic account fallback remains available independently. Cooldown expiry and later pool growth do not refill it. This is a failover count, not a distinct-key guarantee or a new combo-wide budget; existing shared physical-send admission remains authoritative. Before cancelling a superseded response or choosing another key, the ordinary response path checks both limits. Generic reset-only initial fetches, key-rotation rebuilds and terminal continuations consume the same physical-send budget; opting out of transient-5xx retries does not bypass the child ceiling or enable 5xx retries. Initial and continuation recovery admission probes release their unused dispatch reservations immediately; the actual refetch owns the charged send permit. Generic OAuth hops replayed through the retry helpers mark their reservation as externally counted and pass it into the refetch, so its first physical send settles the booking once. An unused retry-helper hop is refunded on failure or abort. Terminal-continuation key recovery in `src/server/responses/adapter-continuation.ts` can draw the same single final reserve after the base allowance is spent. The quota-confirmed compact handoff in `src/server/responses/compact.ts` also transfers an exact prepaid reservation into its routed child; native three-send exhaustion still leaves the one allowed recovery send, without minting a second budget. An intermediate combo scope retains that exact booking and cannot enlarge its inherited total or recovery allowance; an already prepaid first send is preserved even when the remaining budget cannot fund later declared targets. Combo policy and child-scope construction live in `src/server/responses/combo-send-budget.ts`; their scopes derive through `src/lib/request-execution-budget.ts`, sharing both charged reservations and pending external-send settlement. Remaining-allowance checks and unused-permit refunds use that same ledger, while recovery and target-transition ledgers remain scope-local. Each child has a hard total ceiling that preserves one physical send for every later declared target, including when an earlier target uses recovery. An exact parent hop permit supplies the child's prepaid initial send; adapter-owned reservations and external send reports settle that booking once, rather than charging the hop and the physical send twice. Three plain-5xx targets use six physical sends, with the final combo hop eligible for recovery allowance. `tests/lib/execution-budget-permits.test.ts` checks interleaved scopes and refunds; `tests/responses/responses-send-budget-counts.test.ts` counts physical sends and verifies that every target is reached. On refusal, `src/providers/key-failover.ts` accepts `allowRotation: false` to record the proven failed key's cooldown through the existing selection-identity fence, without selecting, persisting or announcing an unattempted replacement. The original failure stays available to normal response handling, including an already-started streaming continuation. `tests/server/server-key-failover-e2e.test.ts` covers expiry, an initially unpooled key, actual web-search/image bridge dispatch, shared continuation limits and earlier send-budget exhaustion; `tests/adapters/key-failover.test.ts` verifies health-only recording, unchanged disk bytes and manual-selection ownership. + + ### Routed service-tier capability OpenAI-compatible service-tier support is resolved only after the final provider/model wire is @@ -743,13 +742,70 @@ budget before it knows whether a rotation is even possible, because the reservat `reserveDispatch` spends, `permit.use()` only confirms which leg sent, and `permit.release()` is idempotent and a no-op once used. Every ladder therefore owes the budget an answer on every exit. -Two shapes are correct and both are in the tree. Where the ladder dispatches inside its own `try` -— `adapter-dispatch.ts`, `run-turn-execution.ts` — it confirms with `use()` immediately before the -send and releases in its `catch`, so one catch covers a pre-dispatch throw and a throw from the -send alike. Where the replay happens after the loop continues — `adapter-continuation.ts` — it must -not confirm, because the send has not happened yet; it only releases. The passthrough ladder is a -third shape: it reserves with `countedExternally: true` and hands the permit to the rebuild through -`pendingHopPermit`, because there the retry helper reports the same physical send. +Combo bookings remain unconfirmed while constructing and dispatching a child. Preparation failure +or a failed child with no dispatch recorded in its fresh attempt log releases the unused booking; +an adapter-owned reservation or external send report settles it at the existing send boundary. +Successful responses (including deferred streams), positive child/nested dispatch records, and +unknown throws or cancellation after child entry retain the existing conservative one-send charge +when their transport does not settle the booking itself. Dispatch records can precede pacing: +this refunds proven local refusal, not every zero-wire failure. The iteration-wide cleanup covers +preparation, abort, failure and early return. `tests/responses/responses-send-budget-counts.test.ts` +checks local refusal followed by four plus two Vertex sends; the opaque runTurn and hosted search +cases in `tests/server/server-combo-failover-e2e.test.ts` preserve their observed dispatch charges. + +An initial terminal-guard continuation repairs a no-tool completion, even without a recovery +log label. It may spend the shared final reserve after base sends are exhausted. The retry helper +settles that permit once; a missing or spent reserve prevents another dispatch or credential hop. +`tests/server/terminal-guard-server.test.ts` verifies reset exhaustion on Anthropic and opted-in +OpenAI Chat, no-reserve refusal, and a final continuation 429 without an unfunded key change. + +Caller-counted adapter dispatch confirms its hop after request shaping and pacing. Retry-helper +dispatch reserves with `countedExternally: true` and passes the booking through `pendingHopPermit`. +An adapter declaring `fetchResponseUsesSendBudget` (Kiro, Command Code, MiMo and Google +Vertex/Antigravity) also receives an externally +counted booking, but adopts it through a derived budget at its own physical-send boundary. +An initial Kiro empty-completion repair reserves the same final send when no credential hop is +pending. Its derived scope shares the request ceiling and adds no recovery reserve. Native Responses +and translated retry-helper replays include the prepaid first send alongside remaining base retries, confirming that permit +only on their first dispatch. Other `fetchResponse` +implementations retain caller-owned confirmation; having that method alone does not establish +that the adapter consumes budget permits. A queued continuation carries an unused booking into +its next dispatch and releases it if shaping, pacing or admission fails before a send. +Caller-owned fallback continuation hops confirm on the first executor invocation. Budget-aware +Command Code, MiMo and Google use `src/adapters/physical-send.ts` to admit every inference +attempt, including effort removal, JWT retry and Google reset/5xx/400-repair paths. An unused +reservation refunds on helper pacing, abort, backoff or adapter setup failure. Consumption starts +at the underlying executor invocation: its own subsequent preflight can still reject, conservatively +consuming that attempt. Budget refusal preserves an available prior HTTP response and performs +no retry backoff. Model-catalog and credential discovery stay separate from inference accounting. +Ordinary Google AI Studio remains on the canonical server retry-helper path. The real Command Code OAuth +empty-completion regression in `tests/providers/command-code-provider.test.ts` bounds subsequent +credential rotations after a dispatched hop; MiMo executor and bootstrap separation is covered +in `tests/providers/mimo-free-provider.test.ts`. +`tests/adapters/physical-send.test.ts` verifies prepaid adoption, refusal and refund; +`tests/adapters/google/google-vertex-http.test.ts` bounds Google repair and retry sends. +Budget-aware adapter rebuilds can fund a key/OAuth recovery from the remaining final reserve; +an explicitly supplied prepaid scope is reused without another reservation. Key-429 admission +probes the same reserve and releases that probe; the actual rebuild owns the funded scope. +Initial, rebuilt and continuation `fetchResponse` calls pass their already-acquired pacing slot +to the executor. The physical-send helper consumes that slot once, then obtains one new slot +per internal retry. It reserves before pacing, so a refused internal retry takes no extra slot +or backoff; this does not remove the outer entry-point pacing wait. +`tests/server/server-key-failover-e2e.test.ts` checks Vertex key-401/key-429 and empty continuations, +unchanged selection without a reserve, and one pacing interval per actual inference. +Pre-stream 401 recovery reserves its inference send before token refresh, static-key selection +or response-body cancellation. This applies to translated OAuth, generic Responses passthrough, +and the native main/stored-pool one-shot replay. Refusal retains the original 401 and selection; +refresh/build/pacing failure or an absent alternate refunds the unused reservation. Native pool +replay retains its existing refresh, quarantine, lease and callback rules, confirms its already +charged permit at executor dispatch, and does not report that send a second time. Legacy numeric +holders instead report that direct send once because they did not receive a charged permit. +The native prepaid-child boundary is covered by `tests/responses/responses-native-main-refresh.test.ts`; +generic translated/passthrough parity is covered by `tests/server/server-google-antigravity-oauth-401-replay.test.ts`. + +`tests/server/server-kiro-oauth-401-replay.test.ts` counts actual Kiro requests across three stored +OAuth accounts, including an earlier connection reset and the third account's success or quota +response. Hop booking and adapter dispatch charge one physical send, not two. What must not happen is a ladder that charges and then returns through a path that neither confirms nor releases. That is not a lost send; it is a send the request never made, spending an allowance a diff --git a/structure/transports/streaming-health.md b/structure/transports/streaming-health.md index 682fbb2ca2..5ecd3bf102 100644 --- a/structure/transports/streaming-health.md +++ b/structure/transports/streaming-health.md @@ -1,10 +1,14 @@ # Streaming Health And WebSocket +401 recovery admits its inference before credential mutation or body disposal under the [recovery contract](responses.md#credential-hop-reservations), including native main/stored-account replay. Combo cleanup refunds proven local refusal while retaining successful, observed or uncertain child-dispatch charges under that contract. + +Budget-aware rebuilds reuse or fund a recovery reservation; the first helper send reuses the outer pacing slot under the [recovery and pacing contract](responses.md#credential-hop-reservations). + Native and translated delivery now have separate owners in the [core module ownership](responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays, including budget-aware adapters, settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Terminal and budget-aware Kiro empty-completion repairs share that final reserve; prepaid native Responses and translated helper retries retain their remaining base allowance, and caller-owned continuation hops refund if no dispatch occurs. Codex WebSocket quota-family normalization remains generic; retired-model evidence is filtered by the [OpenAI quota owner](../providers/openai-tiers.md#public-provider-contract), not by diff --git a/tests/adapters/adapter-inner-send-budget.test.ts b/tests/adapters/adapter-inner-send-budget.test.ts index 09406cb506..2ba454295d 100644 --- a/tests/adapters/adapter-inner-send-budget.test.ts +++ b/tests/adapters/adapter-inner-send-budget.test.ts @@ -6,6 +6,7 @@ import type { CursorRunRequest, CursorServerMessage } from "../../src/adapters/c import type { CursorTransport } from "../../src/adapters/cursor/transport"; import { createRequestExecutionBudget, type RequestExecutionBudgetPolicy } from "../../src/lib/request-execution-budget"; import { SendBudgetExhaustedError } from "../../src/lib/upstream-retry"; +import { comboExecutionBudgetPolicy, comboTargetSendBudget } from "../../src/server/responses/combo-send-budget"; /** * Adapters that retry INSIDE one adapter call are the layer a per-request cap cannot see from @@ -52,6 +53,23 @@ function alwaysResets(): { calls: number } { } describe("Kiro inner retries and the request send budget", () => { + test("combo hop bookings fund actual Kiro sends once and leave every target reachable", async () => { + const upstream = alwaysResets(); + const combo = createRequestExecutionBudget(comboExecutionBudgetPolicy(3)); + const counts: number[] = []; + for (let target = 0; target < 3; target++) { + const hop = combo.reserveDispatch({ sendClass: target === 0 ? "initial" : "combo-failover", targetKey: `target-${target}`, countedExternally: true }); + if (!hop.allowed) throw new Error(`target ${target} was starved`); + hop.permit.use(); + const child = comboTargetSendBudget(combo, 2 - target, hop.permit); + const before = upstream.calls; + await expect(fetchKiroWithRetry(kiroRequest, { timeoutMs: 5_000, sendBudget: child })).rejects.toBeDefined(); + counts.push(upstream.calls - before); + } + expect(counts).toEqual([3, 2, 1]); + expect(combo.used).toBe(upstream.calls); + expect(upstream.calls).toBe(6); + }); test("a context without a budget keeps the adapter's own reset ladder", async () => { const upstream = alwaysResets(); const observed: Array<{ ordinal: number; recovery?: string }> = []; diff --git a/tests/adapters/google/google-vertex-http.test.ts b/tests/adapters/google/google-vertex-http.test.ts index 7d91793226..d396348fbb 100644 --- a/tests/adapters/google/google-vertex-http.test.ts +++ b/tests/adapters/google/google-vertex-http.test.ts @@ -1,4 +1,6 @@ -import { afterEach, describe, expect, test } from "bun:test"; +import { afterEach, describe, expect, spyOn, test } from "bun:test"; +import * as retry from "../../../src/lib/upstream-retry"; +import { createRequestExecutionBudget } from "../../../src/lib/request-execution-budget"; import type { AdapterRequest } from "../../../src/adapters/base"; import { fetchAntigravityWithRetry, fetchDirectGeminiWithRetry, fetchVertexWithRetry } from "../../../src/adapters/google-http"; import { safeVertexHttpErrorMessage, retryableGoogleStatus } from "../../../src/adapters/google-errors"; @@ -30,6 +32,41 @@ function vertexError(code: number, status: string, message: string): string { } describe("vertex retry fetch", () => { + for (const [name, fetchResponse] of [["Vertex", fetchVertexWithRetry], ["Antigravity", fetchAntigravityWithRetry]] as const) { + test.each([400, 429, 503, "reset"] as const)(`${name} prepaid final send prevents another inference or backoff (%s)`, async status => { + const parent = createRequestExecutionBudget(); + parent.used = 3; + const hop = parent.reserveDispatch({ sendClass: "auth-recovery", targetKey: request.url, countedExternally: true }); + if (!hop.allowed) throw new Error("Expected final prepaid send"); + const scope = parent.deriveScope({ ...parent.policy, finalRecoveryAllowance: 0 }, hop.permit); + const raw = status === 400 ? vertexError(400, "INVALID_ARGUMENT", "tools.0.custom.input_schema: JSON schema is invalid") : `fixture ${status}`; + const first = status === "reset" ? Object.assign(new Error("fixture reset"), { code: "ECONNRESET" }) + : new Response(raw, { status, headers: { "Retry-After": "60" } }); + const fixture = mockFetch([first, new Response("unexpected replay")]); + const waits = spyOn(retry, "sleepWithAbort").mockImplementation(async () => {}); + const ordinals: number[] = []; + try { + const result = fetchResponse({ ...request, body: JSON.stringify({ request: { + contents: [{ role: "user", parts: [{ text: "hi" }] }], + tools: [{ functionDeclarations: [{ name: "replace_in_files", parameters: { + type: "object", properties: { occurrence_ids: { type: "array", items: { type: "string" } } }, + } }] }], + } }) }, { sendBudget: scope, returnRawErrors: true, onPhysicalSend: send => ordinals.push(send.ordinal) }); + if (status === "reset") await expect(result).rejects.toBeInstanceOf(retry.SendBudgetExhaustedError); + else { + const response = await result; + expect(response).toBe(first); + expect(await response.text()).toBe(raw); + } + hop.permit.release(); + expect(fixture.calls).toHaveLength(1); + expect(waits).not.toHaveBeenCalled(); + expect(parent.used).toBe(4); + expect(ordinals).toEqual([1]); + } finally { waits.mockRestore(); } + }); + } + test("successful response bodies survive beyond the response-header timeout", async () => { globalThis.fetch = (async () => new Response(new ReadableStream({ async start(controller) { @@ -331,10 +368,13 @@ describe("adapter fetchResponse wiring", () => { const aistudio = createGoogleAdapter({ adapter: "google", baseUrl: "https://generativelanguage.googleapis.com", apiKey: "k" } as never); const antigravity = createGoogleAdapter({ adapter: "google", baseUrl: "https://daily-cloudcode-pa.googleapis.com", googleMode: "cloud-code-assist" } as never); expect(typeof vertex.fetchResponse).toBe("function"); + expect(vertex.fetchResponseUsesSendBudget).toBe(true); expect(typeof vertex.formatErrorBody).toBe("function"); expect(typeof antigravity.fetchResponse).toBe("function"); + expect(antigravity.fetchResponseUsesSendBudget).toBe(true); expect(typeof antigravity.formatErrorBody).toBe("function"); expect(aistudio.fetchResponse).toBeUndefined(); + expect(aistudio.fetchResponseUsesSendBudget).toBeUndefined(); expect(aistudio.formatErrorBody).toBeUndefined(); }); diff --git a/tests/adapters/key-failover.test.ts b/tests/adapters/key-failover.test.ts index 561c1cefa9..757b1df82f 100644 --- a/tests/adapters/key-failover.test.ts +++ b/tests/adapters/key-failover.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { mkdtempSync, unlinkSync, writeFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, unlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { createOpenAIChatAdapter } from "../../src/adapters/openai-chat"; @@ -95,6 +95,29 @@ describe("hasKeyPoolFailover", () => { }); describe("rotateKeyOn429", () => { + test.each([false, true])("health-only 429 preserves selection ownership and disk bytes (superseded=%s)", superseded => { + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); + const routed = routedProviderConfig("p", config.providers.p); + if (superseded) { + expect(setActiveProviderApiKey(config, "p", "k2")).toBe(true); + expect(setActiveProviderApiKey(config, "p", "k1")).toBe(true); + } + const before = readFileSync(getConfigPath(), "utf8"); + const events: unknown[] = []; + const unsubscribe = subscribeAccountSelections(event => { events.push(event); }); + const now = 10_000; + try { + expect(rotateProviderTransportOn429(config, "p", routed, { + allowRotation: false, attemptedKey: routed.apiKey, retryAfter: "2", now, + })).toBeNull(); + expect(readFileSync(getConfigPath(), "utf8")).toBe(before); + expect(loadConfig().providers.p.apiKey).toBe("key-alpha-000111222333"); + expect(events).toEqual([]); + expect(getKeyCooldownUntil("p", "k1", now)).toBe(superseded ? null : now + 2_000); + expect(getKeyCooldownUntil("p", "k2", now)).toBeNull(); + } finally { unsubscribe(); } + }); + test("an old attempt cannot overwrite a newer manual key selection or its ABA revision", () => { const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); const routed = routedProviderConfig("p", config.providers.p); diff --git a/tests/adapters/physical-send.test.ts b/tests/adapters/physical-send.test.ts new file mode 100644 index 0000000000..c1ca9360c1 --- /dev/null +++ b/tests/adapters/physical-send.test.ts @@ -0,0 +1,70 @@ +import { describe, expect, test } from "bun:test"; +import { createAdapterPhysicalSend } from "../../src/adapters/physical-send"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { SendBudgetExhaustedError } from "../../src/lib/upstream-retry"; + +const url = "https://adapter-fixture.invalid/inference"; +function prepaid() { + const parent = createRequestExecutionBudget(); + parent.used = 3; + const hop = parent.reserveDispatch({ sendClass: "auth-recovery", targetKey: url, countedExternally: true }); + if (!hop.allowed) throw new Error("Expected prepaid final send"); + const scope = parent.deriveScope({ ...parent.policy, finalRecoveryAllowance: 0 }, hop.permit); + return { parent, scope, hop: hop.permit }; +} + +describe("adapter physical inference admission", () => { + test("a prepaid scope admits exactly one physical send and rejects replay before backoff", async () => { + const { parent, scope, hop } = prepaid(); + let sends = 0, waits = 0, pacingSlots = 0; + const ordinals: number[] = []; + const send = createAdapterPhysicalSend({ sendBudget: scope, onPhysicalSend: event => ordinals.push(event.ordinal) }, + Object.assign(async () => { sends += 1; return new Response("ok"); }, { + waitForPacing: async () => { pacingSlots += 1; }, + }) as typeof fetch); + await send({ url, dispatch: executor => executor(url) }); + await expect(send({ url, sendClass: "repair", beforeDispatch: () => { waits += 1; }, + dispatch: executor => executor(url) })).rejects.toBeInstanceOf(SendBudgetExhaustedError); + hop.release(); + expect(sends).toBe(1); + expect(waits).toBe(0); + expect(pacingSlots).toBe(1); + expect(ordinals).toEqual([1]); + expect(parent.used).toBe(4); + }); + + test.each(["pacing", "backoff", "abort", "adapter"] as const)("unused prepaid send refunds after %s refusal", async phase => { + const { parent, scope, hop } = prepaid(); + let sends = 0; + const controller = new AbortController(); + const failure = new Error(`fixture ${phase} refusal`); + const executor = Object.assign(async () => { sends += 1; return new Response("unexpected"); }, { + waitForPacing: async () => { if (phase === "pacing") throw failure; }, + }) as typeof fetch; + const send = createAdapterPhysicalSend({ sendBudget: scope, abortSignal: controller.signal }, executor); + await expect(send({ url, beforeDispatch: () => { + if (phase === "backoff") throw failure; + if (phase === "abort") controller.abort(failure); + }, dispatch: physical => { + if (phase === "adapter") throw failure; + return physical(url); + } })).rejects.toBe(failure); + hop.release(); + expect(parent.used).toBe(3); + expect(parent.reserveSpent).toBe(false); + expect(sends).toBe(0); + }); + + test("an exhausted initial send performs no inference or retry preparation", async () => { + const budget = createRequestExecutionBudget(); + budget.used = 4; + let prepared = false, sends = 0; + const send = createAdapterPhysicalSend({ sendBudget: budget }, + (async () => { sends += 1; return new Response("unexpected"); }) as typeof fetch); + await expect(send({ url, beforeDispatch: () => { prepared = true; }, + dispatch: physical => physical(url) })).rejects.toBeInstanceOf(SendBudgetExhaustedError); + expect(prepared).toBe(false); + expect(sends).toBe(0); + expect(budget.used).toBe(4); + }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index abd9538ec0..348167c7ae 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,5 @@ { + "physical-send.test.ts": "adapters", "responses-core-modules.test.ts": "responses", "chat-responses-control-integration.test.ts": "responses", "coding-agent-tool-result-images.test.ts": "adapters", diff --git a/tests/helpers/responses-core-source.ts b/tests/helpers/responses-core-source.ts index 35fea3be44..52e31367f0 100644 --- a/tests/helpers/responses-core-source.ts +++ b/tests/helpers/responses-core-source.ts @@ -17,6 +17,7 @@ export const RESPONSES_CORE_MODULES = [ "core-auth.ts", "core-normalize.ts", "core-combo.ts", + "combo-send-budget.ts", "request-prepare.ts", "request-transport.ts", "request-sidecar-auth.ts", diff --git a/tests/lib/execution-budget-permits.test.ts b/tests/lib/execution-budget-permits.test.ts index 6aff76df69..e3cc28f558 100644 --- a/tests/lib/execution-budget-permits.test.ts +++ b/tests/lib/execution-budget-permits.test.ts @@ -1,5 +1,6 @@ import { describe, expect, test } from "bun:test"; import { readFileSync } from "node:fs"; +import { comboExecutionBudgetPolicy, comboTargetSendBudget, deriveSendBudgetScope } from "../../src/server/responses/combo-send-budget"; import { CODEX_TEXT_GUARDED_BUDGET_POLICY, createRequestExecutionBudget, @@ -126,6 +127,136 @@ describe("atomic dispatch permits", () => { }); }); +describe("combo scopes share reservation accounting", () => { + test("a prepaid compact handoff survives an intermediate scope without a second charge", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const hop = parent.reserveDispatch({ sendClass: "initial", targetKey: "handoff", countedExternally: true }); + if (!hop.allowed) throw new Error("expected hop"); + const handoff = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + const combo = deriveSendBudgetScope(handoff, ONE_SEND_LEFT); + expect(combo.remainingBaseSends(3)).toBe(1); + const first = combo.reserveDispatch({ sendClass: "initial", targetKey: "combo", countedExternally: true }); + expect(first.allowed).toBe(true); + if (!first.allowed) throw new Error("prepaid send lost"); + first.permit.use(); parent.used += 1; + expect(parent.used).toBe(1); + expect(handoff.reserveDispatch({ sendClass: "initial", targetKey: "duplicate" }).allowed).toBe(false); + }); + + test("a combo handoff keeps its prepaid recovery and cannot enlarge the compact ceiling", () => { + const parent = createRequestExecutionBudget(); parent.used = 3; + const hop = parent.reserveDispatch({ sendClass: "account-failover", targetKey: "handoff", countedExternally: true }); + if (!hop.allowed) throw new Error("expected recovery"); + const handoff = deriveSendBudgetScope(parent, { ...parent.policy, baseSendAllowance: 4, finalRecoveryAllowance: 0 }, hop.permit); + const combo = deriveSendBudgetScope(handoff, comboExecutionBudgetPolicy(3)); + expect(combo.policy.maxTotalModelSends).toBe(4); + const booked = combo.reserveDispatch({ sendClass: "initial", targetKey: "first", countedExternally: true }); + if (!booked.allowed) throw new Error("lost prepaid recovery"); + const target = comboTargetSendBudget(combo, 2, booked.permit); + expect(target.remainingBaseSends(3)).toBe(1); + const physical = target.reserveDispatch({ sendClass: "transient", targetKey: "first" }); + expect(physical.allowed).toBe(true); + expect(parent.used).toBe(4); + expect(combo.reserveDispatch({ sendClass: "combo-failover", targetKey: "second" }).allowed).toBe(false); + }); + + test("a child sees the last send reserved by its parent before dispatch", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const child = deriveSendBudgetScope(parent, ONE_SEND_LEFT); + const reserved = parent.reserveDispatch({ sendClass: "initial", targetKey: "parent" }); + expect(reserved.allowed).toBe(true); + expect(child.remainingBaseSends(5)).toBe(0); + expect(child.reserveDispatch({ sendClass: "initial", targetKey: "child" }).allowed).toBe(false); + }); + + test("child release refunds the shared booking and external reports settle it once", () => { + const parent = createRequestExecutionBudget(); + const child = deriveSendBudgetScope(parent, parent.policy); + const first = child.reserveDispatch({ sendClass: "initial", targetKey: "child", countedExternally: true }); + if (!first.allowed) throw new Error("expected first permit"); + expect(parent.used).toBe(1); + first.permit.release(); + expect(parent.used).toBe(0); + const sent = child.reserveDispatch({ sendClass: "initial", targetKey: "child", countedExternally: true }); + if (!sent.allowed) throw new Error("expected second permit"); + parent.used += 1; + expect(child.used).toBe(1); + sent.permit.release(); + expect(parent.used).toBe(1); + child.used += 1; + expect(parent.used).toBe(2); + }); + + test("three failed targets cannot each refill the request-wide ladder", () => { + const parent = createRequestExecutionBudget(); + const combo = deriveSendBudgetScope(parent, comboExecutionBudgetPolicy(3)); + let sends = 0; + const byTarget: number[] = []; + for (let target = 0; target < 3; target++) { + const scope = comboTargetSendBudget(combo, 2 - target); + const before = sends; + for (let attempt = 0; attempt < 8; attempt++) { + const decision = scope.reserveDispatch({ sendClass: attempt === 0 ? "initial" : "auth-recovery", targetKey: `target-${target}` }); + if (!decision.allowed) break; + expect(decision.permit.use()).toBe(true); + sends++; + } + byTarget.push(sends - before); + } + expect(byTarget).toEqual([4, 1, 1]); + expect(sends).toBe(combo.policy.maxTotalModelSends); + expect(parent.used).toBe(sends); + expect(combo.remainingBaseSends(100)).toBe(0); + }); + + test("target transition ledgers remain local to each scope", () => { + const parent = createRequestExecutionBudget(comboExecutionBudgetPolicy(4)); + for (const name of ["a", "b"]) { + const child = deriveSendBudgetScope(parent, parent.policy); + for (const key of [name, `${name}-alternate`]) { + const decision = child.reserveDispatch({ sendClass: "auth-recovery", targetKey: key }); + expect(decision.allowed).toBe(true); + if (decision.allowed) decision.permit.use(); + } + expect(child.targetTransitions).toBe(1); + } + expect(parent.targetTransitions).toBe(0); + expect(parent.used).toBe(4); + }); + + test("two external reporters cannot adopt the same prepaid hop", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const hop = parent.reserveDispatch({ sendClass: "initial", targetKey: "combo", countedExternally: true }); + if (!hop.allowed) throw new Error("expected hop"); + hop.permit.use(); + const first = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + const second = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + expect(first.reserveDispatch({ sendClass: "transient", targetKey: "a", countedExternally: true }).allowed).toBe(true); + expect(second.reserveDispatch({ sendClass: "transient", targetKey: "b", countedExternally: true }).allowed).toBe(false); + parent.used += 1; + expect(parent.used).toBe(1); + }); + + test("an adapter adopts its exact prepaid hop once and can return it before dispatch", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const hop = parent.reserveDispatch({ sendClass: "initial", targetKey: "combo", countedExternally: true }); + if (!hop.allowed) throw new Error("expected hop"); + hop.permit.use(); + const child = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + expect(child.remainingBaseSends(3)).toBe(1); + const first = child.reserveDispatch({ sendClass: "initial", targetKey: "adapter" }); + if (!first.allowed) throw new Error("expected prepaid initial send"); + expect(parent.used).toBe(1); + first.permit.release(); + expect(child.remainingBaseSends(3)).toBe(1); + const retry = child.reserveDispatch({ sendClass: "initial", targetKey: "adapter" }); + if (!retry.allowed) throw new Error("expected returned booking"); + expect(retry.permit.use()).toBe(true); + expect(parent.used).toBe(1); + expect(child.reserveDispatch({ sendClass: "transient", targetKey: "adapter" }).allowed).toBe(false); + }); +}); + describe("layer caps intersect the shared budget", () => { test("a roster credential hop walks within the shared total; a cross-pool move does not", () => { // The two classes answer different questions and must not be conflated. A credential @@ -249,8 +380,18 @@ describe("generic-OAuth hop reservations are handed back when no send happens", "adapter-recovery-oauth-429", "attemptOpaqueBlobRecovery", ); - expect(block).toContain('rebuildAndRefetch("oauth-account-429", () => { hop.permit?.use(); })'); - expect(block).toMatch(/if \("failed" in result\) \{[^}]*hop\.permit\?\.release\(\)/); + // The confirm callback is adapter-owned: when the adapter carries its own fetchResponse the + // dispatch boundary lives inside it, so the callback is handed down; otherwise the reservation + // is parked in pendingHopPermit for the retry helper to settle. Match the shape rather than an + // exact source substring, because an exact multi-line substring breaks on any reformatting in + // this block without the behaviour having changed. + expect(block).toMatch(/rebuildAndRefetch\(\s*"oauth-account-429",[\s\S]{0,240}?hop\.permit\?\.use\(\)/); + expect(block).toMatch(/pendingHopPermit = [^;]*fetchResponse \? undefined : hop\.permit/); + // The refund moved out of the failed arm and into a finally, so one idempotent release now + // covers the failed return, the success path and a throw. Assert that shape rather than a + // per-branch release that no longer exists. + expect(block).toMatch(/if \("failed" in result\) return result\.failed;/); + expect(block).toMatch(/finally \{[^}]*hop\.permit\?\.release\(\)/); expect(block).toMatch(refundsOnThrow); }); diff --git a/tests/lib/transient-budget-scope-source.test.ts b/tests/lib/transient-budget-scope-source.test.ts index 8bc9c8d0ea..49055d72a6 100644 --- a/tests/lib/transient-budget-scope-source.test.ts +++ b/tests/lib/transient-budget-scope-source.test.ts @@ -19,10 +19,104 @@ import { readResponsesCoreSource } from "../helpers/responses-core-source"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import { repoPath } from "../helpers/repo-root"; +import { createScanner, SyntaxKind } from "typescript/unstable/ast"; const source = (relative: string): string => readFileSync(repoPath("src", ...relative.split("/")), "utf8"); +/** Check balanced option objects and their local allowance declarations, ignoring textual decoys. */ +function retryBudgetWiring(text: string): { reporters: number; invalid: string[] } { + // TypeScript 7's lexical scanner is in-process: no compiler server or fixture files. + const scanner = createScanner(true, undefined, text); + const tokens: string[] = []; + const skipTemplate = (): void => { + let depth = 0; + for (let kind = scanner.scan(); kind !== SyntaxKind.EndOfFile; kind = scanner.scan()) { + if (kind === SyntaxKind.TemplateHead) skipTemplate(); + else if (kind === SyntaxKind.OpenBraceToken) depth++; + else if (kind === SyntaxKind.CloseBraceToken) { + if (depth) depth--; + else if (scanner.reScanTemplateToken(false) === SyntaxKind.TemplateTail) return; + } + } + }; + for (let kind = scanner.scan(); kind !== SyntaxKind.EndOfFile; kind = scanner.scan()) { + if (kind === SyntaxKind.TemplateHead) { skipTemplate(); tokens.push("#literal"); } + else if (kind === SyntaxKind.StringLiteral) tokens.push(JSON.stringify(scanner.getTokenValue())); + else if (kind === SyntaxKind.NoSubstitutionTemplateLiteral) tokens.push("#literal"); + else tokens.push(scanner.getTokenText()); + } + const close = new Map(); + const parent = new Map(); + const scopeAt: number[] = []; + const stack: number[] = []; + let scope = -1; + tokens.forEach((token, index) => { + scopeAt[index] = scope; + if (["(", "[", "{"].includes(token)) { + stack.push(index); + if (token === "{") { parent.set(index, scope); scope = index; } + } else if ([")", "]", "}"].includes(token)) { + const start = stack.pop(); + if (start !== undefined) close.set(start, index); + if (token === "}") scope = parent.get(scope) ?? -1; + } + }); + const bindings = new Map>(); + tokens.forEach((token, index) => { + if ((token === "const" || token === "let") && tokens[index + 2] === "=") { + const scope = scopeAt[index]!; + if (!bindings.has(scope)) bindings.set(scope, new Map()); + bindings.get(scope)!.set(tokens[index + 1]!, index + 3); + } + }); + const unwrap = (start: number, end: number): [number, number] => { + while (tokens[start] === "(" && close.get(start) === end - 1) { start++; end--; } + return [start, end]; + }; + const shared = (start: number, end: number, scope: number): boolean => { + [start, end] = unwrap(start, end); + if (tokens[start] === "remainingTransientSendBudget" && tokens[start + 1] === "(" + && close.get(start + 1) === end - 1) return true; + if (end - start !== 3 || tokens[start + 1] !== "." || tokens[start + 2] !== "attempts") return false; + while (scope >= -1) { + const declaration = bindings.get(scope)?.get(tokens[start]!); + if (declaration !== undefined) { + const callEnd = close.get(declaration + 1); + return tokens[declaration] === "recoverySendAllowance" && tokens[declaration + 1] === "(" + && callEnd !== undefined && [";", ","].includes(tokens[callEnd + 1]!); + } + if (scope === -1) break; + scope = parent.get(scope) ?? -1; + } + return false; + }; + let reporters = 0; + const invalid: string[] = []; + for (const [object, end] of close) { + if (tokens[object] !== "{") continue; + const properties = new Map(); + for (let index = object + 1; index < end; index++) { + if (tokens[index + 1] === ":") { + const name = tokens[index]!.replace(/^"|"$/g, ""); + const start = index + 2; + index = start; + while (index < end && tokens[index] !== ",") index = (close.get(index) ?? index) + 1; + if (!properties.has(name)) properties.set(name, []); + properties.get(name)!.push(unwrap(start, index)); + } else if (close.has(index)) index = close.get(index)!; + } + if (properties.get("onSendsConsumed")?.some(([start, end]) => end - start === 1 && tokens[start] === "noteTransientSends")) { + reporters++; + const attempts = properties.get("attempts") ?? []; + if (attempts.length !== 1 || !shared(...attempts[0]!, object)) { + invalid.push(`reporter ${reporters}: unchecked attempts`); + } + } + } + return { reporters, invalid }; +} + /** * `transientRetryOn5xx.attempts` is ONE request-wide total-send budget, not a per-leg * allowance. A Responses request can reach upstream on several legs — the initial send, a @@ -61,24 +155,15 @@ describe("transient send budget stays request-scoped", () => { // rebuild refetch, OAuth 401 replay, rate-limit 429 replay). The passthrough four were added // for #4546: the owner used to be declared BELOW that branch, which put it in the temporal // dead zone there, so each of those legs silently took the helper's fresh default of 3. - expect(core.match(/onSendsConsumed: noteTransientSends/g)).toHaveLength(7); - - // EVERY leg asks for the remainder now, including the adapter initial send. That one used - // to pass the raw policy on the argument that nothing had been spent yet -- true for a first - // turn, false for a combo child, which inherits the parent's holder and then took a fresh - // full allowance on its own first send. Five sites spell it directly; the two rebuild legs - // go through recoverySendAllowance, which spends the base allowance first and only then - // draws the single shared final-recovery reserve. - expect(core.match(/attempts: remainingTransientSendBudget\(/g)).toHaveLength(5); - expect(core).toContain("attempts: remainingTransientSendBudget(transientPolicy.attempts)"); - expect(core).toContain("attempts: remainingTransientSendBudget(continuationTransientPolicy.attempts)"); - // The reserve path: an account move and a validated rebuild share ONE final send, so a - // request cannot take both and reach five. - expect(core.match(/recoverySendAllowance\(/g)).toHaveLength(2); + // Every reporter must draw either directly from the remainder or from the recovery + // allowance declared in its own lexical scope. The split between these forms may change: + // the first terminal repair now draws the final reserve without an explicit recovery label. + // A whole-tree substring count could pass while one site took a fresh policy allowance. + const wiring = retryBudgetWiring(core); + // Generic OAuth401 now shares rebuildAndRefetch instead of duplicating a retry reporter. + expect(wiring.reporters).toBe(6); + expect(wiring.invalid).toEqual([]); expect(core).toContain("countedExternally: true"); - // The passthrough legs have no adapter policy to draw from, so they name the helper's own - // ceiling rather than re-spelling the number. - expect(core).toContain("attempts: remainingTransientSendBudget(TRANSIENT_RETRY_MAX_ATTEMPTS)"); // The trap that would make the passthrough wiring a silent no-op: transientRetryPolicyFor // returns null for Codex forward auth, so gating these sites on it would restore a fresh 3. expect(core).not.toContain("transientPolicy ? { attempts: remainingTransientSendBudget(TRANSIENT_RETRY_MAX_ATTEMPTS)"); @@ -89,6 +174,34 @@ describe("transient send budget stays request-scoped", () => { expect(core).not.toContain("attempts: transientPolicy.attempts,"); }); + test("the wiring oracle ignores formatting and decoy text but rejects fresh allowances", () => { + const options = (attempts: string): string => `retry(fetch, { + "onSendsConsumed": (noteTransientSends), + /* a comment between the property and its value is not a different budget */ + attempts: (${attempts}), + });`; + const decoy = `// attempts: remainingTransientSendBudget(3), onSendsConsumed: noteTransientSends + const text = "attempts: remainingTransientSendBudget(3), onSendsConsumed: noteTransientSends";`; + const valid = `${decoy} + ${options("remainingTransientSendBudget /* format */ (policy.attempts)")} + const allowance = recoverySendAllowance(cap, "repair", target); + ${options("allowance.attempts")}`; + expect(retryBudgetWiring(valid)).toEqual({ reporters: 2, invalid: [] }); + expect(retryBudgetWiring(valid.replaceAll("\n", "\r\n\t").replaceAll(": (", ":\n ("))) + .toEqual({ reporters: 2, invalid: [] }); + for (const fresh of ["3", "policy.attempts"]) { + expect(retryBudgetWiring(`${decoy}\n${options(fresh)}`).invalid).toHaveLength(1); + // An unrelated valid allowance, even with the same name, cannot hide a fresh local one. + const shadowed = `const allowance = recoverySendAllowance(cap, "repair", target); + function continuation() { const allowance = { attempts: ${fresh} }; ${options("allowance.attempts")} }`; + expect(retryBudgetWiring(shadowed).invalid).toHaveLength(1); + } + expect(retryBudgetWiring(`const allowance = recoverySendAllowance(cap) || policy; + ${options("allowance.attempts")}`).invalid).toHaveLength(1); + expect(retryBudgetWiring("retry(fetch, { onSendsConsumed: noteTransientSends });").invalid) + .toHaveLength(1); + }); + test("the helper still exposes the seam those call sites depend on", () => { const retry = source("lib/upstream-retry.ts"); expect(retry).toContain("onSendsConsumed?: (sends: number) => void;"); @@ -141,16 +254,16 @@ describe("every dispatch path reports into the shared budget", () => { // fresh four. expect(compact).toContain("turnAdmissionLease, sendBudget,"); // The handoff child already inherited; both paths must keep doing so. - expect(compact).toContain("{ ...options, sendBudget }"); + expect(compact).toContain("{ ...options, sendBudget: handoffBudget }"); + expect(compact).toContain("sendBudget.deriveScope({"); + expect(compact).toContain("}, hop.permit)"); }); test("credential hops keep their roster cap AND reserve from the shared budget", () => { const core = readResponsesCoreSource(); - // Six hop sites: the native passthrough 429, the shared sidecar hook's generic and - // Anthropic arms, the runTurn preflight 429, the adapter recovery loop, and the - // continuation loop. The last two were the arms that actually iterate the roster, so - // leaving them out meant the claim held everywhere except where it mattered most. - expect(core.match(/reserveCredentialHop\(/g)).toHaveLength(6); + // Existing six roster-hop sites plus native main/pool401, generic passthrough OAuth401, + // translated OAuth401 and static-key401 all admit before credential mutation/body disposal. + expect(core.match(/reserveCredentialHop\(/g)).toHaveLength(10); // The per-roster caps are NOT replaced. The effective allowance is the intersection, so // removing either half is a behaviour change that has to be argued for. expect(core).toContain("genericFailovers < GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST"); diff --git a/tests/providers/command-code-provider.test.ts b/tests/providers/command-code-provider.test.ts index d588c550ee..b41f75b54b 100644 --- a/tests/providers/command-code-provider.test.ts +++ b/tests/providers/command-code-provider.test.ts @@ -1,4 +1,13 @@ import { afterEach, describe, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveCredential, getAccountSet, setActiveAccount } from "../../src/oauth/store"; +import { clearGenericFailoverHealth } from "../../src/oauth/generic-account-failover"; +import { createRequestExecutionBudget, CODEX_TEXT_GUARDED_BUDGET_POLICY } from "../../src/lib/request-execution-budget"; +import { handleResponses } from "../../src/server/responses"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import type { OcxConfig } from "../../src/types"; import { commandCodeSessionId, createCommandCodeAdapter } from "../../src/adapters/command-code"; import { loginCommandCode, parseCommandCodeCallback, shouldImportLocalCommandCodeAuth } from "../../src/oauth/command-code"; import { buildModelsRequest, OAUTH_PROVIDERS } from "../../src/oauth"; @@ -39,6 +48,50 @@ async function builtRequest(...args: Parameters resetCommandCodeReasoningEffortsForTest()); describe("Command Code provider", () => { + test("empty-completion OAuth continuation counts initial sends and keeps its prepaid hop charged", async () => { + const previousHome = process.env.OPENCODEX_HOME; + const fixtureHome = mkdtempSync(join(tmpdir(), "ocx-command-hop-")); + process.env.OPENCODEX_HOME = fixtureHome; + const originalFetch = globalThis.fetch; + clearGenericFailoverHealth(); + try { + for (let index = 0; index < 4; index++) await saveCredential("command-code", { + access: `synthetic-command-${index}`, refresh: `synthetic-refresh-${index}`, + expires: Date.now() + 3_600_000, accountId: `fixture-${index}`, source: "oauth", + }, { addAccount: true }); + await setActiveAccount("command-code", getAccountSet("command-code")!.accounts[0]!.id); + // Every physical inference send, including the initial and continuation, shares this cap. + const budget = createRequestExecutionBudget({ ...CODEX_TEXT_GUARDED_BUDGET_POLICY, + maxTotalModelSends: 3, baseSendAllowance: 3, finalRecoveryAllowance: 0 }); + const authorizations: string[] = []; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url !== "https://api.commandcode.ai/alpha/generate") throw new Error(`Unexpected fixture request: ${url}`); + authorizations.push(new Headers(init?.headers).get("authorization") ?? ""); + if (authorizations.length === 1) return new Response('{"type":"finish","finishReason":"stop"}\n'); + return Response.json({ error: { message: "rate limited" } }, { status: 429 }); + }) as typeof fetch; + const cfg = { defaultProvider: "command-code", emptyCompletionRetry: true, providers: { + "command-code": { adapter: "command-code", baseUrl: "https://api.commandcode.ai", authMode: "oauth", + models: ["deepseek/deepseek-v4-flash"] }, + } } as OcxConfig; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "command-code/deepseek/deepseek-v4-flash", input: "hello", stream: false }), + }), cfg, { model: "", provider: "" }, { sendBudget: budget }); + await response.text(); + expect(authorizations).toEqual(["Bearer synthetic-command-0", "Bearer synthetic-command-0", "Bearer synthetic-command-1"]); + expect(budget.used).toBe(3); + expect(getAccountSet("command-code")!.activeAccountId).toBe(getAccountSet("command-code")!.accounts[1]!.id); + } finally { + globalThis.fetch = originalFetch; + clearGenericFailoverHealth(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(fixtureHome); + } + }, 20_000); + test("registry and OAuth surfaces stay in parity", () => { const registry = PROVIDER_REGISTRY.find(row => row.id === "command-code"); expect(registry).toMatchObject({ @@ -650,7 +703,8 @@ describe("Command Code provider", () => { expect(JSON.parse(bareBuilt.body).params.tools).toEqual(tools); }); - test("refreshes a stale official effort record only after a reasoning rejection and retries without it", async () => { + test.each(["fallback", "supplied", "prepaid"] as const)("refreshes stale effort metadata separately from inference executor (%s)", async mode => { + const supplied = mode !== "fallback"; const requests: Array<{ url: string; body?: string }> = []; const fetch = (async (url: string | URL | Request, init?: RequestInit) => { const href = String(url); @@ -664,11 +718,30 @@ describe("Command Code provider", () => { }) as typeof globalThis.fetch; const adapter = createCommandCodeAdapter({ ...provider, fetch } as OcxProviderConfig); const request = await adapter.buildRequest({ ...parsed(), options: { reasoning: "max" } }); - const response = await adapter.fetchResponse!(request); - expect(response.ok).toBe(true); + let suppliedCalls = 0; + const executor = (async (input, init) => { + expect(String(input).endsWith("/alpha/generate")).toBe(true); + suppliedCalls += 1; + return fetch(input, init); + }) as typeof globalThis.fetch; + const budget = createRequestExecutionBudget(); + if (mode === "prepaid") budget.used = 3; + const hop = mode === "prepaid" ? budget.reserveDispatch({ sendClass: "auth-recovery", targetKey: request.url, countedExternally: true }) : undefined; + if (hop && !hop.allowed) throw new Error("Expected final prepaid send"); + const scope = hop?.allowed ? budget.deriveScope({ ...budget.policy, finalRecoveryAllowance: 0 }, hop.permit) : budget; + const observed: number[] = []; + const response = await adapter.fetchResponse!(request, { ...(supplied ? { executor } : {}), sendBudget: scope, + onPhysicalSend: send => observed.push(send.ordinal) }); + if (hop?.allowed) hop.permit.release(); + expect(suppliedCalls).toBe(supplied ? mode === "prepaid" ? 1 : 2 : 0); + expect(response.ok).toBe(mode !== "prepaid"); + expect(budget.used).toBe(mode === "prepaid" ? 4 : 2); + expect(observed).toEqual(mode === "prepaid" ? [1] : [1, 2]); expect(commandCodeReasoningEfforts("deepseek/deepseek-v4-flash")).toEqual(["high"]); const generated = requests.filter(request => request.url.endsWith("/alpha/generate")); - expect(JSON.parse(generated[1]!.body!).params).not.toHaveProperty("reasoning_effort"); + expect(generated).toHaveLength(mode === "prepaid" ? 1 : 2); + if (mode === "prepaid") expect(await response.text()).toContain("unsupported reasoning_effort"); + else expect(JSON.parse(generated[1]!.body!).params).not.toHaveProperty("reasoning_effort"); }); // Pins the profileUrl of each id added for #2647 — nothing more. diff --git a/tests/providers/mimo-free-provider.test.ts b/tests/providers/mimo-free-provider.test.ts index 00176d2589..a6e63274a0 100644 --- a/tests/providers/mimo-free-provider.test.ts +++ b/tests/providers/mimo-free-provider.test.ts @@ -16,6 +16,7 @@ import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; for (const phase of ["bootstrap", "chat", "401-replay"] as const) test.each([307, 308])(`MiMo ${phase} never follows %i`, async status => { const nativeFetch = globalThis.fetch; @@ -319,7 +320,8 @@ describe("mimo-free auth retry predicate", () => { return createMimoFreeAdapter(provider); } - test("401 retries exactly once with a fresh JWT after draining the first body", async () => { + test.each(["fallback", "supplied", "prepaid"] as const)("401 retry preserves inference admission and separate bootstrap (%s)", async mode => { + const supplied = mode !== "fallback"; const fakeJwt = "h." + Buffer.from(JSON.stringify({ exp: Math.floor(Date.now() / 1000) + 3600 })).toString("base64") + ".s"; const calls: string[] = []; const originalFetch = globalThis.fetch; @@ -338,16 +340,33 @@ describe("mimo-free auth retry predicate", () => { }) as unknown as typeof fetch; try { const adapter = adapterForRetry(); + let suppliedCalls = 0; + const executor = (async (input, init) => { + expect(String(input)).toBe(MIMO_CHAT_URL); + suppliedCalls += 1; + return globalThis.fetch(input, init); + }) as typeof fetch; + const budget = createRequestExecutionBudget(); + if (mode === "prepaid") budget.used = 3; + const hop = mode === "prepaid" ? budget.reserveDispatch({ sendClass: "auth-recovery", targetKey: MIMO_CHAT_URL, countedExternally: true }) : undefined; + if (hop && !hop.allowed) throw new Error("Expected final prepaid send"); + const scope = hop?.allowed ? budget.deriveScope({ ...budget.policy, finalRecoveryAllowance: 0 }, hop.permit) : budget; + const observed: number[] = []; const res = await adapter.fetchResponse!( { url: MIMO_CHAT_URL, method: "POST", headers: { "Authorization": "Bearer stale" }, body: "{}" }, - {} as never, + { ...(supplied ? { executor } : {}), sendBudget: scope, onPhysicalSend: send => observed.push(send.ordinal) }, ); - expect(res.status).toBe(200); + if (hop?.allowed) hop.permit.release(); + expect(suppliedCalls).toBe(supplied ? mode === "prepaid" ? 1 : 2 : 0); + expect(res.status).toBe(mode === "prepaid" ? 401 : 200); + expect(budget.used).toBe(mode === "prepaid" ? 4 : 2); + expect(observed).toEqual(mode === "prepaid" ? [1] : [1, 2]); // Sequence: first chat with stale token -> 401 -> bootstrap -> retry with fresh JWT. expect(calls[0]).toBe("chat:Bearer stale"); - expect(calls[1]).toBe("bootstrap"); - expect(calls[2]).toBe(`chat:Bearer ${fakeJwt}`); - expect(calls.length).toBe(3); + if (mode !== "prepaid") expect(calls[1]).toBe("bootstrap"); + if (mode === "prepaid") expect(await res.text()).toBe("expired"); + else expect(calls[2]).toBe(`chat:Bearer ${fakeJwt}`); + expect(calls.length).toBe(mode === "prepaid" ? 1 : 3); } finally { globalThis.fetch = originalFetch; resetMimoJwtCache(); diff --git a/tests/responses/responses-compaction-routing.test.ts b/tests/responses/responses-compaction-routing.test.ts index c612aed502..78852e0665 100644 --- a/tests/responses/responses-compaction-routing.test.ts +++ b/tests/responses/responses-compaction-routing.test.ts @@ -1464,10 +1464,10 @@ describe("compact alternate-account attempt (#913)", () => { }); }); - test("a quota-blocked previous-model compact retries the same thread's successful routed handoff target (#2723)", async () => { + test.each([false, true])("a quota-blocked previous-model compact retries its routed handoff (combo: %s)", async combo => { await withPoolEnv("ocx-compact-routed-handoff-", async config => { config.providers.deepseek = { - adapter: "openai-chat", + adapter: combo ? "openai-responses" : "openai-chat", baseUrl: "https://api.deepseek.com", authMode: "key", apiKey: "deepseek-test-key", @@ -1481,6 +1481,7 @@ describe("compact alternate-account attempt (#913)", () => { models: ["gpt-5.6-sol"], }; const headers = { "x-codex-parent-thread-id": "compact-routed-handoff-thread" }; + config.combos = { summarizer: { strategy: "failover", targets: [{ provider: "deepseek", model: "deepseek-v4-flash" }] } }; const calls: Array<{ model: string; nativeCompact: boolean }> = []; globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => { const url = typeof input === "string" @@ -1496,12 +1497,12 @@ describe("compact alternate-account attempt (#913)", () => { status: 502, }); } - return jsonResponse(completedPayload("DeepSeek handoff summary")); + return combo ? sseResponse([{ type: "response.completed", response: completedPayload("DeepSeek handoff summary") }]) + : jsonResponse(completedPayload("DeepSeek handoff summary")); }) as typeof fetch; - const manual = await handleResponsesCompact( compactionRequest( - baseCompactionBody({ model: "deepseek/deepseek-v4-flash" }), + baseCompactionBody({ model: combo ? "combo/summarizer" : "deepseek/deepseek-v4-flash" }), undefined, headers, ), @@ -1525,7 +1526,6 @@ describe("compact alternate-account attempt (#913)", () => { expect(calls.length).toBeGreaterThan(0); expect(calls.every(call => call.model === "gpt-5.6-sol" && call.nativeCompact)).toBe(true); calls.length = 0; - const logCtx: RequestLogContext = { model: "", provider: "" }; const automatic = await handleResponsesCompact( compactionRequest( @@ -1540,9 +1540,9 @@ describe("compact alternate-account attempt (#913)", () => { expect(automatic.status).toBe(200); const output = await automatic.json() as { output?: unknown[] }; expect(output.output?.length).toBeGreaterThan(0); - expect(logCtx.provider).toBe("deepseek"); + expect(logCtx.provider).toBe(combo ? "combo" : "deepseek"); expect(calls.at(-1)).toEqual({ model: "deepseek-v4-flash", nativeCompact: false }); - expect(calls.slice(0, -1).length).toBeGreaterThan(0); + expect(calls.slice(0, -1)).toHaveLength(3); // Native ladder + one handoff = four sends. expect(calls.slice(0, -1).every(call => ( call.model === "gpt-5.6-sol" && call.nativeCompact ))).toBe(true); diff --git a/tests/responses/responses-core-modules.test.ts b/tests/responses/responses-core-modules.test.ts index 2d94f02f98..7a13cfb07b 100644 --- a/tests/responses/responses-core-modules.test.ts +++ b/tests/responses/responses-core-modules.test.ts @@ -6,6 +6,8 @@ import { readResponsesCoreModule, } from "../helpers/responses-core-source"; import { createResponsesSendBudget } from "../../src/server/responses/request-send-budget"; +import { createAdapterContinuations } from "../../src/server/responses/adapter-continuation"; +import type { AdapterFetchContext, AdapterRequest } from "../../src/adapters/base"; import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; import { createTranslatorBudget } from "../../src/lib/translator-budget"; import type { TransientSendBudget } from "../../src/lib/upstream-retry"; @@ -118,7 +120,9 @@ function budgetOwner(sendBudget: TransientSendBudget) { req: new Request("http://localhost/v1/responses"), logCtx: { model: "test", provider: "test" }, options: { translatorBudget, sendBudget }, - }); + }, { route: { providerName: "test", modelId: "test", provider: { + adapter: "openai-chat", authMode: "key", baseUrl: "https://budget-fixture.invalid/v1", + } } } as Parameters[1]); if (result instanceof Response) { translatorBudget.dispose(); throw new Error("Unexpected workflow refusal without a workflow root"); @@ -127,6 +131,109 @@ function budgetOwner(sendBudget: TransientSendBudget) { } describe("Responses request-owned send budget after extraction", () => { + test("a prepaid retry-helper continuation retains the remaining base retry after a reset", async () => { + const holder = createRequestExecutionBudget(); + holder.used = 1; + const { owner, dispose } = budgetOwner(holder); + const hop = owner.reserveCredentialHop("auth-recovery", "fixture", true); + expect(hop.allowed).toBe(true); + owner.pendingHopPermit = hop.permit; + const translatorBudget = createTranslatorBudget(); + const parsed = { modelId: "fixture", stream: false }; + let sends = 0; + const adapter = { + name: "openai-chat", + async buildRequest() { return { url: "https://fixture.invalid/", method: "POST", headers: {}, body: "{}" }; }, + async parseResponse() { return []; }, + }; + type Args = Parameters; + const continuation = createAdapterContinuations( + { options: {}, config: {}, logCtx: { model: "fixture", provider: "fixture" } } as Args[0], + { route: { providerName: "fixture", modelId: "fixture", provider: { + adapter: "openai-chat", baseUrl: "https://fixture.invalid/", + } }, parsed, translatorBudget } as Args[1], + { activeAdapter: adapter, oauthDispatch: () => async () => { + sends += 1; + if (sends === 1) throw Object.assign(new Error("fixture ECONNRESET"), { code: "ECONNRESET" }); + return new Response("{}"); + } } as unknown as Args[2], + { routedCompaction: false }, owner, + { upstream: new AbortController(), connectMs: 100, rateLimitPolicy: null, rateLimitRetries: 0, + stallTimeoutMs: 100, keyPool429RetryAllowed: () => false }, + ); + try { + const events = []; + for await (const event of continuation.fetchTerminalGuardContinuation(parsed as Args[1]["parsed"], "oauth-account-429")) events.push(event); + expect(sends).toBe(2); + expect(holder.used).toBe(3); + expect(holder.reserveSpent).toBe(false); + expect(owner.pendingHopPermit).toBeUndefined(); + expect(events).toEqual([]); + hop.permit?.release(); + expect(holder.used).toBe(3); + } finally { translatorBudget.dispose(); dispose(); } + }); + + for (const [adapterOwned, prepaid] of [[true, false], [true, true], [false, true]]) { + test.each(["build", "pacing", "adapter", "sent"] as const)(`continuation refunds only unused permits (adapterOwned=${adapterOwned}, prepaid=${prepaid}, %s)`, async failure => { + const holder = createRequestExecutionBudget(); + holder.used = 3; + const { owner, dispose } = budgetOwner(holder); + if (prepaid) { + const hop = owner.reserveCredentialHop("auth-recovery", "fixture", adapterOwned); + expect(hop.allowed).toBe(true); + owner.pendingHopPermit = hop.permit; + } + const translatorBudget = createTranslatorBudget(); + const upstream = new AbortController(); + if (failure === "pacing") upstream.abort(); + let sends = 0; + const parsed = { modelId: "fixture", stream: false }; + const adapter = { + name: adapterOwned ? "kiro" : "google", fetchResponseUsesSendBudget: adapterOwned, + async buildRequest() { + if (failure === "build") throw new Error("fixture build failure"); + return { url: "https://fixture.invalid/", method: "POST", headers: {}, body: "{}" }; + }, + async fetchResponse(request: AdapterRequest, ctx: AdapterFetchContext) { + if (failure === "adapter") throw new Error("fixture before physical send"); + if (adapterOwned) { + const send = ctx.sendBudget!.reserveDispatch({ sendClass: "transient", targetKey: request.url }); + expect(send.allowed).toBe(true); + if (!send.allowed) throw new Error("Expected funded physical send"); + expect(send.permit.use()).toBe(true); + } else { + await ctx.executor!(request.url, { method: request.method, body: request.body }); + } + sends += 1; + return new Response("{}"); + }, + async parseResponse() { return []; }, + }; + type Args = Parameters; + const continuation = createAdapterContinuations( + { options: {}, config: {}, logCtx: { model: "fixture", provider: "fixture" } } as Args[0], + { route: { providerName: "fixture", modelId: "fixture", provider: { + adapter: "kiro", baseUrl: "https://fixture.invalid/", + requestPacing: { enabled: true, minIntervalMs: failure === "pacing" ? 1 : 0 }, + } }, parsed, translatorBudget } as Args[1], + { activeAdapter: adapter, oauthDispatch: () => async () => new Response("{}") } as unknown as Args[2], + { routedCompaction: false }, owner, + { upstream, connectMs: 100, rateLimitPolicy: null, rateLimitRetries: 0, stallTimeoutMs: 100, + keyPool429RetryAllowed: () => false }, + ); + try { + const events = []; + for await (const event of continuation.fetchTerminalGuardContinuation(parsed as Args[1]["parsed"], "empty-completion")) events.push(event); + expect(sends).toBe(failure === "sent" ? 1 : 0); + expect(holder.used).toBe(failure === "sent" ? 4 : 3); + expect(holder.reserveSpent).toBe(failure === "sent"); + expect(owner.pendingHopPermit).toBeUndefined(); + expect(events.some(event => event.type === "error")).toBe(failure !== "sent"); + } finally { translatorBudget.dispose(); dispose(); } + }); + } + test("legacy holders retain identity and an exhausted remainder stays zero", () => { const holder = { used: 2 }; const { owner, dispose } = budgetOwner(holder); diff --git a/tests/responses/responses-native-main-refresh.test.ts b/tests/responses/responses-native-main-refresh.test.ts index 3b78ff6402..3f06d0feff 100644 --- a/tests/responses/responses-native-main-refresh.test.ts +++ b/tests/responses/responses-native-main-refresh.test.ts @@ -13,6 +13,7 @@ import { handleResponses, handleResponsesCompact } from "../../src/server/respon import type { RequestLogContext } from "../../src/server/request-log"; import type { OcxConfig } from "../../src/types"; import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; const originalFetch = globalThis.fetch; let home = ""; @@ -109,6 +110,35 @@ function install401ThenRefreshHarness(): { sends: string[]; refreshes: string[] } describe("native main 401 refresh and replay", () => { + for (const account of ["main", "stored"] as const) test.each(["fresh", "prepaid"] as const)( + `native ${account} 401 replay requires a separately admitted send (%s)`, async mode => { + const cfg = config({ secondAccount: account === "stored" }); + if (account === "stored") { + cfg.activeCodexAccountId = OTHER_ACCOUNT_ID; + saveCodexAccountCredential(OTHER_ACCOUNT_ID, { accessToken: "rejected-access", refreshToken: "other-refresh", + expiresAt: Date.now() + 3_600_000, chatgptAccountId: "account-other" }); + } + const harness = install401ThenRefreshHarness(); + const budget = createRequestExecutionBudget(); + if (mode === "prepaid") budget.used = 3; + const hop = mode === "prepaid" ? budget.reserveDispatch({ sendClass: "combo-failover", targetKey: "native", countedExternally: true }) : undefined; + if (hop && !hop.allowed) throw new Error("Expected prepaid initial send"); + const scope = hop?.allowed ? budget.deriveScope({ ...budget.policy, finalRecoveryAllowance: 0 }, hop.permit) : budget; + let storedReplays = 0; + const response = await handleResponses(request("/v1/responses"), cfg, { model: "", provider: "" }, { + sendBudget: scope, onStoredPool401ReplayDispatched: () => { storedReplays += 1; }, + }); + const body = await response.text(); + if (hop?.allowed) hop.permit.release(); + expect(response.status).toBe(mode === "prepaid" ? 401 : 200); + expect(harness.sends).toEqual(mode === "prepaid" ? ["Bearer rejected-access"] : ["Bearer rejected-access", "Bearer refreshed-access"]); + expect(harness.refreshes).toEqual(mode === "prepaid" ? [] : [account === "stored" ? "other-refresh" : "refresh-grant"]); + expect(budget.used).toBe(mode === "prepaid" ? 4 : 2); + expect(storedReplays).toBe(account === "stored" && mode === "fresh" ? 1 : 0); + if (mode === "prepaid") expect(body).toContain("expired bearer"); + }, 20_000, + ); + test("refreshes a refresh-only native main credential before upstream I/O", async () => { writeFileSync(join(home, "auth.json"), JSON.stringify({ tokens: { refresh_token: "refresh-grant", account_id: "account-main" }, diff --git a/tests/responses/responses-send-budget-counts.test.ts b/tests/responses/responses-send-budget-counts.test.ts index 78f5a42856..4cfa380360 100644 --- a/tests/responses/responses-send-budget-counts.test.ts +++ b/tests/responses/responses-send-budget-counts.test.ts @@ -1,9 +1,15 @@ -import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import * as comboRequests from "../../src/combos"; +import { executeComboResponses } from "../../src/server/responses/core-combo"; +import { createTranslatorBudget } from "../../src/lib/translator-budget"; +import type { RequestExecutionBudget } from "../../src/lib/request-execution-budget"; import { clearComboSelectionState, clearComboTargetCooldowns } from "../../src/combos"; import { clearKeyCooldowns } from "../../src/providers/key-failover"; import { handleResponses } from "../../src/server/responses/core"; import type { RequestLogContext } from "../../src/server/request-log"; import type { OcxConfig } from "../../src/types"; +import { saveConfig } from "../../src/config"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; /** * One logical request, one send budget -- asserted as a COUNT, because the defect in #4546 is a @@ -11,11 +17,10 @@ import type { OcxConfig } from "../../src/types"; * only assertion that catches a regression here is the exact number of times the proxy reached * upstream for one client turn. * - * These rows use a key-auth `openai-chat` provider with `transientRetryOn5xx` because that is the - * counted path: the generic adapter branch draws `attempts` from the request budget and reports - * every physical send back through `onSendsConsumed`, and `noteAttemptSend` records the same send - * on the attempt. An adapter without an opted-in transient policy keeps reset-only semantics and - * hops on the first 5xx, so it would pin a 1 for every shape and prove nothing. + * These 5xx rows opt into `transientRetryOn5xx` to exercise same-target retries. Both generic + * retry policies draw from the request budget and report physical sends through + * `onSendsConsumed`. Reset-only still hops on the first HTTP 5xx; its key-rotation and socket + * reset counts are covered by the server-key-failover end-to-end fixture instead. */ const originalFetch = globalThis.fetch; @@ -87,6 +92,73 @@ const totalSends = (logCtx: RequestLogContext): number => sendCounts(logCtx).reduce((sum, count) => sum + count, 0); describe("upstream sends per logical request", () => { + test.each(["prepare-throw", "prepare-abort", "child-throw", "child-abort", "settled-send", "opaque-send", "nested-observed", "opaque-success"] as const)( + "combo reservation cleanup preserves observed work and refunds local exits (%s)", async mode => { + const budget = createRequestExecutionBudget(); + const translatorBudget = createTranslatorBudget(); + const controller = new AbortController(); + const failure = new Error("fixture pre-dispatch failure"); + const originalPrepare = comboRequests.concreteComboRequestBody; + const preparation = mode.startsWith("prepare-") + ? spyOn(comboRequests, "concreteComboRequestBody").mockImplementationOnce((...args) => { + if (mode === "prepare-throw") throw failure; + controller.abort(); + return originalPrepare(...args); + }) : undefined; + let children = 0; + try { + const pending = executeComboResponses(responsesRequest("combo/fan"), { model: "combo/fan", input: "hello", stream: false }, + "fan", comboOverTargets(1), { model: "", provider: "" }, { translatorBudget, sendBudget: budget, abortSignal: controller.signal }, { + handleComboResponses: async () => { throw new Error("Unexpected nested combo"); }, + handleResponses: async (_request, _config, log, options) => { + children += 1; + if (mode === "child-throw") throw failure; + if (mode === "child-abort") { controller.abort(); return new Response(null, { status: 499 }); } + if (mode === "settled-send") { + const send = (options!.sendBudget as RequestExecutionBudget).reserveDispatch({ sendClass: "transient", targetKey: "fixture" }); + if (!send.allowed || !send.permit.use()) throw new Error("Expected prepaid child send"); + } + if (mode === "opaque-send") log.activeAttempt!.sendCount = 1; + if (mode === "nested-observed") log.attempts = [{ ...log.activeAttempt!, sendCount: 1 }]; + return Response.json({ error: { message: "fixture provider failure" } }, { status: mode === "opaque-success" ? 200 : 502 }); + }, + }); + if (mode.endsWith("throw")) await expect(pending).rejects.toBe(failure); + else await (await pending).text(); + expect(children).toBe(mode.startsWith("prepare-") ? 0 : 1); + expect(budget.used).toBe(mode.startsWith("prepare-") ? 0 : 1); + } finally { preparation?.mockRestore(); translatorBudget.dispose(); } + }, + ); + + test("a local input refusal refunds its combo booking so later targets keep four plus two sends", async () => { + const cfg = comboOverTargets(3); + cfg.providers.t0!.contextWindow = 1; + for (const name of ["t1", "t2"]) cfg.providers[name] = { + ...cfg.providers[name]!, adapter: "google", googleMode: "vertex", baseUrl: "https://aiplatform.googleapis.com", + }; + cfg.providers.t1!.apiKeyPool = [{ id: "a", key: "sk-t1" }, { id: "b", key: "sk-t1-recovery" }]; + saveConfig(cfg); + const seen: string[] = []; + globalThis.fetch = (async (_input, init) => { + const key = new Headers(init?.headers).get("x-goog-api-key") ?? ""; + seen.push(key); + const status = key === "sk-t1" && seen.length === 3 ? 429 : 502; + return Response.json({ error: { message: "fixture provider failure" } }, { status, headers: { "Retry-After": "0" } }); + }) as typeof fetch; + const logCtx: RequestLogContext = { model: "", provider: "" }; + const budget = createRequestExecutionBudget(); + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/fan", stream: false, input: "large input ".repeat(100) }), + }), cfg, logCtx, { sendBudget: budget }); + expect(response.status).toBe(502); + await response.text(); + expect(seen).toEqual(["sk-t1", "sk-t1", "sk-t1", "sk-t1-recovery", "sk-t2", "sk-t2"]); + expect(sendCounts(logCtx)).toEqual([0, 4, 2]); + expect(budget.used).toBe(6); + }); + test("a 5xx streak on a single target spends the base allowance and stops", async () => { const upstream = alwaysFailing(502, "upstream busy"); const logCtx: RequestLogContext = { model: "", provider: "" }; @@ -120,7 +192,7 @@ describe("upstream sends per logical request", () => { expect(sendCounts(logCtx)).toEqual([3]); }); - test("a three-target combo fan-out gives every declared target a send and totals six", async () => { + test("a three-target 5xx combo reaches every target within six physical sends", async () => { const upstream = alwaysFailing(502, "upstream busy"); const logCtx: RequestLogContext = { model: "", provider: "" }; @@ -133,10 +205,8 @@ describe("upstream sends per logical request", () => { // the later targets to zero. The first target runs its own ladder, each later target draws // what is left, and the clamp holds back one send for every target still declared, so the // last target is still reached. - // Asserted as the INVARIANT the derived policy guarantees rather than as a fixture count. - // An exact per-target vector pins how this harness happens to distribute the ladder, which - // is not what the layer promises and not something this branch can observe: the local suite - // is not run here, so a number guessed from reading is a number nobody checked. + // Count physical sends through the adapter; unit-only shared-counter assertions are not + // enough to show that retries and later targets consume the same reservation ledger. const bearers = upstream.authorizations; // Every declared target is still reached. Starving the last target is the failure mode that // sharing one counter WITHOUT a per-target policy produces. @@ -147,15 +217,10 @@ describe("upstream sends per logical request", () => { // Bounded by the derived total: the first target's ladder, one send per further declared // target, and the single shared final-recovery reserve. The measured regression in #4546 was // twelve, four per target, because each child drew a fresh full allowance. - // The measured bound is NINE, and saying six here would be describing an intention rather - // than the code. #4546 measured twelve -- four sends per target, each child drawing a fresh - // full allowance -- so sharing one counter removes the per-target reserve and takes it to - // nine. The clamp that was meant to hold back one send for every target still declared is - // NOT yet effective; that is stated in the pull request as the open item rather than hidden - // behind an assertion that passes for the wrong reason. - expect(bearers.length).toBeLessThanOrEqual(9); - expect(bearers.length).toBeLessThan(12); - expect(bearers.length).toBeGreaterThanOrEqual(3); + // The final combo hop can use the shared recovery allowance. A prepaid hop must settle + // against its first physical send rather than shrinking the next target's ladder twice. + expect(bearers).toHaveLength(6); + expect(totalSends(logCtx)).toBe(6); }); // REMOVED: "a 401 before the 5xx streak spends one of the same three sends". diff --git a/tests/server/server-combo-failover-e2e.test.ts b/tests/server/server-combo-failover-e2e.test.ts index f636e3480f..0286ba698c 100644 --- a/tests/server/server-combo-failover-e2e.test.ts +++ b/tests/server/server-combo-failover-e2e.test.ts @@ -48,6 +48,7 @@ import { clearKeyCooldowns } from "../../src/providers/key-failover"; import { consumeComboFailure, createChildPassthroughCallbackGate } from "../../src/server/responses/core"; import { clearComboRecallForTests, recallComboForLane, reconcileComboRecall } from "../../src/server/responses/combo-session-recall"; import { captureConfigGeneration } from "../../src/lib/state-store-sweeper"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; // Full-suite Windows load: startServer + combo rename/delete management flows exceed the // default 5s per-test budget (same flake class as 810fa115 / claude-management-api). @@ -1939,7 +1940,10 @@ describe("server combo failover 030 activation matrix", () => { }); test("runTurn combo attempts retain requested effort without adapter wire metadata", async () => { + const sendBudget = createRequestExecutionBudget(); + const dispatchedModels: string[] = []; customRunTurn = async (parsed, _incoming, emit) => { + dispatchedModels.push(parsed.modelId); if (parsed.modelId === "m1") { emit({ type: "error", message: "first target unavailable" }); return; @@ -1954,9 +1958,11 @@ describe("server combo failover 030 activation matrix", () => { const response = await postLogged(config, { reasoning: { effort: "high" }, - }); + }, { sendBudget }); expect(response.status).toBe(200); expect(JSON.stringify(await response.json())).toContain("runTurn backup"); + expect(dispatchedModels).toEqual(["m1", "m2"]); + expect(sendBudget.used).toBe(2); const { log, usage } = await latestAttemptReceipts(config); for (const receipt of [log, usage]) { @@ -1975,6 +1981,7 @@ describe("server combo failover 030 activation matrix", () => { }); test("hosted web-search eager model failure hops through the loop path", async () => { + const sendBudget = createRequestExecutionBudget(); const modelHits: Array<{ model?: string; hasWebTool: boolean; authorization: string | null; account: string | null }> = []; const routed = serve(async request => { const body = await request.json() as { model?: string; tools?: Array<{ type?: string }> }; @@ -2007,7 +2014,7 @@ describe("server combo failover 030 activation matrix", () => { const response = await post(config, { stream: true, tools: [{ type: "web_search" }], - }, { onResponseComplete: model => models.push(model) }, { + }, { sendBudget, onResponseComplete: model => models.push(model) }, { session_id: "web-search-recall", authorization: `Bearer ${fakeChatGptJwt({ chatgpt_account_id: "acct-combo-search" })}`, "chatgpt-account-id": "acct-combo-search", @@ -2015,6 +2022,7 @@ describe("server combo failover 030 activation matrix", () => { expect(response.status).toBe(200); expect(JSON.stringify(await collectSse(response))).toContain("web loop backup"); expect(modelHits.map(hit => hit.model)).toEqual(["m1", "m2"]); + expect(sendBudget.used).toBe(2); expect(modelHits.every(hit => hit.hasWebTool)).toBe(true); expect(modelHits.map(hit => hit.authorization)).toEqual(["Bearer key-a", "Bearer key-b"]); expect(modelHits.every(hit => hit.account === null)).toBe(true); diff --git a/tests/server/server-google-antigravity-oauth-401-replay.test.ts b/tests/server/server-google-antigravity-oauth-401-replay.test.ts index b1f55c238c..3ecca2131a 100644 --- a/tests/server/server-google-antigravity-oauth-401-replay.test.ts +++ b/tests/server/server-google-antigravity-oauth-401-replay.test.ts @@ -9,6 +9,8 @@ import { startServer } from "../../src/server"; import type { OcxConfig } from "../../src/types"; import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { handleResponses } from "../../src/server/responses"; +import { createRequestExecutionBudget, CODEX_TEXT_GUARDED_BUDGET_POLICY } from "../../src/lib/request-execution-budget"; const GOOGLE_TOKEN_ENDPOINT = "https://oauth2.googleapis.com/token"; const PROD_API_BASE = "https://cloudcode-pa.googleapis.com"; @@ -274,6 +276,28 @@ function installOAuthFetch( } describe("Google Antigravity OAuth upstream 401 replay", () => { + for (const native of [false, true]) test.each(["allowed", "denied", "refresh-failed"] as const)( + `OAuth401 after base exhaustion reserves before refresh (native=${native}, %s)`, async mode => { + await seedOAuth(); + const cfg = native ? antigravityPassthroughConfig() : antigravityConfig(); + const observed = installOAuthFetch([503, 503, 401, 200], mode === "refresh-failed" ? { tokenErrorDescription: "fixture invalid grant" } : {}); + const budget = createRequestExecutionBudget({ ...CODEX_TEXT_GUARDED_BUDGET_POLICY, + finalRecoveryAllowance: mode === "denied" ? 0 : 1 }); + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "google-antigravity/gemini-3.8-flash", input: "hello", stream: false }), + }), cfg, { model: "", provider: "" }, { sendBudget: budget }); + const body = await response.text(); + const succeeds = mode === "allowed"; + expect({ status: response.status, refreshes: observed.counts.refresh, sends: observed.chatAuth.length, used: budget.used }) + .toEqual({ status: succeeds ? 200 : 401, refreshes: mode === "denied" ? 0 : 1, sends: succeeds ? 4 : 3, used: succeeds ? 4 : 3 }); + expect(budget.reserveSpent).toBe(succeeds); + expect(body).not.toContain("budget exhausted"); + if (succeeds) expect(observed.chatAuth[3]).toBe("Bearer fresh-access"); + else expect(getAccountSet("google-antigravity")!.accounts[0]!.credential.access).toBe("rejected-access"); + }, 20_000, + ); + test.each([200, 401])("native passthrough replays once and returns the second HTTP %i", async secondStatus => { await seedOAuth(); saveConfig(antigravityPassthroughConfig()); diff --git a/tests/server/server-key-failover-e2e.test.ts b/tests/server/server-key-failover-e2e.test.ts index 418ef993ca..eb3a5a05ef 100644 --- a/tests/server/server-key-failover-e2e.test.ts +++ b/tests/server/server-key-failover-e2e.test.ts @@ -1,9 +1,9 @@ -import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; import { mkdtempSync} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { loadConfig, saveConfig } from "../../src/config"; -import { clearKeyCooldowns, rotateKeyOn429 } from "../../src/providers/key-failover"; +import { clearKeyCooldowns, getKeyCooldownUntil, rotateKeyOn429 } from "../../src/providers/key-failover"; import { deriveXaiConvId } from "../../src/providers/xai-transport"; import { clearReasoningReplayCacheForTests } from "../../src/responses/reasoning-replay-cache"; import { startServer } from "../../src/server"; @@ -15,6 +15,10 @@ import { resetProviderRequestPacingForTest, setProviderRequestPacingRuntimeForTe import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; import { routedProviderConfig } from "../../src/router"; import type { OcxProviderTransport } from "../../src/providers/xai-transport"; +import { getAccountSet, saveCredential, setActiveAccount } from "../../src/oauth/store"; +import { clearGenericFailoverHealth } from "../../src/oauth/generic-account-failover"; +import { handleResponses } from "../../src/server/responses"; +import { createRequestExecutionBudget, CODEX_TEXT_GUARDED_BUDGET_POLICY } from "../../src/lib/request-execution-budget"; let testDir = ""; let previousHome: string | undefined; @@ -43,6 +47,359 @@ afterEach(() => { }); describe("server 429 key failover (end-to-end)", () => { + test("static-key 401 refunds a reserved hop when no alternate credential remains", async () => { + const cfg = { defaultProvider: "key401-no-alternate", providers: { + "key401-no-alternate": { adapter: "openai-chat", baseUrl: "https://key401-no-alternate.invalid/v1", authMode: "key", + apiKey: "fixture-key-a", models: ["fixture"], apiKeyPool: [{ id: "a", key: "fixture-key-a" }, { id: "b", key: "fixture-key-b" }], + }, + } } as OcxConfig; + saveConfig(cfg); + const budget = createRequestExecutionBudget(); + const originalFetch = globalThis.fetch; + let sends = 0; + globalThis.fetch = (async () => { sends += 1; return Response.json({ error: { message: "no-alternate-401" } }, { status: 401 }); }) as typeof fetch; + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "key401-no-alternate/fixture", input: "hello", stream: false }), + }), cfg, { model: "", provider: "" }, { sendBudget: budget }); + expect({ status: response.status, selected: loadConfig().providers["key401-no-alternate"]!.apiKey, sends, used: budget.used }) + .toEqual({ status: 401, selected: "fixture-key-b", sends: 2, used: 2 }); + expect(await response.text()).toContain("no-alternate-401"); + } finally { globalThis.fetch = originalFetch; } + }); + + test.each([0, 1])("static-key 401 admits replay before selecting another credential (reserve=%i)", async reserve => { + const cfg = { defaultProvider: "key401-reserve", providers: { + "key401-reserve": { adapter: "openai-chat", baseUrl: "https://key401-reserve.invalid/v1", authMode: "key", + apiKey: "fixture-key-a", models: ["fixture"], apiKeyPool: [ + { id: "a", key: "fixture-key-a" }, { id: "b", key: "fixture-key-b" }, { id: "c", key: "fixture-key-c" }, + ], + }, + } } as OcxConfig; + saveConfig(cfg); + const budget = createRequestExecutionBudget({ ...CODEX_TEXT_GUARDED_BUDGET_POLICY, finalRecoveryAllowance: reserve }); + const originalFetch = globalThis.fetch; + let sends = 0; + globalThis.fetch = (async () => { + sends += 1; + if (sends <= 2) throw Object.assign(new Error("fixture ECONNRESET"), { code: "ECONNRESET" }); + return Response.json({ error: { message: "static401-evidence" } }, { status: 401 }); + }) as typeof fetch; + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "key401-reserve/fixture", input: "hello", stream: false }), + }), cfg, { model: "", provider: "" }, { sendBudget: budget }); + const body = await response.text(); + expect({ status: response.status, selected: loadConfig().providers["key401-reserve"]!.apiKey, sends, used: budget.used }) + .toEqual({ status: 401, selected: reserve ? "fixture-key-b" : "fixture-key-a", sends: 3 + reserve, used: 3 + reserve }); + expect(body).toContain("static401-evidence"); + } finally { globalThis.fetch = originalFetch; } + }); + + test.each(["initial-success", "retry-success", "initial-429", "initial-401", "continuation-429", "reserve-refused"] as const)( + "Vertex budget-aware recovery uses its final send and one pacing slot per inference (%s)", async mode => { + let now = 0, timerId = 0; + const timers = new Set(); + setProviderRequestPacingRuntimeForTest({ now: () => now, setTimer(callback, delayMs) { + const id = ++timerId; timers.add(id); + queueMicrotask(() => { if (timers.delete(id)) { now += delayMs; callback(); } }); + return id; + }, clearTimer(id) { timers.delete(id as number); }, enqueueMicrotask: queueMicrotask }); + const cfg = { defaultProvider: "vertex-budget", emptyCompletionRetry: mode === "continuation-429", providers: { + "vertex-budget": { adapter: "google", googleMode: "vertex", authMode: "key", + baseUrl: "https://aiplatform.googleapis.com", models: ["gemini-3-pro"], apiKey: "vertex-key-a", + apiKeyPool: [{ id: "a", key: "vertex-key-a" }, { id: "b", key: "vertex-key-b" }], + requestPacing: { enabled: true, minIntervalMs: 100 }, + }, + } } as OcxConfig; + saveConfig(cfg); + const originalFetch = globalThis.fetch; + const keys: Array = [], times: number[] = []; + const budget = createRequestExecutionBudget({ ...CODEX_TEXT_GUARDED_BUDGET_POLICY, + finalRecoveryAllowance: mode === "reserve-refused" ? 0 : 1 }); + const expectedSends = mode === "initial-success" ? 1 : mode === "retry-success" ? 2 : mode === "reserve-refused" ? 3 : 4; + globalThis.fetch = (async (_input, init) => { + keys.push(new Headers(init?.headers).get("x-goog-api-key")); times.push(now); + if (keys.length > 4) throw new Error("fixture send ceiling exceeded"); + if (mode === "continuation-429" && keys.length === 1) return Response.json({ + candidates: [{ content: { role: "model", parts: [] }, finishReason: "STOP" }], + }); + if (mode !== "reserve-refused" && keys.length === expectedSends) return Response.json({ + candidates: [{ content: { role: "model", parts: [{ text: "vertex recovered" }] }, finishReason: "STOP" }], + }); + const status = keys.length < 3 ? 503 : mode === "initial-401" ? 401 : 429; + return Response.json({ error: { code: status, message: "temporary failure" } }, + { status, headers: { "Retry-After": "0" } }); + }) as typeof fetch; + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "vertex-budget/gemini-3-pro", input: "hello", stream: false }), + }), cfg, { model: "", provider: "" }, { sendBudget: budget }); + const body = await response.text(); + expect(keys).toEqual(Array.from({ length: expectedSends }, (_, index) => index === 3 ? "vertex-key-b" : "vertex-key-a")); + expect(response.status).toBe(mode === "reserve-refused" ? 429 : 200); + expect(body).toContain(mode === "reserve-refused" ? "temporary failure" : "vertex recovered"); + expect(budget.used).toBe(expectedSends); + expect(times).toEqual(Array.from({ length: expectedSends }, (_, index) => index * 100)); + if (mode === "reserve-refused") expect(loadConfig().providers["vertex-budget"]!.apiKey).toBe("vertex-key-a"); + } finally { globalThis.fetch = originalFetch; resetProviderRequestPacingForTest(); } + }, 20_000, + ); + + test.each([false, true])("OAuth hops charge each reset-only refetch once (third succeeds: %s)", async succeeds => { + const originalFetch = globalThis.fetch; + const authorizations: string[] = []; + let server: ReturnType | undefined; + clearGenericFailoverHealth(); + try { + for (let i = 0; i < 3; i++) { + await saveCredential("nous", { access: `synthetic-oauth-${i}`, refresh: `synthetic-refresh-${i}`, + expires: Date.now() + 3_600_000, accountId: `fixture-account-${i}` }, { addAccount: true }); + } + await setActiveAccount("nous", getAccountSet("nous")!.accounts[0]!.id); + saveConfig({ port: 0, hostname: "127.0.0.1", defaultProvider: "nous", providers: { nous: { + adapter: "openai-chat", authMode: "oauth", baseUrl: "https://oauth-refetch.invalid/v1", models: ["test"], + } } } as OcxConfig); + server = startServer(0); + globalThis.fetch = (async (_input, init) => { + authorizations.push(new Headers(init?.headers).get("authorization") ?? ""); + if (succeeds && authorizations.length === 3) return Response.json({ id: "third-account", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "third account works" }, finish_reason: "stop" }] }); + return Response.json({ error: { message: `oauth-quota-${authorizations.length}`, type: "rate_limit_error" } }, + { status: 429, headers: { "retry-after": "30" } }); + }) as typeof fetch; + const response = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "nous/test", stream: false, input: "hello" }), + }); + const text = await response.text(); + expect(authorizations).toEqual(["Bearer synthetic-oauth-0", "Bearer synthetic-oauth-1", "Bearer synthetic-oauth-2"]); + expect(response.status).toBe(succeeds ? 200 : 429); + expect(text).toContain(succeeds ? "third account works" : "oauth-quota-3"); + } finally { + globalThis.fetch = originalFetch; + await server?.stop(true); + clearGenericFailoverHealth(); + } + }); + + test.each(["429", "5xx", "reset"] as const)("reset-only combo key rotations preserve later targets (%s)", async mode => { + const originalFetch = globalThis.fetch; + const counts = [0, 0, 0]; + const targets = counts.map((_, index) => ({ provider: `reset-key-t${index}`, model: "test" })); + const providers = Object.fromEntries(targets.map(({ provider }) => [provider, { + adapter: "openai-chat", authMode: "key", baseUrl: `https://${provider}.invalid/v1`, + apiKey: `synthetic-${provider}-0`, + apiKeyPool: Array.from({ length: 6 }, (_, key) => ({ id: `k${key}`, key: `synthetic-${provider}-${key}` })), + }])); + const config = { port: 0, hostname: "127.0.0.1", defaultProvider: targets[0]!.provider, + providers, combos: { fan: { strategy: "failover", targets } }, + } as OcxConfig; + let server: ReturnType | undefined; + try { + saveConfig(config); + server = startServer(0); + globalThis.fetch = (async (input) => { + const url = new URL(input instanceof Request ? input.url : String(input)); + const index = targets.findIndex(target => url.hostname === `${target.provider}.invalid`); + if (index < 0) throw new Error("unexpected reset-only combo fixture request"); + counts[index]!++; + if (counts.reduce((a, b) => a + b, 0) > 18) throw new Error("fixture send ceiling exceeded"); + if (mode === "reset" && (index !== 0 || counts[index]! > 1)) { + throw Object.assign(new Error("socket reset fixture"), { code: "ECONNRESET" }); + } + return Response.json({ error: { message: "key quota exhausted", type: "rate_limit_error" } }, + { status: mode === "5xx" ? 502 : 429, headers: { "retry-after": "0" } }); + }) as typeof fetch; + const response = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.timeout(10_000), + body: JSON.stringify({ model: "combo/fan", input: "hello", stream: false }), + }); + await response.text(); + expect(response.ok).toBe(false); + expect(counts).toEqual(mode === "5xx" ? [1, 1, 1] : mode === "reset" ? [3, 2, 1] : [4, 1, 1]); + expect(counts.reduce((a, b) => a + b, 0)).toBe(mode === "5xx" ? 3 : 6); + } finally { globalThis.fetch = originalFetch; await server?.stop(true); } + }, 15_000); + + test.each(["web-search", "image"] as const)( + "%s bridge bounds rotations after short cooldowns expire", + async bridge => { + const originalFetch = globalThis.fetch; + const endpoint = "https://sidecar-key429-fixture.invalid/v1/chat/completions"; + const seen: Array<{ authorization: string | null; tools: string[] }> = []; + let now = Date.now(); + let restoreClock: (() => void) | undefined; + let server: ReturnType | undefined; + const config = { + port: 0, hostname: "127.0.0.1", defaultProvider: "sidecar429", + providers: { + sidecar429: { + adapter: "openai-chat", authMode: "key", baseUrl: "https://sidecar-key429-fixture.invalid/v1", + apiKey: "synthetic-sidecar-a", apiKeyPool: [ + { id: "a", key: "synthetic-sidecar-a" }, { id: "b", key: "synthetic-sidecar-b" }, + ], + }, + // Arms image planning without OAuth or an actual image-service request. + ...(bridge === "image" ? { xai: { + adapter: "openai-chat", authMode: "key", baseUrl: "https://image-plan-fixture.invalid/v1", + apiKey: "synthetic-image-plan-token", + } } : {}), + }, + ...(bridge === "web-search" + ? { webSearchSidecar: { enabled: true, backend: "exa", exaApiKey: "synthetic-exa-plan-token" } } + : { images: { bridgeEnabled: true } }), + } as OcxConfig; + try { + saveConfig(config); + server = startServer(0); + const clock = spyOn(Date, "now").mockImplementation(() => now); + restoreClock = () => clock.mockRestore(); + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url !== endpoint) throw new Error("unexpected outbound request in sidecar 429 fixture"); + const body = JSON.parse(String(init?.body)) as { tools?: Array<{ function?: { name?: string } }> }; + seen.push({ authorization: new Headers(init?.headers).get("authorization"), + tools: (body.tools ?? []).map(tool => tool.function?.name ?? ""), + }); + // A broken rotation loop is bounded by six mocked sends, never by an infinite wait. + if (seen.length >= 6) throw new Error("sidecar 429 fixture send ceiling exceeded"); + now += 1_000; + return Response.json({ error: { message: `sidecar429-final-${seen.length}`, type: "rate_limit_error" } }, + { status: 429, headers: { "retry-after": "0" } }); + }) as typeof fetch; + const response = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.timeout(10_000), + body: JSON.stringify({ model: "sidecar429/test", input: "Use the provided tool.", stream: true, + tools: [{ type: bridge === "web-search" ? "web_search" : "image_generation" }], + }), + }); + const text = await response.text(); + expect(seen.map(call => call.authorization)).toEqual(["Bearer synthetic-sidecar-a", "Bearer synthetic-sidecar-b"]); + // Verifies bridge activation rather than accidentally exercising the generic 429 loop. + for (const call of seen) expect(call.tools).toContain(bridge === "web-search" ? "web_search" : "image_gen"); + expect(response.status).toBe(429); + expect(text).toContain("Provider error 429"); + expect(text).toContain("sidecar429-final-2"); + expect(getKeyCooldownUntil("sidecar429", "b", now)).toBe(now + 1); + } finally { + globalThis.fetch = originalFetch; + restoreClock?.(); + await server?.stop(true); + } + }, 15_000, + ); + + test.each(["exhausted", "continuation", "transient", "budget-exhausted", "recovery-success", "unpooled"] as const)( + "429 rotation stays request-bounded after every earlier cooldown expires (%s)", + async mode => { + const originalFetch = globalThis.fetch; + const endpoint = "https://key429-fixture.invalid/v1/chat/completions"; + const usesFinalReserve = mode === "budget-exhausted" || mode === "recovery-success"; + const expectedSends = mode === "exhausted" ? 2 : usesFinalReserve ? 4 : 3; + const seen: string[] = []; + const cancelled: number[] = []; + let now = Date.now(); + let restoreClock: (() => void) | undefined; + let server: ReturnType | undefined; + const config = { + port: 0, hostname: "127.0.0.1", defaultProvider: "key429fixture", + providers: { key429fixture: { + adapter: "openai-chat", baseUrl: "https://key429-fixture.invalid/v1", authMode: "key", + apiKey: mode === "unpooled" ? "synthetic-key-outside" : "synthetic-key-a", apiKeyPool: [ + { id: "a", key: "synthetic-key-a" }, { id: "b", key: "synthetic-key-b" }, + ...(usesFinalReserve ? [{ id: "c", key: "synthetic-key-c" }] : []), + ], + ...(mode === "continuation" ? { terminalContinuationGuard: true } : {}), + ...(mode === "transient" || usesFinalReserve ? { transientRetryOn5xx: { enabled: true, attempts: 3 } } : {}), + } }, + } as OcxConfig; + try { + saveConfig(config); + server = startServer(0); + const clock = spyOn(Date, "now").mockImplementation(() => now); + restoreClock = () => clock.mockRestore(); + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url !== endpoint) throw new Error("unexpected outbound request in key-rotation fixture"); + seen.push(new Headers(init?.headers).get("authorization") ?? ""); + // The old loop must terminate deterministically instead of waiting for a test timeout. + if (seen.length > 6) throw new Error("key-rotation fixture send ceiling exceeded"); + const send = seen.length; + // Retry-After: 0 means a 1ms cooldown. Every subsequent response arrives after it. + now += 1_000; + if (mode === "recovery-success" && send === 4) { + return Response.json({ id: "chatcmpl-final-recovery", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "recovered-with-final-send" }, finish_reason: "stop" }], + usage: { prompt_tokens: 2, completion_tokens: 2, total_tokens: 4 }, + }); + } + if (mode === "continuation" && send === 2) { + return Response.json({ id: "chatcmpl-plan", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "I will edit the file now." }, finish_reason: "stop" }], + usage: { prompt_tokens: 2, completion_tokens: 2, total_tokens: 4 }, + }); + } + const status = (mode === "transient" && send === 1) || (usesFinalReserve && send < 3) ? 503 : 429; + const text = JSON.stringify({ error: { message: `key429-final-${send}`, type: "rate_limit_error" } }); + const bytes = new TextEncoder().encode(text); + const body = new ReadableStream({ + start(controller) { controller.enqueue(bytes); controller.close(); }, + cancel() { cancelled.push(send); }, + }); + return new Response(body, { status, headers: { + "content-type": "application/json", "retry-after": "0", + } }); + }) as typeof fetch; + const result = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "key429fixture/test", stream: false, + input: mode === "continuation" ? "Edit the file using the tool." : "hello", + ...(mode === "continuation" ? { tools: [{ type: "function", name: "edit_file", + description: "Synthetic edit tool; never executed", parameters: { type: "object", properties: {} }, + }] } : {}), + }), + }); + const text = await result.text(); + expect(seen).toHaveLength(expectedSends); + expect(seen).toEqual(mode === "unpooled" + ? ["Bearer synthetic-key-outside", "Bearer synthetic-key-a", "Bearer synthetic-key-b"] + : usesFinalReserve + ? ["Bearer synthetic-key-a", "Bearer synthetic-key-a", "Bearer synthetic-key-a", "Bearer synthetic-key-b"] + : mode === "transient" + ? ["Bearer synthetic-key-a", "Bearer synthetic-key-a", "Bearer synthetic-key-b"] + : mode === "continuation" + ? ["Bearer synthetic-key-a", "Bearer synthetic-key-b", "Bearer synthetic-key-b"] + : ["Bearer synthetic-key-a", "Bearer synthetic-key-b"]); + // Exhausted initial rotation must not cancel the final error it returns to the caller. + if (mode === "recovery-success") { + // The admission probe must not spend the reserve before the real B dispatch. + expect(result.status).toBe(200); + expect(text).toContain("recovered-with-final-send"); + } else if (mode !== "continuation") { + expect(result.status).toBe(429); + expect(result.headers.get("retry-after")).toBe("0"); + expect(text).toContain(`key429-final-${expectedSends}`); + expect(cancelled).not.toContain(expectedSends); + } else { + // A continuation error is represented inside the already-started Responses result. + expect(text).toContain("key429-final-3"); + } + expect(cancelled).toContain(usesFinalReserve ? 3 : mode === "transient" ? 2 : 1); + expect(getKeyCooldownUntil("key429fixture", "b", now)).toBe(mode === "recovery-success" ? null : now + 1); + } finally { + globalThis.fetch = originalFetch; + restoreClock?.(); + await server?.stop(true); + } + }, 15_000, + ); + test("physical key selection rejects disabled, removed, and changed-auth providers", () => { const provider = { adapter: "openai-chat", baseUrl: "https://example.test/v1", authMode: "key", apiKey: "synthetic-first" } as const; const config = { providers: { current: { ...provider } } } as unknown as OcxConfig; diff --git a/tests/server/server-kiro-oauth-401-replay.test.ts b/tests/server/server-kiro-oauth-401-replay.test.ts index 80c39e6f3c..2dc27b6b94 100644 --- a/tests/server/server-kiro-oauth-401-replay.test.ts +++ b/tests/server/server-kiro-oauth-401-replay.test.ts @@ -5,8 +5,12 @@ import { join } from "node:path"; import { KIRO_COMPLETION_TOOL_NAME } from "../../src/adapters/kiro-constants"; import { encodeMessage } from "../../src/lib/eventstream-decoder"; import { saveConfig } from "../../src/config"; -import { saveCredential } from "../../src/oauth/store"; +import { getAccountSet, saveCredential, setActiveAccount } from "../../src/oauth/store"; +import { clearGenericFailoverHealth } from "../../src/oauth/generic-account-failover"; +import { resetKiroThrottleStateForTests } from "../../src/adapters/kiro-retry"; import { startServer } from "../../src/server"; +import { handleResponses } from "../../src/server/responses"; +import { createRequestExecutionBudget, CODEX_TEXT_GUARDED_BUDGET_POLICY } from "../../src/lib/request-execution-budget"; import type { OcxConfig } from "../../src/types"; import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; import { removeTreeWithRetry } from "../helpers/remove-tree"; @@ -142,6 +146,88 @@ function installFetch(chatStatuses: number[]): { chatAuth: string[]; refreshCall } describe("Kiro OAuth upstream 401 replay", () => { + test.each([1, 0])("Kiro empty completion after two resets uses only its available repair reserve (%i)", async reserve => { + resetKiroThrottleStateForTests(); + await seedOAuth(); + const cfg = { ...config(), emptyCompletionRetry: true }; + const budget = createRequestExecutionBudget({ ...CODEX_TEXT_GUARDED_BUDGET_POLICY, finalRecoveryAllowance: reserve }); + let sends = 0; + globalThis.fetch = (async input => { + const url = input instanceof Request ? input.url : String(input); + if (url !== CHAT_ENDPOINT) throw new Error(`Unexpected fixture request: ${url}`); + sends += 1; + if (sends <= 2) throw Object.assign(new Error("fixture ECONNRESET"), { code: "ECONNRESET" }); + return new Response(sends === 3 ? reasoningStream("Thinking without an answer") : eventStream("repair completed"), { + headers: { "content-type": "application/vnd.amazon.eventstream" }, + }); + }) as typeof fetch; + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "kiro/claude-sonnet-4.5", input: "hello", stream: false }), + }), cfg, { model: "", provider: "" }, { sendBudget: budget }); + const text = await response.text(); + expect(sends).toBe(3 + reserve); + expect(budget.used).toBe(sends); + expect(budget.reserveSpent).toBe(reserve === 1); + if (reserve) expect(text).toContain("repair completed"); + else expect(text).not.toContain("repair completed"); + } finally { resetKiroThrottleStateForTests(); } + }, 20_000); + + test.each(["quota", "success", "reset-success", "empty-retry"] as const)("Kiro OAuth hops charge adapter-owned physical sends once (%s)", async mode => { + const authorizations: string[] = []; + clearGenericFailoverHealth(); + resetKiroThrottleStateForTests(); + for (let index = 0; index < 3; index++) { + await saveCredential("kiro", { + access: `synthetic-kiro-${index}`, refresh: `synthetic-refresh-${index}`, + expires: Date.now() + 3_600_000, accountId: `kiro-fixture-${index}`, source: "oauth", + }, { addAccount: true }); + } + await setActiveAccount("kiro", getAccountSet("kiro")!.accounts[0]!.id); + expect(getAccountSet("kiro")!.accounts).toHaveLength(3); + saveConfig({ ...config(), ...(mode === "empty-retry" ? { emptyCompletionRetry: true } : {}) }); + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url !== CHAT_ENDPOINT) throw new Error(`Unexpected fixture request: ${url}`); + const bearer = new Headers(init?.headers).get("authorization") ?? ""; + authorizations.push(bearer); + if (mode === "empty-retry" && authorizations.length === 1) { + // Reasoning-only ends with done but no user content; a truly empty Kiro stream is incomplete. + return new Response(reasoningStream("Thinking without an answer"), { + headers: { "content-type": "application/vnd.amazon.eventstream" }, + }); + } + if (mode === "reset-success" && authorizations.length === 1) { + throw Object.assign(new Error("fixture ECONNRESET"), { code: "ECONNRESET" }); + } + if (mode !== "quota" && bearer === "Bearer synthetic-kiro-2") { + return new Response(eventStream("third Kiro account works"), { + headers: { "content-type": "application/vnd.amazon.eventstream" }, + }); + } + // Exhausted quota bypasses Kiro's same-account throttle ladder and reaches OAuth rotation. + return Response.json({ message: "insufficient_quota", code: "insufficient_quota" }, { status: 429 }); + }) as typeof fetch; + const server = startServer(0); + try { + const response = await post(server); + const body = await response.text(); + expect(authorizations).toEqual([ + ...(mode === "reset-success" || mode === "empty-retry" ? ["Bearer synthetic-kiro-0"] : []), + "Bearer synthetic-kiro-0", "Bearer synthetic-kiro-1", "Bearer synthetic-kiro-2", + ]); + expect(response.status).toBe(mode === "quota" ? 429 : 200); + expect(body).toContain(mode === "quota" ? "insufficient_quota" : "third Kiro account works"); + } finally { + globalThis.fetch = originalFetch; + await server.stop(true); + clearGenericFailoverHealth(); + resetKiroThrottleStateForTests(); + } + }, 20_000); + test("selected OAuth account supplies its own Kiro runtime region and profile", async () => { const profileArn = "arn:aws:codewhisperer:eu-west-1:123456789012:profile/account-b"; await saveCredential("kiro", { diff --git a/tests/server/server-xai-oauth-401-replay.test.ts b/tests/server/server-xai-oauth-401-replay.test.ts index aab488ccb5..f5ed0b2125 100644 --- a/tests/server/server-xai-oauth-401-replay.test.ts +++ b/tests/server/server-xai-oauth-401-replay.test.ts @@ -4,7 +4,10 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../../src/config"; import { XAI_OAUTH_DISCOVERY_URL } from "../../src/oauth/xai"; -import { saveCredential } from "../../src/oauth/store"; +import { saveCredential, getAccountSet, setActiveAccount } from "../../src/oauth/store"; +import { clearGenericFailoverHealth } from "../../src/oauth/generic-account-failover"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { handleResponses } from "../../src/server/responses"; import { XAI_GROK_CLI_BASE_URL } from "../../src/providers/xai-transport"; import { readUsageEntries, usageLogPath } from "../../src/usage/log"; import { startServer } from "../../src/server"; @@ -146,6 +149,39 @@ function installOAuthFetch( } describe("xAI OAuth Responses opt-in upstream 401 replay", () => { + test("passthrough OAuth hop retains a base retry after its prepaid first send resets", async () => { + clearGenericFailoverHealth(); + await seedOAuth(); + const firstId = getAccountSet("xai")!.activeAccountId; + await saveCredential("xai", { + access: "alternate-access", refresh: "alternate-refresh", expires: Date.now() + 3_600_000, + accountId: "xai-alternate-account", source: "oauth", + }, { addAccount: true }); + await setActiveAccount("xai", firstId); + const budget = createRequestExecutionBudget(); + const authorizations: string[] = []; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url !== OAUTH_RESPONSES_ENDPOINT) throw new Error(`Unexpected fixture request: ${url}`); + authorizations.push(new Headers(init?.headers).get("authorization") ?? ""); + if (authorizations.length === 1) return Response.json({ error: { message: "rate limited" } }, { status: 429 }); + if (authorizations.length === 2) throw Object.assign(new Error("fixture ECONNRESET"), { code: "ECONNRESET" }); + return new Response(successBody("alternate retry completed"), { headers: { "content-type": "application/json" } }); + }) as typeof fetch; + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "xai/grok-4.5", input: "hello", stream: false }), + }), xaiConfig(), { model: "", provider: "" }, { sendBudget: budget }); + const body = await response.text(); + expect(authorizations).toEqual(["Bearer rejected-access", "Bearer alternate-access", "Bearer alternate-access"]); + expect(response.status).toBe(200); + expect(body).toContain("alternate retry completed"); + expect(budget.used).toBe(3); + expect(budget.reserveSpent).toBe(false); + } finally { clearGenericFailoverHealth(); } + }); + test("initial OAuth refresh projects raw provider failures before responding", async () => { await seedOAuth(0); saveConfig(xaiConfig()); diff --git a/tests/server/terminal-guard-server.test.ts b/tests/server/terminal-guard-server.test.ts index c2c889c3d8..d725c21fa1 100644 --- a/tests/server/terminal-guard-server.test.ts +++ b/tests/server/terminal-guard-server.test.ts @@ -4,6 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../../src/config"; import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { createRequestExecutionBudget, CODEX_TEXT_GUARDED_BUDGET_POLICY } from "../../src/lib/request-execution-budget"; import { handleResponses } from "../../src/server/responses"; import type { OcxConfig } from "../../src/types"; import { removeTreeWithRetry } from "../helpers/remove-tree"; @@ -99,6 +100,7 @@ describe("server terminal guard integration", () => { afterEach(() => { globalThis.fetch = originalFetch; + clearKeyCooldowns("claude-se"); }); test("re-asks Claude once inside the same Responses turn and forwards the tool call", async () => { @@ -123,6 +125,57 @@ describe("server terminal guard integration", () => { expect(messages.at(-1)?.content?.[0]?.text).toContain("你刚才只描述了计划"); }); + test.each([ + { adapter: "anthropic", reserve: 1, rateLimited: false }, + { adapter: "openai-chat", reserve: 1, rateLimited: false }, + { adapter: "anthropic", reserve: 0, rateLimited: false }, + { adapter: "anthropic", reserve: 1, rateLimited: true }, + ] as const)("terminal repair after reset exhaustion uses only the available reserve: %j", async ({ adapter, reserve, rateLimited }) => { + const isChat = adapter === "openai-chat"; + const requestConfig = isChat ? openAiChatConfig(true) : structuredClone(config); + if (rateLimited) requestConfig.providers["claude-se"]!.apiKeyPool = [ + { id: "k1", key: "sk-test", addedAt: 1 }, + { id: "k2", key: "sk-test-2", addedAt: 2 }, + ]; + const budget = createRequestExecutionBudget({ ...CODEX_TEXT_GUARDED_BUDGET_POLICY, finalRecoveryAllowance: reserve }); + let sends = 0; + const keys: Array = []; + globalThis.fetch = (async (_input, init) => { + sends += 1; + keys.push(new Headers(init?.headers).get("x-api-key")); + if (sends <= 2) throw Object.assign(new Error("connection reset by peer"), { code: "ECONNRESET" }); + if (sends > 3 && rateLimited) return new Response("rate limited", { status: 429 }); + return isChat + ? chatSse(sends === 3 ? chatFirstTurn : chatContinuationTurn) + : anthropicSse(sends === 3 ? firstTurn : continuationTurn); + }) as typeof fetch; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: isChat ? "glm-gw/glm-5.2" : "se-claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", parameters: { type: "object" } }], + }), + }), requestConfig, { model: "", provider: "" }, { sendBudget: budget }); + const text = await response.text(); + expect(sends).toBe(3 + reserve); + expect(budget.used).toBe(sends); + expect(budget.reserveSpent).toBe(reserve === 1); + if (rateLimited) { + expect(text).toContain("Provider continuation error 429"); + expect(keys).toEqual(["sk-test", "sk-test", "sk-test", "sk-test"]); + expect(budget.alternateTargetSends).toBe(0); + } else if (reserve) { + expect(text).toContain("response.function_call_arguments.done"); + expect(text).not.toContain("Provider continuation failed"); + } else { + expect(text).toContain("Provider continuation failed"); + expect(text).not.toContain("response.function_call_arguments.done"); + } + }); + test("terminal-guard continuation 429 replays on the same key before surfacing", async () => { const retryConfig = { ...config,