diff --git a/docs-site/src/content/docs/ja/reference/management-api.md b/docs-site/src/content/docs/ja/reference/management-api.md index 4398ff81180..d76a211f454 100644 --- a/docs-site/src/content/docs/ja/reference/management-api.md +++ b/docs-site/src/content/docs/ja/reference/management-api.md @@ -136,6 +136,8 @@ Authorization: Bearer 行が既存のパーサーのサイズ上限を超えた場合、`GET /api/usage` と `GET /api/keys` は読み取れる行の集計を維持し、応答全体に `usageIncomplete: true` と `usageIncompleteReason: "oversized_rows"` を追加します。この診断はキャッシュや増分追記後も維持され、結果が空または一致なしでも返されます。再構築時には再計算されます。プロバイダー、モデル、API キーの識別子は短縮しません。フラグがないことは全行が有効だった証明にはなりません。`historyTruncated`、`entriesTruncated`、トークン測定カバレッジとは別の情報です。 +API キーの `accountLogLabel` の導出、再試行時のキー選択と使用量、フラットな `attempts` の集計規則は、[英語版の使用量帰属の仕様](/reference/management-api/#api-key-usage-attribution)を参照してください。コンボの親合計は重ねて加算せず、未報告の使用量や過去のラベルなし記録を推定しません。サブスクリプションのクォータは別の観測値です。 + `models`、`providers`、および `days[].models` の各行にも `cacheHitRate` が含まれます。これは、プロバイダーのプロンプト キャッシュから供給された入力トークンの割合で、`[0, 1]` の範囲に制限されます。プロバイダーがキャッシュ テレメトリを報告しなかった場合、または行に入力トークンがない場合は、`0` ではなく `null` になります。「キャッシュ データなし」と「実際のヒット率 0%」は異なる事実であり、それらを同じように描画するチャートは誤解を招くためです。 :::caution diff --git a/docs-site/src/content/docs/ko/reference/management-api.md b/docs-site/src/content/docs/ko/reference/management-api.md index b2c7951e89b..1605cbca830 100644 --- a/docs-site/src/content/docs/ko/reference/management-api.md +++ b/docs-site/src/content/docs/ko/reference/management-api.md @@ -140,6 +140,8 @@ Authorization: Bearer 행이 기존 파서의 크기 제한을 넘으면 `GET /api/usage`와 `GET /api/keys`는 읽을 수 있는 행의 집계를 유지하고 응답 전체에 `usageIncomplete: true`, `usageIncompleteReason: "oversized_rows"`를 추가합니다. 이 진단은 캐시와 증분 추가에서도 유지되며, 빈 결과나 필터 일치 결과가 없는 경우에도 반환됩니다. 재구축 시에는 다시 계산합니다. 행을 맞추기 위해 공급자·모델·API 키 식별자를 줄이지 않습니다. 플래그가 없다고 모든 기록이 유효했다는 뜻은 아닙니다. `historyTruncated`, `entriesTruncated`, 토큰 측정 커버리지와는 별개입니다. +API 키의 `accountLogLabel` 계산 방식, 재시도 시 키 선택과 사용량 기록, 평탄한 `attempts` 집계 규칙은 [영문 사용량 귀속 규칙](/reference/management-api/#api-key-usage-attribution)을 기준으로 합니다. 콤보 부모의 합계는 중복해서 더하지 않으며, 보고되지 않은 사용량이나 라벨이 없는 과거 기록은 추정하지 않습니다. 구독 쿼타는 별도로 관측한 값입니다. + `models`, `providers`, `days[].models`의 행에도 `cacheHitRate`가 포함됩니다. 이 값은 공급자의 프롬프트 캐시에서 제공된 입력 토큰의 비율이며 `[0, 1]` 범위로 제한됩니다. 공급자가 캐시 텔레메트리를 보고하지 않았거나 행에 입력 토큰이 없으면 `0`이 아니라 항상 `null`입니다. "캐시 데이터 없음"과 "실제 적중률 0%"는 서로 다른 사실이며, diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index b973add5d96..783c86c128d 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -246,6 +246,29 @@ final provider. Custom destinations and historic rows omit the field; consumers infer subscription usage from the current configuration, model name, or inbound API key. The log reports usage, not subscription invoice amounts. + + +API-key attempts also record `accountLogLabel` as `k` followed by 32 lowercase hex digits. +The label is the first 128 bits of SHA-256 over +`JSON.stringify(["ocx-key-account-v1", providerName, entryId ?? null, reference])`. +The reference is the configured key value captured for the physical request, before environment +or keychain resolution. Raw keys, references, and pool IDs are not written to the label field. +A consumer can derive the same label from its local configuration without resolving secrets. +This is a deterministic pseudonymous identifier, not encryption: anyone with a candidate +configuration can recompute its label. +Changing a literal key or reference changes the label; replacing the secret behind an unchanged +reference keeps the same logical account. Older unlabeled records cannot be attributed reliably. + +Key selection is recorded after queued requests have been rebuilt for the current selection. +When a retry changes keys, `attempts` retains a separate record for the preceding key, including +reported usage from failed responses. Missing usage remains unreported. Routed adapter terminals +are observed before image/search loops or continuation guards combine their usage. Consumers +sum the flat attempts by provider/account and do not add the parent combo total again. These +records identify usage; provider quota percentages remain separate upstream observations. +Command Code reasoning-effort retries consume the shared request send budget and record one +additional physical send as `reasoning-effort-downgrade`. A refused retry preserves the original +upstream error response and does not add a send. + `GET /api/usage` reads `~/.opencodex/usage.jsonl` from the beginning through the current ledger snapshot on a cold start. It processes fixed 1 MiB chunks and retains compact aggregate state rather than every normalized request row. Later refreshes validate the previous line boundary and fold only diff --git a/docs-site/src/content/docs/ru/reference/management-api.md b/docs-site/src/content/docs/ru/reference/management-api.md index 978744bbb0d..6dd261bb10a 100644 --- a/docs-site/src/content/docs/ru/reference/management-api.md +++ b/docs-site/src/content/docs/ru/reference/management-api.md @@ -158,6 +158,8 @@ GUI-сессия в стиле loopback не выпускается. Если строка превышает существующий лимит размера парсера, `GET /api/usage` и `GET /api/keys` сохраняют агрегаты читаемых строк и добавляют в ответ `usageIncomplete: true` и `usageIncompleteReason: "oversized_rows"`. Диагностика сохраняется в кеше и при инкрементальных добавлениях, в том числе для пустых результатов и отсутствующих совпадений; при перестроении она вычисляется заново. Идентификаторы провайдеров, моделей и API-ключей не сокращаются. Отсутствие флага не доказывает корректность всех строк. Это отдельный сигнал от `historyTruncated`, `entriesTruncated` и покрытия измерений токенов. +Правила вычисления `accountLogLabel` для API-ключей, выбора ключа и учёта использования при повторных запросах, а также суммирования плоского массива `attempts` приведены в [канонической спецификации на английском языке](/reference/management-api/#api-key-usage-attribution). Итог родительского комбо не добавляется повторно; отсутствующие данные использования и старые записи без меток не восстанавливаются предположениями. Квоты подписки измеряются отдельно. + Строки в `models`, `providers` и `days[].models` также содержат `cacheHitRate` — долю входных токенов, полученных из кэша промптов провайдера и ограниченную диапазоном `[0, 1]`. Значение равно `null`, а не `0`, если провайдер не передал телеметрию кэша или в строке нет входных токенов: отсутствие diff --git a/docs-site/src/content/docs/zh-cn/reference/management-api.md b/docs-site/src/content/docs/zh-cn/reference/management-api.md index 480accdf331..f7c57d41669 100644 --- a/docs-site/src/content/docs/zh-cn/reference/management-api.md +++ b/docs-site/src/content/docs/zh-cn/reference/management-api.md @@ -140,6 +140,8 @@ Authorization: Bearer 如果某行超过现有解析器的大小限制,`GET /api/usage` 和 `GET /api/keys` 会保留可读取行的汇总,并在响应级别添加 `usageIncomplete: true` 和 `usageIncompleteReason: "oversized_rows"`。缓存和增量追加会保留该诊断,即使结果为空或没有筛选匹配;重建时会重新计算。不会缩短供应商、模型或 API 密钥标识来容纳该行。没有此标记不代表所有记录均有效。它与 `historyTruncated`、`entriesTruncated` 及 token 测量覆盖率相互独立。 +API 密钥的 `accountLogLabel` 生成方式、重试时的密钥选择与用量记录,以及扁平 `attempts` 的汇总规则,以[英文用量归属规范](/reference/management-api/#api-key-usage-attribution)为准。不要重复累加组合的父级总量,也不要推断未报告的用量或历史无标签记录。订阅配额是单独观测的数据。 + `models`、`providers` 和 `days[].models` 中的记录也带有 `cacheHitRate`:它表示由提供方提示缓存提供的输入 token 比例,并限制在 `[0, 1]` 范围内。当提供方未报告缓存遥测数据或该记录没有输入 token 时,其值为 `null`,绝不会是 `0`,因为“没有缓存数据”与“实际命中率为 0%”是不同的事实,将两者显示为相同结果的图表会产生误导。 :::caution diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 885d2abcd40..f5930b256d5 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -167,6 +167,7 @@ } }, "explicit": { + "key-attribution.test.ts": "usage", "responses-core-modules.test.ts": "responses", "chat-responses-control-integration.test.ts": "responses", "coding-agent-tool-result-images.test.ts": "adapters", @@ -548,6 +549,7 @@ "combos.test.ts": "codex-integration", "command-code-error-finish.test.ts": "providers", "command-code-provider.test.ts": "providers", + "command-code-retry.test.ts": "providers", "command-code-quota.test.ts": "providers", "command-code-workspace-cache.test.ts": "providers", "commandcode-provider.test.ts": "providers", diff --git a/src/adapters/command-code.ts b/src/adapters/command-code.ts index 4b7c707d7ef..22ca052a7e5 100644 --- a/src/adapters/command-code.ts +++ b/src/adapters/command-code.ts @@ -8,6 +8,7 @@ import type { AdapterFetchContext, AdapterRequest, ProviderAdapter } from "./bas import type { TranslatorBudget } from "../lib/translator-budget"; import { readBoundedResponseBody } from "../lib/bounded-body"; import { debugDroppedFrame } from "../lib/debug"; +import { SendBudgetExhaustedError } from "../lib/upstream-retry"; import { configuredReasoningEfforts } from "../reasoning-effort"; import { commandCodeReasoningEfforts, refreshCommandCodeReasoningEfforts } from "../providers/command-code-efforts"; import { identifyRoutedModel } from "./identity"; @@ -469,7 +470,7 @@ async function fetchCommandCode(request: AdapterRequest, ctx: AdapterFetchContex const timer = setTimeout(() => timeout.abort(new DOMException("Timeout elapsed", "TimeoutError")), ctx?.timeoutMs ?? 200_000); const callerSignal = ctx?.abortSignal ?? new AbortController().signal; try { - return await executor(request.url, { + return await (ctx?.executor ?? executor)(request.url, { method: request.method, headers: request.headers, body: request.body, @@ -556,6 +557,9 @@ export function createCommandCodeAdapter(provider: OcxProviderConfig): ProviderA }; }, async fetchResponse(request: AdapterRequest, ctx?: AdapterFetchContext): Promise { + // The outer caller records the entry send but does not reserve adapter-owned dispatches. + const initial = ctx?.sendBudget?.reserveDispatch({ sendClass: "initial", targetKey: request.url }); + if (initial && (!initial.allowed || !initial.permit.use())) throw new SendBudgetExhaustedError(request.url); const response = await fetchCommandCode(request, ctx, executor); if (response.ok) return response; const currentEffort = (() => { @@ -577,7 +581,11 @@ export function createCommandCodeAdapter(provider: OcxProviderConfig): ProviderA if (!refreshed || refreshed.includes(currentEffort)) return response; const retry = requestWithoutReasoningEffort(request); if (!retry) return response; + const decision = ctx?.sendBudget?.reserveDispatch({ sendClass: "repair", targetKey: retry.url }); + if (decision && (!decision.allowed || !decision.permit.use())) return response; try { void response.body?.cancel(); } catch { /* already closed */ } + // The caller owns the entry send; this adapter owns only its additional retry. + ctx?.onPhysicalSend?.({ ordinal: 2, recovery: "reasoning-effort-downgrade" }); return fetchCommandCode(retry, ctx, executor); }, async *parseStream(response: Response, budget: TranslatorBudget): AsyncGenerator { diff --git a/src/codex/account-label.ts b/src/codex/account-label.ts index b0a4ab76029..046462670bb 100644 --- a/src/codex/account-label.ts +++ b/src/codex/account-label.ts @@ -1,17 +1,19 @@ import { createHash, randomBytes } from "node:crypto"; import type { CodexAccount, OcxConfig } from "../types"; import type { CodexAuthContext } from "./auth-context"; +import type { ProviderApiKeySelection } from "../types/provider"; import { MAIN_CODEX_ACCOUNT_ID } from "./main-account"; export const CODEX_ACCOUNT_LOG_LABEL_RE = /^p[a-f0-9]{6}$/; /** - * Account log labels come in two families (#2699): + * Account log labels come in three families: * * - `p` (plus the literal `main`) — a Codex pool account. * - `o` — a non-Codex OAuth provider account (xai, cursor, and siblings). + * - `k` — a request-owned API-key selection, scoped to provider and reference. * - * Both are sha256-derived digests, never an email and never a raw provider account id. That is + * Labels never contain an email, raw key/reference, or raw provider account id. That is * a privacy requirement, not a formatting preference: these labels are written to the usage log * and served over the management API. * @@ -20,7 +22,16 @@ export const CODEX_ACCOUNT_LOG_LABEL_RE = /^p[a-f0-9]{6}$/; * accepted cost of keeping the existing `p` format byte-compatible. */ export const OAUTH_ACCOUNT_LOG_LABEL_RE = /^o[a-f0-9]{6}$/; -export const ACCOUNT_LOG_LABEL_RE = /^(?:main|[po][a-f0-9]{6})$/; +export const KEY_ACCOUNT_LOG_LABEL_RE = /^k[a-f0-9]{32}$/; +export const ACCOUNT_LOG_LABEL_RE = /^(?:main|[po][a-f0-9]{6}|k[a-f0-9]{32})$/; + +/** Digest the request-owned configured selection, never serialize its key/reference. */ +export function apiKeyAccountLogLabel(provider: string, selection: ProviderApiKeySelection | undefined): `k${string}` | undefined { + if (!selection || typeof selection.reference !== "string" || !selection.reference.length) return undefined; + return `k${createHash("sha256").update(JSON.stringify([ + "ocx-key-account-v1", provider, selection.entryId ?? null, selection.reference, + ])).digest("hex").slice(0, 32)}`; +} export function oauthAccountLogLabel(accountId: string, provider = ""): string { return `o${createHash("sha256").update(`${provider}\0${accountId}`).digest("hex").slice(0, 6)}`; diff --git a/src/providers/label.ts b/src/providers/label.ts index 099bf04870d..38b472a1a05 100644 --- a/src/providers/label.ts +++ b/src/providers/label.ts @@ -1,10 +1,28 @@ -import { CODEX_ACCOUNT_LOG_LABEL_RE, oauthAccountLogLabel } from "../codex/account-label"; +import { CODEX_ACCOUNT_LOG_LABEL_RE, KEY_ACCOUNT_LOG_LABEL_RE, apiKeyAccountLogLabel, oauthAccountLogLabel } from "../codex/account-label"; import type { OcxProviderConfig } from "../types"; export function canonicalUsageProviderLabel(provider: string): string { return provider === "chatgpt" || provider === "openai-multi" ? "openai" : provider; } +export function usesApiKeyAccount(provider: Pick): boolean { + return provider.authMode === "key" + || (provider.authMode === undefined && !!provider._apiKeyAttempt?.reference); +} + +/** Key identity comes from the captured selection, before env/keychain resolution. */ +export function stampApiKeyAccountLabel( + logCtx: { accountLogLabel?: string }, + providerName: string, + provider: Pick, +): void { + if (usesApiKeyAccount(provider)) { + logCtx.accountLogLabel = apiKeyAccountLogLabel(providerName, provider._apiKeyAttempt); + } else if (KEY_ACCOUNT_LOG_LABEL_RE.test(logCtx.accountLogLabel ?? "")) { + delete logCtx.accountLogLabel; + } +} + export function baseProviderLabel(provider: string): string { const canonical = canonicalUsageProviderLabel(provider); if (canonical !== provider) return canonical; diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index b122467103c..791c687bd01 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -51,7 +51,9 @@ import { linkAbortSignal } from "./responses"; import { addFinalRequestLog, beginRequestAttempt, - noteAttemptSend, + noteProviderAttemptSend, + recordKeyAttemptFailure, + recordKeyWireAttemptUsage, recordFirstOutput, recordAttemptCredentialSource, sealRequestAttemptIdentity, @@ -344,10 +346,12 @@ export async function handleNativeChatCompletions(options: HandleNativeChatOptio const encoding = new Headers(init.headers).get("accept-encoding"); if (!headers.has("accept-encoding") && encoding) headers.set("accept-encoding", encoding); if (init.signal?.aborted) throw init.signal.reason; - noteAttemptSend(attempt, logCtx.usageLogInputTokens, transportRecovery ?? recovery); - return ((activeProvider as OcxProviderTransport).fetch ?? execute)(request.url, applyUpstreamRecoveryInit({ + noteProviderAttemptSend(logCtx, route.providerName, activeProvider, logCtx.usageLogInputTokens, transportRecovery ?? recovery); + const dispatched = await ((activeProvider as OcxProviderTransport).fetch ?? execute)(request.url, applyUpstreamRecoveryInit({ ...init, method: request.method, headers, body: request.body, }, transportRecovery)); + if (!dispatched.ok) await recordKeyAttemptFailure(logCtx, dispatched, init.signal ?? upstream.signal); + return dispatched; }, }), ); @@ -509,8 +513,10 @@ export async function handleNativeChatCompletions(options: HandleNativeChatOptio stallTimeoutSec: config.stallTimeoutSec, onFirstOutput: logIds ? () => recordFirstOutput(logCtx, logIds.start) : undefined, onUsage: usage => { - logCtx.usage = usage; - attempt.usage = usage; + if (!recordKeyWireAttemptUsage(logCtx, usage)) { + logCtx.usage = usage; + attempt.usage = usage; + } }, onTerminal: (status: number, message?: string) => { terminalStatus = status; @@ -600,8 +606,10 @@ export async function handleNativeChatCompletions(options: HandleNativeChatOptio if (!completion) return fail(502, "upstream response contained no choices", "upstream_error"); const usage = usageFromChat(completion.usage); if (usage) { - logCtx.usage = usage; - attempt.usage = usage; + if (!recordKeyWireAttemptUsage(logCtx, usage)) { + logCtx.usage = usage; + attempt.usage = usage; + } } if (logIds) recordFirstOutput(logCtx, logIds.start); try { diff --git a/src/server/request-log.ts b/src/server/request-log.ts index b7f486053d4..95c77d4f923 100644 --- a/src/server/request-log.ts +++ b/src/server/request-log.ts @@ -1,5 +1,8 @@ import { existsSync, readFileSync } from "node:fs"; import { randomBytes } from "node:crypto"; +import { stampApiKeyAccountLabel, usesApiKeyAccount } from "../providers/label"; +import { KEY_ACCOUNT_LOG_LABEL_RE } from "../codex/account-label"; +import { readBoundedResponseBody } from "../lib/bounded-body"; import type { ResponsesTerminalStatus } from "../bridge"; import { classifyError, @@ -772,8 +775,10 @@ export function applyResponseLogMetadata(logCtx: RequestLogContext, payload: unk } const usage = usageFromResponsesPayload((source as { usage?: unknown }).usage); if (usage && !logCtx.usageFromBridge) { - logCtx.usage = usage; - if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; + if (!recordKeyWireAttemptUsage(logCtx, usage)) { + logCtx.usage = usage; + if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; + } // Counts taken off a wire, not reported raw. The zero-default token-detail objects strict // clients require are indistinguishable here from a measured zero, so the cache detail these // counts carry is recorded as synthesized rather than as an observed miss. @@ -1206,6 +1211,31 @@ export function recordNoAccountAffinityFailure( logCtx.errorCode ??= "codex_no_account"; return resolved; } +// Attempt identity can change in place while a combo parent retains an older context copy. +// These objects own their usage even after a rotation to an unknown key identity. +const keyUsageOwners = new WeakSet(); +const keyWireUsageBaselines = new WeakMap(); + +function cloneKeyUsage(usage: OcxUsage | undefined): OcxUsage | undefined { + return usage ? { ...usage } : undefined; +} + +/** Replace this physical send's wire snapshot against the pre-send baseline; repeats do not sum. */ +export function recordKeyWireAttemptUsage(logCtx: RequestLogContext, usage: OcxUsage | undefined): boolean { + if (!usage) return false; + const attempt = logCtx.activeAttempt; + if (!attempt || !keyUsageOwners.has(attempt) || !keyWireUsageBaselines.has(attempt)) return false; + const baseline = keyWireUsageBaselines.get(attempt); + const current = { ...usage }; + attempt.usage = baseline + ? aggregateAttemptUsage([ + { ...attempt, usage: baseline, usageStatus: baseline.estimated ? "estimated" : "reported" }, + { ...attempt, usage: current, usageStatus: current.estimated ? "estimated" : "reported" }, + ]).usage + : current; + logCtx.usage = attempt.usage; + return true; +} export function addFinalRequestLog( requestId: string, @@ -1237,7 +1267,9 @@ export function addFinalRequestLog( logCtx.activeAttempt, effectiveStatus, Date.now() - (logCtx.activeAttemptStartedAt ?? start), - logCtx.usage, + keyUsageOwners.has(logCtx.activeAttempt) + ? logCtx.activeAttempt.usage + : logCtx.usage, ); // The final row and its active physical attempt describe the same terminal. Preserve the // semantic code on both so detailed attempt telemetry cannot regress to a generic status code. @@ -1528,6 +1560,84 @@ export function sealRequestAttemptIdentity( attempt.provider = provider; attempt.adapter = adapter; if (isCodexUsageAccountLogLabel(accountLogLabel)) attempt.accountLogLabel = accountLogLabel; + else delete attempt.accountLogLabel; +} + +/** Preserve metered JSON failures before key recovery consumes/cancels their body. */ +export async function recordKeyAttemptFailure(logCtx: RequestLogContext, response: Response, signal?: AbortSignal): Promise { + const attempt = logCtx.activeAttempt; + if (!attempt || !KEY_ACCOUNT_LOG_LABEL_RE.test(attempt.accountLogLabel ?? "")) return; + attempt.status = response.status; + const cancelOriginal = (): void => { try { void response.body?.cancel().catch(() => {}); } catch { /* closed */ } }; + signal?.addEventListener("abort", cancelOriginal, { once: true }); + try { + if (signal?.aborted) { cancelOriginal(); return; } + const body = await readBoundedResponseBody(response.clone(), { signal, totalTimeoutMs: 1000, inactivityTimeoutMs: 1000 }); + if (body.truncated || body.oversized) return; + const value = JSON.parse(body.text); + const usage = usageFromResponsesPayload(value?.usage ?? value?.response?.usage); + if (usage) recordKeyWireAttemptUsage(logCtx, usage); + } catch { /* Absent/malformed usage remains unknown; recovery still owns the response. */ } + finally { signal?.removeEventListener("abort", cancelOriginal); } +} + +/** Add raw per-response usage before a bridge combines multiple rounds for the client. */ +export function recordKeyAttemptUsage(logCtx: RequestLogContext, usage: OcxUsage | undefined): void { + const attempt = logCtx.activeAttempt; + if (!attempt || !usage) return; + attempt.usage = attempt.usage + ? aggregateAttemptUsage([{ ...attempt, usageStatus: attempt.usage.estimated ? "estimated" : "reported" }, + { ...attempt, usage, usageStatus: usage.estimated ? "estimated" : "reported" }]).usage + : { ...usage }; + logCtx.usage = attempt.usage; +} + +/** A stable active object lets combo/stream callbacks keep pointing at the final attempt. + * Earlier key segments are immutable, flat snapshots inserted before that active object. */ +export function noteProviderAttemptSend( + logCtx: RequestLogContext, + providerName: string, + provider: OcxProviderConfig, + inputTokenEstimate: number | undefined, + recovery?: AttemptRecoveryKind, +): void { + const attempt = logCtx.activeAttempt; + const previous = attempt?.accountLogLabel; + stampApiKeyAccountLabel(logCtx, providerName, provider); + const next = logCtx.accountLogLabel; + if (attempt && usesApiKeyAccount(provider)) keyUsageOwners.add(attempt); + if (attempt && attempt.sendCount > 0 && previous !== next + && (KEY_ACCOUNT_LOG_LABEL_RE.test(previous ?? "") || KEY_ACCOUNT_LOG_LABEL_RE.test(next ?? ""))) { + // An input estimate is not evidence that a failed send used that many tokens. + delete attempt.inputTokenEstimate; + finishRequestAttempt(attempt, attempt.status >= 100 ? attempt.status + : recovery === "key-401" ? 401 : recovery?.includes("429") ? 429 : 502, + Date.now() - (logCtx.activeAttemptStartedAt ?? Date.now()), attempt.usage); + const completed = { ...attempt, recoveryKinds: [...attempt.recoveryKinds], + ...(attempt.usage ? { usage: { ...attempt.usage } } : {}), + ...(attempt.tierOutcome ? { tierOutcome: { ...attempt.tierOutcome } } : {}) }; + const attempts = logCtx.attempts ??= [attempt]; + const index = attempts.indexOf(attempt); + if (index >= 0) attempts.splice(index, 0, completed); + else attempts.push(completed, attempt); + const fresh = beginRequestAttempt(completed.ordinal + 1, providerName, completed.model, completed.adapter); + // Effort/tier metadata describes the request and is captured before the physical send. + for (const key of ["requestedEffort", "effectiveEffort", "reasoningWireField", "reasoningWireValue", "tierOutcome"] as const) { + if (completed[key] !== undefined) Object.assign(fresh, { [key]: completed[key] }); + } + for (const key of Object.keys(attempt)) delete (attempt as unknown as Record)[key]; + Object.assign(attempt, fresh); + delete logCtx.usage; + logCtx.activeAttemptStartedAt = Date.now(); + } + if (attempt) { + sealRequestAttemptIdentity(attempt, logCtx.provider, attempt.adapter, next); + recordAttemptCredentialSource(attempt, providerName, provider, attempt.adapter); + } + noteAttemptSend(attempt, inputTokenEstimate, recovery); + if (attempt && keyUsageOwners.has(attempt)) { + keyWireUsageBaselines.set(attempt, cloneKeyUsage(attempt.usage)); + } } /** Capture only the resolved upstream route; inbound auth and today's config cannot label old usage. */ diff --git a/src/server/responses/adapter-continuation.ts b/src/server/responses/adapter-continuation.ts index a1db9398d50..f0cd291eab8 100644 --- a/src/server/responses/adapter-continuation.ts +++ b/src/server/responses/adapter-continuation.ts @@ -10,7 +10,6 @@ import type { AdapterRequest } from "../../adapters/base"; import { recordAdapterReasoning, recordAdapterTier, - noteAttemptSend, sealRequestAttemptIdentity, recordAttemptCredentialSource, } from "../request-log"; @@ -78,12 +77,13 @@ export function createAdapterContinuations( | "genericFailoverAccountId" | "genericFailovers" | "applyFailoverSnapshot" + | "noteRoutedAttemptSend" + | "noteAdapterPhysicalSend" >, sidecarState: Pick, sendBudgetState: Pick< ResponsesSendBudget, | "adapterSendBudget" - | "noteAdapterPhysicalSend" | "remainingTransientSendBudget" | "noteTransientSends" | "reserveCredentialHop" @@ -111,7 +111,6 @@ export function createAdapterContinuations( const { upstream, connectMs, rateLimitPolicy, stallTimeoutMs } = adapterExchange; const { adapterSendBudget, - noteAdapterPhysicalSend, remainingTransientSendBudget, noteTransientSends, reserveCredentialHop, @@ -182,15 +181,16 @@ export function createAdapterContinuations( const replayKind: AttemptRecoveryKind | undefined = recoveryKind; try { if (transportState.activeAdapter.fetchResponse) { - noteAttemptSend(logCtx.activeAttempt, continuationEstimate, replayKind); + transportState.noteRoutedAttemptSend(continuationEstimate, replayKind); await waitForProviderRequestSlot(route.providerName, route.provider, nextParsed.modelId, upstream.signal); return await transportState.activeAdapter.fetchResponse(builtContinuationRequest, { abortSignal: upstream.signal, timeoutMs: connectMs, sendBudget: adapterSendBudget, - onPhysicalSend: send => noteAdapterPhysicalSend(continuationEstimate, send), + onPhysicalSend: send => transportState.noteAdapterPhysicalSend(continuationEstimate, send), stream: nextParsed.stream, executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, dispatchOverride: oauthDispatch(builtContinuationRequest, nextParsed), providerName: route.providerName, modelId: nextParsed.modelId, @@ -205,7 +205,7 @@ export function createAdapterContinuations( : fetchWithResetRetry; return await fetchContinuationWithRetryPolicy( recovery => { - noteAttemptSend(logCtx.activeAttempt, continuationEstimate, recovery ?? replayKind); + transportState.noteRoutedAttemptSend(continuationEstimate, recovery ?? replayKind); return fetchWithHeaderTimeout( builtContinuationRequest.url, applyUpstreamRecoveryInit({ diff --git a/src/server/responses/adapter-delivery.ts b/src/server/responses/adapter-delivery.ts index a46d1faf8b3..3f6330b6c44 100644 --- a/src/server/responses/adapter-delivery.ts +++ b/src/server/responses/adapter-delivery.ts @@ -26,7 +26,7 @@ export async function deliverAdapterResponse( | "rememberKiroDeliveredFinalAnswer" | "responseStateOptions" >, - transportState: Pick, + transportState: Pick, sidecarState: Pick, responseEffects: Pick< ResponsesEffects, @@ -106,11 +106,7 @@ export async function deliverAdapterResponse( ...(logCtx.surface === "grok" ? { heartbeatStyle: "comment" as const } : {}), onUsage: usage => { // Raw adapter usage, pre wire-normalization (see the runTurn branch above). - logCtx.usageFromBridge = true; - if (usage) { - logCtx.usage = usage; - if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; - } + transportState.bindKeyUsageFromBridge(usage); }, onCompletedResponse: (response: Record, providerState?: OcxProviderContinuationState) => { commitReasoningReplayServingRoute(); @@ -184,11 +180,7 @@ export async function deliverAdapterResponse( ...(routedCompaction ? { compaction: true } : {}), onProviderState: state => { providerState = state; }, onUsage: usage => { - logCtx.usageFromBridge = true; - if (usage) { - logCtx.usage = usage; - if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; - } + transportState.bindKeyUsageFromBridge(usage); }, }); // See the streaming branch: compaction turns skip the continuation cache. diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index 57eadf6a28a..e6ee2ad0948 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -11,7 +11,6 @@ import { trackStreamLifetime } from "../lifecycle"; import { recordAdapterReasoning, recordAdapterTier, - noteAttemptSend, sealRequestAttemptIdentity, recordAttemptCredentialSource, } from "../request-log"; @@ -115,12 +114,13 @@ export async function prepareAdapterExchange( | "genericFailoverAccountId" | "genericFailovers" | "applyFailoverSnapshot" + | "noteRoutedAttemptSend" + | "noteAdapterPhysicalSend" >, responseEffects: Pick, sendBudgetState: Pick< ResponsesSendBudget, | "adapterSendBudget" - | "noteAdapterPhysicalSend" | "remainingTransientSendBudget" | "noteTransientSends" | "recoverySendAllowance" @@ -152,7 +152,6 @@ export async function prepareAdapterExchange( const { cancelResponseCompletion, notifyResponseComplete, refreshRequestToolAliases } = responseEffects; const { adapterSendBudget, - noteAdapterPhysicalSend, remainingTransientSendBudget, noteTransientSends, recoverySendAllowance, @@ -272,15 +271,16 @@ export async function prepareAdapterExchange( let upstreamResponse: Response; try { if (transportState.activeAdapter.fetchResponse) { - noteAttemptSend(logCtx.activeAttempt, inputTokenEstimate); + transportState.noteRoutedAttemptSend(inputTokenEstimate); await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal); upstreamResponse = await transportState.activeAdapter.fetchResponse(builtInitialRequest, { abortSignal: upstream.signal, timeoutMs: connectMs, sendBudget: adapterSendBudget, - onPhysicalSend: send => noteAdapterPhysicalSend(inputTokenEstimate, send), + onPhysicalSend: send => transportState.noteAdapterPhysicalSend(inputTokenEstimate, send), stream: parsed.stream, executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, dispatchOverride: oauthDispatch(builtInitialRequest), providerName: route.providerName, modelId: route.modelId, @@ -300,7 +300,7 @@ export async function prepareAdapterExchange( : fetchWithResetRetry; upstreamResponse = await fetchWithRetryPolicy( recovery => { - noteAttemptSend(logCtx.activeAttempt, inputTokenEstimate, recovery); + transportState.noteRoutedAttemptSend(inputTokenEstimate, recovery); return fetchWithHeaderTimeout(builtInitialRequest.url, applyUpstreamRecoveryInit({ method: builtInitialRequest.method, headers: builtInitialRequest.headers, @@ -412,10 +412,10 @@ export async function prepareAdapterExchange( logCtx.providerAdapter = transportState.activeAdapter.name; sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, transportState.activeAdapter.name, logCtx.accountLogLabel); recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, transportState.activeAdapter.name); - noteAttemptSend(logCtx.activeAttempt, retryEstimate, recovery); try { try { if (transportState.activeAdapter.fetchResponse) { + transportState.noteRoutedAttemptSend(retryEstimate, recovery); await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal); // The dispatch boundary is HERE, not before the pacing wait: that wait can reject for // an abort, a saturated queue, an expired slot or a removed provider, and none of @@ -426,9 +426,10 @@ export async function prepareAdapterExchange( abortSignal: upstream.signal, timeoutMs: connectMs, sendBudget: adapterSendBudget, - onPhysicalSend: send => noteAdapterPhysicalSend(retryEstimate, send), + onPhysicalSend: send => transportState.noteAdapterPhysicalSend(retryEstimate, send), stream: parsed.stream, executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, dispatchOverride: oauthDispatch(retryRequest), providerName: route.providerName, modelId: route.modelId, @@ -461,6 +462,7 @@ export async function prepareAdapterExchange( if (refetchAllowance?.permit && !refetchAllowance.permit.use()) { throw new SendBudgetExhaustedError(safeHostLabel(retryRequest.url)); } + transportState.noteRoutedAttemptSend(retryEstimate, recoveryKind ?? recovery); // Same boundary on the helper path: the thunk is what reaches the wire, and it // can be refused above before it does. use() past the first attempt is a no-op. onDispatch?.(); diff --git a/src/server/responses/collaboration.ts b/src/server/responses/collaboration.ts index ab6a9b55c9b..60b66619363 100644 --- a/src/server/responses/collaboration.ts +++ b/src/server/responses/collaboration.ts @@ -83,7 +83,6 @@ import { catalogModelSupportsServiceTier, finishRequestAttempt, inspectResponseLogJson, - noteAttemptSend, readConfiguredCodexServiceTier, requestLogSpeedLabel, sealRequestAttemptIdentity, diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index 6b5b979d37c..77082bfe00b 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -140,7 +140,6 @@ import { catalogModelSupportsServiceTier, finishRequestAttempt, inspectResponseLogJson, - noteAttemptSend, readConfiguredCodexServiceTier, requestLogSpeedLabel, sealRequestAttemptIdentity, diff --git a/src/server/responses/core-codex-account.ts b/src/server/responses/core-codex-account.ts index 5d0fc501099..f84f83db68e 100644 --- a/src/server/responses/core-codex-account.ts +++ b/src/server/responses/core-codex-account.ts @@ -66,7 +66,7 @@ import { recordAdapterTier, sealRequestAttemptIdentity, recordAttemptCredentialSource, - noteAttemptSend, + noteProviderAttemptSend, } from "../request-log"; import { codexAuthContextLogLabel } from "../../codex/account-label"; import { chargeWorkflowSends } from "../../lib/workflow-budget"; @@ -702,7 +702,7 @@ export async function retryCodexPoolOnAlternateAccount( // The move is a physical send like any other, so the root workflow is charged too. chargeWorkflowSends(args.options.workflowRootId, 1); } - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate); + noteProviderAttemptSend(logCtx, route.providerName, route.provider, passthroughEstimate); try { upstreamResponse = await fetchWithHeaderTimeout( request.url, diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 43863fc2945..32fa2189d5d 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -425,6 +425,9 @@ export async function executeComboResponses( config.providers[pick.target.provider]!.adapter, ); childLog.activeAttempt = attempt; + childLog.activeAttemptStartedAt = started; + childLog.attempts = logCtx.attempts ??= []; + childLog.attempts.push(attempt); let attemptRetained = false; const retainCancelledAttempt = (): void => { if (attemptRetained) return; @@ -435,7 +438,6 @@ export async function executeComboResponses( childLog.accountLogLabel, ); finishRequestAttempt(attempt, 499, Date.now() - started, childLog.usage); - (logCtx.attempts ??= []).push(attempt); attemptRetained = true; }; const completedTarget = { provider: pick.target.provider, model: pick.target.model }; @@ -505,6 +507,7 @@ export async function executeComboResponses( retainCancelledAttempt(); return clientCancelledResponse(); } + finishRequestAttempt(attempt, 502, Date.now() - started, childLog.usage); throw error; } @@ -526,6 +529,7 @@ export async function executeComboResponses( retainCancelledAttempt(); return clientCancelledResponse(); } + finishRequestAttempt(attempt, 502, Date.now() - started, childLog.usage); throw error; } if (preflight.kind === "failed") { @@ -546,7 +550,6 @@ export async function executeComboResponses( childLog.providerAdapter ?? attempt.adapter, childLog.accountLogLabel, ); - (logCtx.attempts ??= []).push(attempt); attemptRetained = true; noteComboSuccess(comboId, combo, pick.target, pick.writerGeneration); Object.assign(logCtx, childLog, { @@ -580,6 +583,7 @@ export async function executeComboResponses( retainCancelledAttempt(); return clientCancelledResponse(); } + finishRequestAttempt(attempt, 502, Date.now() - started, childLog.usage); throw error; } if (options.abortSignal?.aborted) { @@ -598,7 +602,6 @@ export async function executeComboResponses( Date.now() - started, failure.usage, ); - (logCtx.attempts ??= []).push(attempt); attemptRetained = true; lastFailure = failure.response; lastFailedChildLog = childLog; diff --git a/src/server/responses/encrypted-payload.ts b/src/server/responses/encrypted-payload.ts index 066bb9b5226..0e9efddf999 100644 --- a/src/server/responses/encrypted-payload.ts +++ b/src/server/responses/encrypted-payload.ts @@ -78,7 +78,6 @@ import { catalogModelSupportsServiceTier, finishRequestAttempt, inspectResponseLogJson, - noteAttemptSend, readConfiguredCodexServiceTier, requestLogSpeedLabel, sealRequestAttemptIdentity, diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index f5c5b94694a..0292f1d77af 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -60,7 +60,6 @@ import { restorePlaintextV2AgentMessageCalls } from "../../responses/plaintext-v import { recordAdapterReasoning, recordAdapterTier, - noteAttemptSend, sealRequestAttemptIdentity, recordAttemptCredentialSource, } from "../request-log"; @@ -171,6 +170,7 @@ export async function preparePassthroughExchange( | "replayOAuthCredentialSnapshot" | "genericFailovers" | "applyFailoverSnapshot" + | "noteRoutedAttemptSend" >, responseEffects: Pick< ResponsesEffects, @@ -752,7 +752,7 @@ export async function preparePassthroughExchange( // Body is a replayable string; nothing has streamed to the client yet. upstreamResponse = await fetchWithTransientRetry( recovery => { - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, recovery); + transportState.noteRoutedAttemptSend(passthroughEstimate, recovery); return fetchWithHeaderTimeout(request.url, applyUpstreamRecoveryInit({ method: request.method, headers: request.headers, @@ -848,7 +848,7 @@ export async function preparePassthroughExchange( if (allowance.permit && !allowance.permit.use()) { throw new SendBudgetExhaustedError(safeHostLabel(request.url)); } - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, innerRecovery ?? recovery); + transportState.noteRoutedAttemptSend(passthroughEstimate, innerRecovery ?? recovery); return fetchWithHeaderTimeout(request.url, applyUpstreamRecoveryInit({ method: request.method, headers: request.headers, @@ -946,7 +946,7 @@ export async function preparePassthroughExchange( // every other build site; a replay is exactly when a grown payload reappears. const replayBodyRefusal = refuseOversizedOutboundBody(request); if (replayBodyRefusal) return replayBodyRefusal; - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, "oauth-401"); + transportState.noteRoutedAttemptSend(passthroughEstimate, "oauth-401"); upstreamResponse = await fetchWithHeaderTimeout( request.url, { method: request.method, headers: request.headers, body: request.body }, @@ -1075,7 +1075,7 @@ export async function preparePassthroughExchange( try { upstreamResponse = await fetchWithTransientRetry( recovery => { - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, recovery ?? "oauth-401"); + transportState.noteRoutedAttemptSend(passthroughEstimate, recovery ?? "oauth-401"); return fetchWithHeaderTimeout(request.url, applyUpstreamRecoveryInit({ method: request.method, headers: request.headers, @@ -1192,7 +1192,7 @@ export async function preparePassthroughExchange( recovery => { // The first send of every replay is itself a rate-limit retry; inner transient-5xx // recoveries keep their own label (recovery is provided for those). - noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, recovery ?? "rate-limit-429"); + transportState.noteRoutedAttemptSend(passthroughEstimate, recovery ?? "rate-limit-429"); return fetchWithHeaderTimeout(request.url, applyUpstreamRecoveryInit({ method: request.method, headers: request.headers, diff --git a/src/server/responses/request-send-budget.ts b/src/server/responses/request-send-budget.ts index c5879e106fa..7897d64d009 100644 --- a/src/server/responses/request-send-budget.ts +++ b/src/server/responses/request-send-budget.ts @@ -52,7 +52,7 @@ export function createResponsesSendBudget( /** * Records an adapter's OWN inner retries against this attempt. * - * Ordinal 1 is the send each call site already recorded through `noteAttemptSend`, so only + * Ordinal 1 is the send each call site already recorded through `noteRoutedAttemptSend`, so only * the extra physical sends are added here and an adapter that does not retry internally * leaves its log byte-for-byte as it was. Kiro reaches roughly eighteen sends per call and * Cursor re-sends a whole turn, and both reported one; a count that cannot be observed diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index c80f2421210..f5e6173cd92 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -8,7 +8,7 @@ import { credentialGeneration, } from "../../oauth/store"; import type { ProviderAdapter, AdapterRequest } from "../../adapters/base"; -import type { OcxParsedRequest, OcxProviderConfig } from "../../types"; +import type { AdapterEvent, OcxParsedRequest, OcxProviderConfig, OcxUsage } from "../../types"; import type { AnthropicAccountSelectionReason } from "../../oauth/anthropic-routing"; import { isAnthropicAccountPoolEnabled, @@ -33,7 +33,7 @@ import { preferredInitialAccount, noteGenericPoolSelection, } from "../../oauth/generic-account-failover"; -import { stampOAuthAccountLabel } from "../../providers/label"; +import { stampOAuthAccountLabel, usesApiKeyAccount } from "../../providers/label"; import { resolveProviderTransport } from "../../providers/xai-transport"; import { resolveCopilotApiBaseUrl } from "../../oauth/github-copilot"; import { @@ -59,7 +59,11 @@ import { sealRequestAttemptIdentity, recordAttemptCredentialSource, recordAdapterTierMetadata, + noteProviderAttemptSend, + recordKeyAttemptFailure, + recordKeyAttemptUsage, } from "../request-log"; +import type { AttemptRecoveryKind } from "../../usage/log"; import { resolvePassiveRouteSubjectId } from "../passive-route-linker"; /** Owns live credential selection and adapter bindings for one request. */ @@ -241,6 +245,35 @@ export async function prepareResponsesTransport( replayOAuthCredentialSnapshot = { accountId: snapshot.accountId, generation: snapshot.generation }; return true; }; + // Key sends may be rebuilt while queued. Keep metadata pending until the guarded + // physical dispatch binds it to the selection that actually reaches the upstream. + // A process exit before dispatch discards this unsent metadata; it is not usage evidence. + let pendingKeySend: { estimate: number | undefined; recovery?: AttemptRecoveryKind } | undefined; + const noteRoutedAttemptSend = (estimate: number | undefined, recovery?: AttemptRecoveryKind): void => { + if (usesApiKeyAccount(route.provider)) pendingKeySend = { estimate, recovery }; + else noteProviderAttemptSend(logCtx, route.providerName, route.provider, estimate, recovery); + }; + /** Inner adapter retries use the same dispatch owner as the entry send. */ + const noteAdapterPhysicalSend = (estimate: number | undefined, send: { ordinal: number; recovery?: AttemptRecoveryKind }): void => { + if (send.ordinal > 1) noteRoutedAttemptSend(estimate, send.recovery); + }; + const commitKeyAttemptSend = (): void => { + if (!usesApiKeyAccount(route.provider)) return; + noteProviderAttemptSend(logCtx, route.providerName, route.provider, + pendingKeySend?.estimate ?? logCtx.usageLogInputTokens, pendingKeySend?.recovery); + pendingKeySend = undefined; + }; + const bindKeyUsageFromBridge = (usage: OcxUsage | undefined): void => { + logCtx.usageFromBridge = true; + if (usesApiKeyAccount(route.provider)) { + logCtx.usage = logCtx.activeAttempt?.usage; + return; + } + if (usage) { + logCtx.usage = usage; + if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; + } + }; const selectionIsCurrent = (binding: DispatchBinding | undefined): boolean => { if (route.provider.authMode === "forward") return true; if (!binding) return false; @@ -260,6 +293,27 @@ export async function prepareResponsesTransport( : undefined : { kind: "api-key", provider: { ...route.provider } }; if (binding) adapterBindings.set(resolved, binding); + // Observe terminals before search/image loops or continuation guards hide earlier rounds. + // Each adapter parser is called once per physical response; bridge totals are client-only. + const observedResponses = new WeakSet(); + const observeUsage = (event: AdapterEvent, response: object): void => { + if (usesApiKeyAccount(provider) && "usage" in event && event.usage && !observedResponses.has(response)) { + observedResponses.add(response); + recordKeyAttemptUsage(logCtx, event.usage); + } + }; + const parseStream = resolved.parseStream.bind(resolved); + resolved.parseStream = async function* (...args) { + for await (const event of parseStream(...args)) { observeUsage(event, args[0]); yield event; } + }; + if (resolved.parseResponse) { + const parseResponse = resolved.parseResponse.bind(resolved); + resolved.parseResponse = async (...args) => { + const events = await parseResponse(...args); + events.forEach(event => observeUsage(event, args[0])); + return events; + }; + } const build = resolved.buildRequest.bind(resolved); resolved.buildRequest = async (requestParsed, incoming) => { const request = await build(requestParsed, incoming); @@ -268,7 +322,11 @@ export async function prepareResponsesTransport( return request; }; if (resolved.runTurn) { - rawRunTurns.set(resolved, resolved.runTurn.bind(resolved)); + const runTurn = resolved.runTurn.bind(resolved); + rawRunTurns.set(resolved, (requestParsed, incoming, emit) => { + const response = {}; + return runTurn(requestParsed, incoming, event => { observeUsage(event, response); emit(event); }); + }); resolved.runTurn = (requestParsed, incoming, emit) => runSelectedTurn(resolved, requestParsed, incoming, emit); } return resolved; @@ -319,6 +377,7 @@ export async function prepareResponsesTransport( refused = true; throw new Error("Account selection changed before the first turn dispatch"); } + commitKeyAttemptSend(); sent = true; }, }); @@ -351,7 +410,9 @@ export async function prepareResponsesTransport( && sentHeaders?.get("authorization") === `Bearer ${snapshot.accessToken}` && !sentHeaders?.has("x-api-key"); // Reselection can choose a provider override instead of the supplied executor. + commitKeyAttemptSend(); const response = await fetchImpl(destination, { ...dispatchInit, redirect: "manual" }); + if (!response.ok) await recordKeyAttemptFailure(logCtx, response, dispatchInit.signal ?? options.abortSignal); // Observe each physical response before retries replace it. The binding belongs to // this dispatch, so a manual switch cannot file A's headers against B. Header // overrides and credential replacement make ownership unprovable: skip those writes. @@ -736,6 +797,10 @@ export async function prepareResponsesTransport( resolveSelectionAdapter, refreshRunTurnAdapter, oauthDispatch, + noteRoutedAttemptSend, + noteAdapterPhysicalSend, + commitKeyAttemptSend, + bindKeyUsageFromBridge, anthropicSessionKey, isPassthrough, }; diff --git a/src/server/responses/run-turn-execution.ts b/src/server/responses/run-turn-execution.ts index 24e96802fb9..75234f3ffdb 100644 --- a/src/server/responses/run-turn-execution.ts +++ b/src/server/responses/run-turn-execution.ts @@ -12,7 +12,7 @@ import { adapterNeedsForcedContinuation, adapterResponseReachedServingTerminal, } from "./core-replay"; -import { sealRequestAttemptIdentity, noteAttemptSend, recordAttemptCredentialSource } from "../request-log"; +import { sealRequestAttemptIdentity, recordAttemptCredentialSource } from "../request-log"; import { waitForProviderRequestSlot, RequestPacingQueueOverloadError } from "../../providers/request-pacing"; import type { AdapterEventQueue } from "../../adapters/run-turn-queue"; import type { AttemptRecoveryKind } from "../../usage/log"; @@ -65,6 +65,8 @@ export async function executeResponsesRunTurn( | "applyFailoverSnapshot" | "resolveSelectionAdapter" | "adapter" + | "noteRoutedAttemptSend" + | "bindKeyUsageFromBridge" >, sidecarState: Pick, responseEffects: Pick< @@ -140,7 +142,7 @@ export async function executeResponsesRunTurn( await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); } await refreshRunTurnSelection(); - noteAttemptSend(logCtx.activeAttempt, logCtx.usageLogInputTokens, recovery); + transportState.noteRoutedAttemptSend(logCtx.usageLogInputTokens, recovery); const runTurnProviderFetch = providerFetch( route.provider, options.codexWsRuntimeIdentity, @@ -349,11 +351,7 @@ export async function executeResponsesRunTurn( onUsage: usage => { // Raw adapter usage, pre wire-normalization: the bridged SSE now always carries // zero-default detail objects, so provenance must come from here (cache_detail_missing). - logCtx.usageFromBridge = true; - if (usage) { - logCtx.usage = usage; - if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; - } + transportState.bindKeyUsageFromBridge(usage); }, onCompletedResponse: (response: Record, providerState?: OcxProviderContinuationState) => { commitReasoningReplayServingRoute(); @@ -420,11 +418,7 @@ export async function executeResponsesRunTurn( ...(routedCompaction ? { compaction: true } : {}), onProviderState: state => { providerState = state; }, onUsage: usage => { - logCtx.usageFromBridge = true; - if (usage) { - logCtx.usage = usage; - if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; - } + transportState.bindKeyUsageFromBridge(usage); }, }); if (!routedCompaction) { diff --git a/src/server/responses/sidecar-execution.ts b/src/server/responses/sidecar-execution.ts index 7987d5ca932..7aeb9d452da 100644 --- a/src/server/responses/sidecar-execution.ts +++ b/src/server/responses/sidecar-execution.ts @@ -35,7 +35,7 @@ import { bindRouteReasoningReplayScope, adapterNeedsForcedContinuation } from ". import { namespacedToolName } from "../../types"; import { providerFetch } from "./fetch-helpers"; import type { AttemptRecoveryKind } from "../../usage/log"; -import { noteAttemptSend, recordAdapterReasoning, recordAdapterTier } from "../request-log"; +import { recordAdapterReasoning, recordAdapterTier } from "../request-log"; import { normalizeLogConversationId } from "../request-log-conversation"; import { rememberResponseState } from "../../responses/state"; import { trackStreamLifetime } from "../lifecycle"; @@ -65,6 +65,8 @@ export async function executeResponsesSidecars( | "commitResolvedOAuthSelection" | "resolveSelectionAdapter" | "oauthDispatch" + | "noteRoutedAttemptSend" + | "bindKeyUsageFromBridge" >, sidecarState: Pick, responseEffects: Pick< @@ -322,7 +324,7 @@ export async function executeResponsesSidecars( ...(vidPlan ? { videoPlan: vidPlan } : {}), forwardHeaders: requestState.selectedForwardHeaders, onAttemptSend: (recovery?: AttemptRecoveryKind) => - noteAttemptSend(logCtx.activeAttempt, logCtx.usageLogInputTokens, recovery), + transportState.noteRoutedAttemptSend(logCtx.usageLogInputTokens, recovery), abortSignal: options.abortSignal, maxRounds: imgPlan && vidPlan ? clampImageMaxRounds(Math.min(config.images?.maxRounds ?? 3, config.images?.videoMaxRounds ?? 2)) @@ -352,11 +354,7 @@ export async function executeResponsesSidecars( if (!logCtx.conversationId && parsed._cursorConversationId) { logCtx.conversationId = normalizeLogConversationId(parsed._cursorConversationId); } - logCtx.usageFromBridge = true; - if (usage) { - logCtx.usage = usage; - if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; - } + transportState.bindKeyUsageFromBridge(usage); }, on429: rotateSidecarProviderOn429, retryOn429Policy: rateLimitRetryPolicyFor(route.provider), @@ -432,13 +430,9 @@ export async function executeResponsesSidecars( recordAdapterTier(logCtx, request); }, onAttemptSend: (recovery?: AttemptRecoveryKind) => - noteAttemptSend(logCtx.activeAttempt, logCtx.usageLogInputTokens, recovery), + transportState.noteRoutedAttemptSend(logCtx.usageLogInputTokens, recovery), onUsage: usage => { - logCtx.usageFromBridge = true; - if (usage) { - logCtx.usage = usage; - if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage; - } + transportState.bindKeyUsageFromBridge(usage); }, recordSidecarOutcome: wsPlan.forwardSidecar?.recordOutcome, connectTimeoutMs: config.connectTimeoutMs ?? 200_000, diff --git a/src/usage/log.ts b/src/usage/log.ts index de03cbf752f..aadeb1648bb 100644 --- a/src/usage/log.ts +++ b/src/usage/log.ts @@ -38,7 +38,7 @@ export type UsageStatus = "reported" | "unreported" | "unsupported" | "estimated * The old name `CodexUsageAccountLogLabel` is kept as an alias because it is exported and used * across modules; the two predicates below are what callers should choose between. */ -export type UsageAccountLogLabel = "main" | `p${string}` | `o${string}`; +export type UsageAccountLogLabel = "main" | `p${string}` | `o${string}` | `k${string}`; export type CodexUsageAccountLogLabel = UsageAccountLogLabel; /** diff --git a/structure/adapters/registry.md b/structure/adapters/registry.md index 83e8f4f4668..1185d1cdc2d 100644 --- a/structure/adapters/registry.md +++ b/structure/adapters/registry.md @@ -99,7 +99,7 @@ so the schema is not something a user can fix from configuration (issue #2673). > Decision record: [ADR-0093](../decisions/ADR-0093-moonshot-ref-with-siblings-normalization.md) -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. @@ -183,3 +183,6 @@ implement legacy call/result pairing. Modern tool-image carriers are unchanged. raw passthrough; `tests/responses/chat-media-translation.test.ts` reaches the real HTTP translation boundary and verifies that rejection sends no upstream request. Canonical Responses identity sanitation and narrowly scoped pre-output combo recovery follow [request-local target compatibility](../runtime.md#request-local-target-compatibility); other adapter contracts remain unchanged. + +Command Code sends and its reasoning-effort downgrade retry share the request send budget; +its retry observer follows the [physical-attempt attribution contract](../gui-and-management-api.md#upstream-key-account-attribution). diff --git a/structure/catalog.md b/structure/catalog.md index 2fd03722df0..0ee535fd3ca 100644 --- a/structure/catalog.md +++ b/structure/catalog.md @@ -340,7 +340,7 @@ spelling; the V1 and compaction cap exemptions are preserved. Codex display-cache expiry, retained main-policy evidence, and reset history follow the [quota cache contract](providers/openai-tiers.md#quota-cache-and-short-window-history). -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/clients/claude-desktop.md b/structure/clients/claude-desktop.md index 01a583c1826..11e62f4e17a 100644 --- a/structure/clients/claude-desktop.md +++ b/structure/clients/claude-desktop.md @@ -101,7 +101,7 @@ testable on any host: stubbing `process.platform` does not propagate to `os.plat > Decision record: [ADR-0046](../decisions/ADR-0046-claude-desktop-config-library-resolution.md) -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/codex-home.md b/structure/codex-home.md index 339358ea9be..bdacb4ed9cf 100644 --- a/structure/codex-home.md +++ b/structure/codex-home.md @@ -275,3 +275,5 @@ Pool quota producers and account commands follow the [bounded raw-observation co The account history response can include a [low-confidence effective capacity estimate](providers/openai-tiers.md#observed-effective-token-capacity); usage normalization retains local-answer provenance so local responses cannot supply samples. Codex pool settings and their consumers follow the [reset-first ordering contract](providers/openai-tiers.md#reset-first-account-ordering), including independent-quota fallback and preserved affinity. + +Upstream API-key usage follows the [physical-attempt account attribution contract](gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. diff --git a/structure/config.md b/structure/config.md index d00b36b06bb..41515893d77 100644 --- a/structure/config.md +++ b/structure/config.md @@ -269,7 +269,7 @@ The unregistered executor CLI module stores Remote Workspace state separately fr Remote Workspace uses a separate, explicitly enabled server surface with structural WebSocket callbacks and awaited per-server cleanup; [its contract](remote-workspace.md) owns that integration. -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. `dropCodexSafetyBuffering` is an optional boolean, default false. Invalid API candidates reject; malformed persisted values stay disabled. It controls only the allowlisted client-output hints diff --git a/structure/data-planes/images.md b/structure/data-planes/images.md index a7420072bfd..a73fcc6f07d 100644 --- a/structure/data-planes/images.md +++ b/structure/data-planes/images.md @@ -84,7 +84,7 @@ conflicts with `modelSupportsReasoningSummaries: false` for the same model. > Decision record: [ADR-0045](../decisions/ADR-0045-standalone-images.md) -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/data-planes/inbound-compat.md b/structure/data-planes/inbound-compat.md index 33ca8c76b0f..0e1a030e945 100644 --- a/structure/data-planes/inbound-compat.md +++ b/structure/data-planes/inbound-compat.md @@ -183,7 +183,7 @@ reasoning ladder into `thinking.effortOptions`. Missing capabilities stay absent falling back to OpenCodex guesses, and the integration does not write the removed `thinking.effort` / `defaultEffort` fields because MCode owns the active effort per session. -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 4628ca5e501..3b628a66f4d 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -367,6 +367,26 @@ keeps the saved state and renders fixed `ocx sync` guidance without server/accou ## Usage accounting +### Upstream key account attribution + +API-key attempts in `src/usage/log.ts` carry `accountLogLabel` as `k` plus 32 lowercase +hex digits. `src/codex/account-label.ts` derives it from the first 128 bits of SHA-256 over +`JSON.stringify(["ocx-key-account-v1", providerName, entryId ?? null, reference])`. +`reference` is the configured value captured for the physical send, before environment or +keychain resolution. The log contains the digest, not raw keys, references, or pool IDs. +Existing Codex and OAuth label formats remain valid. Replacing a literal or reference changes +identity; rotating the secret behind the same reference preserves the logical account. + +`src/providers/label.ts` stamps only key authentication, including implicit custom-provider +keys. `src/server/request-log.ts` commits identity at dispatch after queued selection changes, +retains separate flat records when retries change keys, and isolates each record's raw usage +from parent combo totals and adapter-loop aggregation. Reported failure usage is retained; +missing usage and historical identities remain unknown. Native wire snapshots replace only the +current physical response contribution, preserving prior sends on the same key without counting +repeated inspections twice. Consumers sum the flat attempts once and keep subscription quota +observations separate from token or API-equivalent cost totals. + + `src/server/hub-usage.ts` serves `GET /v1/usage` on hubs for an explicit configured data key. The authenticated key selects the aggregate; query parameters cannot select an API-key identity. Unscoped environment/admin credentials and loopback bypass are not admitted. The response projects only this client's numeric totals, provider/model/day rows and incomplete-history metadata through `src/remote/hub-usage.ts`; accounts, raw records and key IDs are omitted. Unknown fields are stripped at every object boundary and the serialized body is capped at 1 MiB. Custom usage windows are immutable bounds on the streaming accumulator, applied to each @@ -556,6 +576,12 @@ The shared Responses path follows the [bounded multipart recovery contract](suba Data keys authorize only the data matrix and authenticated catalog. Admin credentials authorize ordinary management and key rotation but cannot mint, exchange, or refresh a `gui-session`. Pairing grants are digest-only, origin-bound, one-use, capped at 128 live grants, burned after five grant failures, and source-limited after ten failures in ten minutes with at most 1,024 source buckets. `POST /api/session/logout` invalidates only the current origin/CSRF-authorized browser session. +The hash is a deterministic pseudonym, not encryption; a consumer holding candidate local +configuration can recompute it. Command Code reserves the initial request and reasoning-effort +retry against the shared send budget. Refusing a retry preserves the original error body. +Its inner-retry observer goes through the same routed dispatch owner as the entry send, so +key requests record exactly one additional send with `reasoning-effort-downgrade`. + ### Model picker ordering settings `GET /api/subagent-models` retains `chosen`, `available`, and `catalogState`, and adds routed-only diff --git a/structure/ops/docs-and-release.md b/structure/ops/docs-and-release.md index 0c0ffe38d08..9a2f6f27d3d 100644 --- a/structure/ops/docs-and-release.md +++ b/structure/ops/docs-and-release.md @@ -336,7 +336,7 @@ The shared atomic replacement publisher also identifies explicit Remote Workspac Remote Workspace uses a separate, explicitly enabled server surface with structural WebSocket callbacks and awaited per-server cleanup; [its contract](../remote-workspace.md) owns that integration. -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Listener startup diagnostics follow [the runtime lifecycle contract](../runtime.md#lifecycle); malformed optional listener blocks follow [config loading](../config.md#config-surface). The Combo guides describe the distinction between display quota and single-credential inference evidence used by routing. See [scoped provider quota](../runtime.md#scoped-provider-quota-for-combo-selection). diff --git a/structure/ops/service-and-sidecars.md b/structure/ops/service-and-sidecars.md index e28575b432f..db1a8653a14 100644 --- a/structure/ops/service-and-sidecars.md +++ b/structure/ops/service-and-sidecars.md @@ -145,7 +145,7 @@ so the flag does not identify the peer responsible for corruption. Existing diag not rewritten. Audio devices, WebRTC media negotiation, captions and spoken handoff delivery remain client responsibilities. -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/providers/chat-compat.md b/structure/providers/chat-compat.md index 5ee17ed8070..08cdc582886 100644 --- a/structure/providers/chat-compat.md +++ b/structure/providers/chat-compat.md @@ -339,3 +339,5 @@ prose the model reads beside them. Vendor tool execution stays disabled on both adapters, and Qoder's explicit refusal of original images is unchanged. Canonical Responses identity sanitation and narrowly scoped pre-output combo recovery follow [request-local target compatibility](../runtime.md#request-local-target-compatibility); other adapter contracts remain unchanged. + +Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. diff --git a/structure/providers/cursor.md b/structure/providers/cursor.md index 5ae38028d88..39f06c39151 100644 --- a/structure/providers/cursor.md +++ b/structure/providers/cursor.md @@ -130,3 +130,5 @@ Translated Chat request construction uses the [inline-image budget](../transport Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. Canonical Responses identity sanitation and narrowly scoped pre-output combo recovery follow [request-local target compatibility](../runtime.md#request-local-target-compatibility); other adapter contracts remain unchanged. + +Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. diff --git a/structure/providers/openai-tiers.md b/structure/providers/openai-tiers.md index ab7511c3b08..fd25e83c460 100644 --- a/structure/providers/openai-tiers.md +++ b/structure/providers/openai-tiers.md @@ -567,3 +567,5 @@ Two call sites need the rule — the live path in `reevaluateAffinityQuota` and `previewReusableAffinityAccount` that subagent fallback reads — and they share one helper rather than restating it, because the suite asserts the two answer identically and a preview that disagreed would hand fallback a different account than the request actually uses. + +Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. diff --git a/structure/providers/xai-grok.md b/structure/providers/xai-grok.md index dc982c5639d..114dbd9f806 100644 --- a/structure/providers/xai-grok.md +++ b/structure/providers/xai-grok.md @@ -68,7 +68,7 @@ malformed, gapped, oversized, contradictory, failed, or incomplete streams stay - **Safety & Idempotency:** Managed via `src/grok/reset-coupon-ledger.ts` using UUIDv4 operation tracking before upstream dispatch to prevent duplicate consumption during network flakes. - **Surfaces:** `ocx account grok-reset-coupons` in the terminal, and the dashboard at Providers > xAI Grok > Accounts, where each OAuth row carries a ticket badge with its remaining count and opens a redemption dialog (`gui/src/hooks/useGrokResetCoupons.ts`, `gui/src/components/provider-workspace/GrokResetCoupons.tsx`). The dashboard reads one `GET /api/grok/reset-coupons` per account with at most three in flight, always sends an explicit `tokenId` and a client-minted `operationId`, and treats redemption truth as the settled `code` rather than HTTP 200 — a replayed *failure* returns 200 with `replayed: true`. After a request times out it issues no further consume call, because a redemption whose ledger record is still `open` re-executes. -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/runtime.md b/structure/runtime.md index bd9ebbd5619..e62b7e38fd9 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -430,3 +430,11 @@ Translated audio/file admission follows the [final-adapter input contract](adapt The combo may advance to its next eligible unattempted target before output commitment. It records no target/provider cooldown for these request-local mismatches and does not silently drop reasoning controls or raise `none` to a supported rung. Cancellation, origin/cyber-policy rejection, non-replayable post-send errors and the existing streaming commit boundary stay authoritative. Other invalid requests remain terminal. Regression coverage: `tests/responses/responses-forward-prompt-envelope.test.ts`, `tests/routing/router-combo-failover-classification.test.ts`, and `tests/server/server-combo-failover-e2e.test.ts`. + +## Upstream key usage identity + +`src/codex/account-label.ts` owns the provider/selection digest and `src/providers/label.ts` +stamps the configured key selected for the physical request. `src/server/request-log.ts` +retains per-key attempt usage, and `src/usage/log.ts` validates and persists labels. The +[account attribution contract](gui-and-management-api.md#upstream-key-account-attribution) +defines identity, unknown records, and aggregation boundaries. diff --git a/structure/subagents.md b/structure/subagents.md index c0c92891f73..1edc9e2e347 100644 --- a/structure/subagents.md +++ b/structure/subagents.md @@ -321,7 +321,7 @@ Native Codex advertisements still follow display priority; private guidance rank Codex display-cache expiry, retained main-policy evidence, and reset history follow the [quota cache contract](providers/openai-tiers.md#quota-cache-and-short-window-history). -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/transports/byte-accounting.md b/structure/transports/byte-accounting.md index 7f01dee197a..239529ad96a 100644 --- a/structure/transports/byte-accounting.md +++ b/structure/transports/byte-accounting.md @@ -39,3 +39,5 @@ These optimizations do not add request queues, retry policies, or RSS-based admi Translated audio/file admission follows the [final-adapter input contract](../adapters/registry.md#untranslated-input-media); native raw passthrough remains separate. Canonical Responses identity sanitation and narrowly scoped pre-output combo recovery follow [request-local target compatibility](../runtime.md#request-local-target-compatibility); other adapter contracts remain unchanged. + +Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index ca80373a8e9..53122d96eaa 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -96,7 +96,7 @@ Caller-owned `provider.fetch` executors are also deferred: they receive literal/ redirect blocking, but cannot inherit DNS classification or peer pinning without a verified-peer executor contract. Main-request migration must not treat that branch as fixed-transport equivalent. -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 4a6a664cadc..672c43d3a80 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -564,6 +564,15 @@ change target order or attempt accounting; provider-400 decisions follow the [re The shared Responses path follows the [bounded multipart recovery contract](../subagents.md#multipart-encrypted-task-recovery); credential admission and retry policy remain unchanged. +## Upstream key attempt accounting + +Key identity is sealed at the guarded physical dispatch after queued selections are rebuilt. +Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; +repeated parsing of one physical response does not count it twice. Key changes preserve the +previous attempt while retaining the active attempt object shared by streaming/combo callbacks. +Bounded failure-body observation retains reported usage and releases cloned readers on abort. +Identity and consumer aggregation follow the [account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution). + ## Combo streaming commit boundary An HTTP 200 does not by itself commit a streaming combo child. The combo parent runs the child's @@ -693,3 +702,9 @@ What must not happen is a ladder that charges and then returns through a path th nor releases. That is not a lost send; it is a send the request never made, spending an allowance a later recovery in the same request then cannot have. `tests/lib/execution-budget-permits.test.ts` pins both ladder shapes against exactly that. + +Adapter-owned retries enter the same pending dispatch metadata path as initial key sends. +The actual dispatch commits their count and recovery label once; unsent pending metadata +is discarded on process exit and is not usage evidence. See [key attribution](../gui-and-management-api.md#upstream-key-account-attribution). +Generic refetches record metadata inside each admitted retry callback, retaining the +transient recovery reason when present and otherwise the outer recovery reason. diff --git a/structure/transports/streaming-health.md b/structure/transports/streaming-health.md index 682fbb2ca2a..06783ad6518 100644 --- a/structure/transports/streaming-health.md +++ b/structure/transports/streaming-health.md @@ -211,7 +211,7 @@ WebSocket clients observe the same canonical lifecycle. frame rather than always emitting `response.completed`. If the response status is `failed`, a `response.failed` frame is sent; otherwise `response.completed` carries through the original status. -Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. +Usage consumers preserve positive incomplete-history metadata as specified in [usage accounting](../gui-and-management-api.md#usage-accounting); readable totals are not represented as a complete ledger. Upstream API-key usage follows the [physical-attempt account attribution contract](../gui-and-management-api.md#upstream-key-account-attribution), independently of subscription quota observations. Connected CLI usage follows the [client-scoped hub usage contract](../gui-and-management-api.md#usage-accounting); local management and account data remain separate. diff --git a/tests/codex-integration/codex-account-label.test.ts b/tests/codex-integration/codex-account-label.test.ts index 9680e3f8b9d..5053d1fc783 100644 --- a/tests/codex-integration/codex-account-label.test.ts +++ b/tests/codex-integration/codex-account-label.test.ts @@ -1,6 +1,8 @@ import { describe, expect, test } from "bun:test"; import { CODEX_ACCOUNT_LOG_LABEL_RE, + ACCOUNT_LOG_LABEL_RE, + apiKeyAccountLogLabel, codexAccountLogLabel, createCodexAccountLogLabel, fallbackCodexAccountLogLabel, @@ -8,6 +10,22 @@ import { } from "../../src/codex/account-label"; describe("codex account privacy labels", () => { + test("key labels follow the shared consumer contract and isolate provider, slot and reference", () => { + expect(apiKeyAccountLogLabel("test-provider", { entryId: "slot-a", reference: "test-key-a" })) + .toBe("k35f7c109222440212853c90de03e7df5"); + expect(apiKeyAccountLogLabel("test-provider", { entryId: "slot-b", reference: "test-key-b" })) + .toBe("kae34539c9f0b302367a033166800ae47"); + expect(apiKeyAccountLogLabel("test-provider", { reference: "test-key-a" })) + .toBe("ke4869182d193d18777b6ce175baaa41a"); + expect(apiKeyAccountLogLabel("other-provider", { entryId: "slot-a", reference: "test-key-a" })) + .toBe("k98c6a69a98c5537c6acd344f116e7579"); + expect(apiKeyAccountLogLabel("test-provider", undefined)).toBeUndefined(); + expect(apiKeyAccountLogLabel("test-provider", { reference: "" })).toBeUndefined(); + expect(apiKeyAccountLogLabel("test-provider", { reference: "env:MISSING_SYNTHETIC_KEY" })) + .toMatch(ACCOUNT_LOG_LABEL_RE); + expect(ACCOUNT_LOG_LABEL_RE.test("kabc123")).toBe(false); + expect(ACCOUNT_LOG_LABEL_RE.test("k" + "a".repeat(33))).toBe(false); + }); test("generates non-PII log labels", () => { expect(createCodexAccountLogLabel()).toMatch(CODEX_ACCOUNT_LOG_LABEL_RE); }); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index d2eb5d244be..139ed857ebc 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,5 @@ { + "key-attribution.test.ts": "usage", "responses-core-modules.test.ts": "responses", "chat-responses-control-integration.test.ts": "responses", "coding-agent-tool-result-images.test.ts": "adapters", @@ -382,6 +383,7 @@ "combos.test.ts": "codex-integration", "command-code-error-finish.test.ts": "providers", "command-code-provider.test.ts": "providers", + "command-code-retry.test.ts": "providers", "command-code-quota.test.ts": "providers", "command-code-workspace-cache.test.ts": "providers", "commandcode-provider.test.ts": "providers", diff --git a/tests/helpers/combo-provider.ts b/tests/helpers/combo-provider.ts new file mode 100644 index 00000000000..4221adbe4cd --- /dev/null +++ b/tests/helpers/combo-provider.ts @@ -0,0 +1,30 @@ +import type { ProviderAdapter } from "../../src/adapters/base"; +import type { OcxProviderConfig } from "../../src/types"; + +/** Keep the fixture upstream behind the same executor used by real provider sends. */ +export function comboProviderFactory( + getFetchResponse: () => ProviderAdapter["fetchResponse"], +) { + return function provider( + adapter: string, + url: string, + apiKey: string, + extra: Partial = {}, + ): OcxProviderConfig { + return { + adapter, + baseUrl: url, + allowPrivateNetwork: url.includes("127.0.0.1"), + authMode: "key", + apiKey, + ...(adapter === "test-response" ? { fetch: (async (input, init) => { + const customFetchResponse = getFetchResponse(); + if (!customFetchResponse) throw new Error("custom fetchResponse not installed"); + return customFetchResponse({ url: String(input), method: init?.method ?? "POST", + headers: Object.fromEntries(new Headers(init?.headers)), body: String(init?.body ?? "") }, + { abortSignal: init?.signal ?? undefined }); + }) as typeof globalThis.fetch } : {}), + ...extra, + }; + }; +} diff --git a/tests/providers/command-code-retry.test.ts b/tests/providers/command-code-retry.test.ts new file mode 100644 index 00000000000..d0f74df9800 --- /dev/null +++ b/tests/providers/command-code-retry.test.ts @@ -0,0 +1,113 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createCommandCodeAdapter } from "../../src/adapters/command-code"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { resetCommandCodeReasoningEffortsForTest } from "../../src/providers/command-code-efforts"; +import { handleResponses } from "../../src/server/responses"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { AdapterFetchContext, AdapterRequest } from "../../src/adapters/base"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +let home: string, previousHome: string | undefined, codexHome: IsolatedCodexHome; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-command-retry-")); + process.env.OPENCODEX_HOME = home; + codexHome = installIsolatedCodexHome("ocx-command-retry-codex-"); + resetCommandCodeReasoningEffortsForTest(); +}); +afterEach(() => { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + codexHome.restore(); + removeTreeWithRetry(home); + resetCommandCodeReasoningEffortsForTest(); +}); + +const request: AdapterRequest = { url: "https://command.test/alpha/generate", method: "POST", headers: {}, + body: JSON.stringify({ params: { model: "deepseek/deepseek-v4-flash", reasoning_effort: "max" } }) }; +const provider: OcxProviderConfig = { adapter: "command-code", baseUrl: "https://command.test", authMode: "key", apiKey: "synthetic-key" }; +function fixture(status = 400) { + const bodies: string[] = []; + const executor = (async (_url, init) => { + bodies.push(String(init?.body)); + return bodies.length === 1 ? Response.json({ error: "unsupported reasoning_effort", usage: { prompt_tokens: 3, completion_tokens: 1 } }, { status }) : new Response([ + { type: "text-delta", text: "answer" }, + { type: "finish", finishReason: "stop", totalUsage: { inputTokens: 10, outputTokens: 2 } }, + ].map(row => JSON.stringify(row) + "\n").join("")); + }) as typeof fetch; + const fetch = (async (url, init) => String(url).includes("commandcode.ai/models/") + ? new Response("Reasoning efforts high are supported; no other reasoning settings.") + : executor(url, init)) as typeof globalThis.fetch; + return { bodies, executor, provider: { ...provider, fetch } as OcxProviderConfig }; +} + +test.each([400, 422])("effort rejection %s reserves and observes only the additional send", async status => { + const f = fixture(status), budget = createRequestExecutionBudget(); + const sends: Parameters>[0][] = []; + const response = await createCommandCodeAdapter(f.provider).fetchResponse!(request, { + executor: f.executor, sendBudget: budget, onPhysicalSend: send => sends.push(send), + }); + expect(response.status).toBe(200); + expect(f.bodies).toHaveLength(2); + expect(JSON.parse(f.bodies[1]!).params).not.toHaveProperty("reasoning_effort"); + expect(budget.used).toBe(2); + expect(sends).toEqual([{ ordinal: 2, recovery: "reasoning-effort-downgrade" }]); +}); + +test("exhaustion preserves the original error body without sending or observing a retry", async () => { + const f = fixture(), budget = createRequestExecutionBudget({ + maxTotalModelSends: 1, baseSendAllowance: 1, finalRecoveryAllowance: 0, maxAlternateTargetSends: 0, maxTargetTransitions: 0, + }); + let observed = 0; + const response = await createCommandCodeAdapter(f.provider).fetchResponse!(request, { + executor: f.executor, sendBudget: budget, onPhysicalSend: () => observed++, + }); + expect(response.status).toBe(400); + expect(await response.json()).toEqual({ error: "unsupported reasoning_effort", usage: { prompt_tokens: 3, completion_tokens: 1 } }); + expect(f.bodies).toHaveLength(1); + expect(observed).toBe(0); + expect(budget.used).toBe(budget.policy.maxTotalModelSends); +}); + +test("an exhausted entry budget cannot send the initial request", async () => { + const f = fixture(), budget = createRequestExecutionBudget(); + budget.used = budget.policy.maxTotalModelSends; + let observed = 0; + await expect(createCommandCodeAdapter(f.provider).fetchResponse!(request, { + executor: f.executor, sendBudget: budget, onPhysicalSend: () => observed++, + })).rejects.toMatchObject({ code: "request_send_budget_exhausted" }); + expect(f.bodies).toHaveLength(0); + expect(observed).toBe(0); +}); + +test("a non-retry response leaves entry accounting to the caller", async () => { + const f = fixture(403), budget = createRequestExecutionBudget(); + let observed = 0; + const response = await createCommandCodeAdapter(f.provider).fetchResponse!(request, { + executor: f.executor, sendBudget: budget, onPhysicalSend: () => observed++, + }); + expect(response.status).toBe(403); + expect(f.bodies).toHaveLength(1); + expect(observed).toBe(0); + expect(budget.used).toBe(1); +}); + +test.each([false, true])("routed key retry counts each dispatch once (stream=%s)", async stream => { + const f = fixture(), logCtx: RequestLogContext = { provider: "", model: "" }; + const budget = createRequestExecutionBudget(); + const config = { port: 0, defaultProvider: "fixture", providers: { fixture: f.provider } } as OcxConfig; + const req = new Request("http://localhost/v1/responses", { method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "fixture/deepseek/deepseek-v4-flash", input: "hello", reasoning: { effort: "max" }, stream }) }); + const response = await handleResponses(req, config, logCtx, { sendBudget: budget }); + expect(await response.text()).toContain("answer"); + expect(response.status).toBe(200); + expect(f.bodies).toHaveLength(2); + expect(budget.used).toBe(2); + expect(logCtx.activeAttempt).toMatchObject({ sendCount: 2, recoveryKinds: ["reasoning-effort-downgrade"], + usage: { inputTokens: 13, outputTokens: 3 } }); +}); diff --git a/tests/providers/rate-limit-retry.test.ts b/tests/providers/rate-limit-retry.test.ts index 709ccdae128..b351057055b 100644 --- a/tests/providers/rate-limit-retry.test.ts +++ b/tests/providers/rate-limit-retry.test.ts @@ -121,14 +121,16 @@ describe("retry loop client-abort handling", () => { test("abort during the wait interrupts the sleep, cancels the 429 body, and returns 499 without replaying", async () => { let sends = 0; let upstreamBodyCancelled = false; + let upstreamBodyDrained = false; globalThis.fetch = (async (input, init) => { const url = input instanceof Request ? input.url : String(input); if (url === "https://llmapi.blsc.cn/chat/completions") { sends += 1; return new Response(new ReadableStream({ - start(controller) { + pull(controller) { controller.enqueue(new TextEncoder().encode(JSON.stringify({ error: { message: "rate limited" } }))); controller.close(); + upstreamBodyDrained = true; }, cancel() { upstreamBodyCancelled = true; @@ -167,7 +169,7 @@ describe("retry loop client-abort handling", () => { const response = await pending; expect(response.status).toBe(499); expect(sends).toBe(1); - expect(upstreamBodyCancelled).toBe(true); + expect(upstreamBodyCancelled || upstreamBodyDrained).toBe(true); const body = await response.json() as { error?: { code?: string } }; expect(body.error?.code).toBe("client_cancelled"); }); @@ -182,7 +184,7 @@ describe("retry loop client-abort handling", () => { return new Response(new ReadableStream({ start(controller) { controller.enqueue(new TextEncoder().encode(JSON.stringify({ error: { message: "rate limited" } }))); - controller.close(); + // Keep the source open so abort must cancel both accounting tee branches. }, cancel() { cancelInitiated = true; diff --git a/tests/responses/chat-completions-endpoint.test.ts b/tests/responses/chat-completions-endpoint.test.ts index 82317d89bd8..5eee77907c1 100644 --- a/tests/responses/chat-completions-endpoint.test.ts +++ b/tests/responses/chat-completions-endpoint.test.ts @@ -1606,57 +1606,6 @@ test("chat-native records terminal key cooldown after the send budget is exhaust } }); -test("chat-native preserves same-key retry, key rotation, usage, and request logging", async () => { - const { clearRequestLogsForTests, getRequestLogEntries } = await import("../../src/server/request-log"); - const { clearKeyCooldowns } = await import("../../src/providers/key-failover"); - clearRequestLogsForTests(); - clearKeyCooldowns("mock"); - const authorizations: Array = []; - const upstream = Bun.serve({ - port: 0, - fetch(req) { - authorizations.push(req.headers.get("authorization")); - if (authorizations.length < 3) { - return Response.json({ error: { message: "rate limited", type: "rate_limit_error" } }, { - status: 429, - headers: { "retry-after": "0" }, - }); - } - return Response.json({ - id: "chatcmpl_retry", - object: "chat.completion", - choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], - usage: { prompt_tokens: 4, completion_tokens: 2 }, - }); - }, - }); - saveConfig(mockConfig(`${upstream.url.toString().replace(/\/$/, "")}/v1`, { - authMode: "key", - apiKey: "key-one", - apiKeyPool: [{ id: "one", key: "key-one" }, { id: "two", key: "key-two" }], - retryOn429: { attempts: 1, intervalMs: 100, maxIntervalMs: 100, respectRetryAfter: false }, - })); - const server = startServer(0); - try { - const response = await fetch(new URL("/v1/chat/completions", server.url), { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ model: "mock/test-model", stream: false, messages: [{ role: "user", content: "hi" }] }), - }); - expect(response.status).toBe(200); - await response.text(); - expect(authorizations).toEqual(["Bearer key-one", "Bearer key-one", "Bearer key-two"]); - const entry = getRequestLogEntries().at(-1); - expect(entry?.status).toBe(200); - expect(entry?.usage).toMatchObject({ inputTokens: 4, outputTokens: 2 }); - expect(entry?.attempts?.[0]?.recoveryKinds).toEqual(["rate-limit-429", "key-429"]); - } finally { - await server.stop(true); - upstream.stop(true); - clearKeyCooldowns("mock"); - } -}); - test("chat-native client cancellation cancels the upstream stream and logs 499", async () => { const { clearRequestLogsForTests, getRequestLogEntries } = await import("../../src/server/request-log"); const { handleChatCompletions } = await import("../../src/server/chat-completions"); diff --git a/tests/responses/empty-completion-core.test.ts b/tests/responses/empty-completion-core.test.ts index d8d4d73325a..6ec5541f123 100644 --- a/tests/responses/empty-completion-core.test.ts +++ b/tests/responses/empty-completion-core.test.ts @@ -60,10 +60,8 @@ function fixtureAdapter(provider: OcxProviderConfig): ProviderAdapter & { passth } : {}), }; }, - async fetchResponse() { - const index = httpCalls; - httpCalls += 1; - return new Response("", { headers: { "x-fixture-attempt": String(index) } }); + async fetchResponse(request, context) { + return context!.executor!(request.url, { method: request.method, headers: request.headers, body: request.body }); }, async *parseStream(response) { const index = Number(response.headers.get("x-fixture-attempt")); @@ -79,6 +77,7 @@ function fixtureAdapter(provider: OcxProviderConfig): ProviderAdapter & { passth await customRunTurn(parsed, _incoming as never, emit); return; } + await (_incoming as { providerFetch: typeof fetch }).providerFetch(provider.baseUrl, { method: "POST" }); const index = runTurnCalls; runTurnCalls += 1; parsedAttempts.push(parsed); @@ -123,6 +122,11 @@ function config( }, ...extra, } as OcxConfig; + (result.providers.fixture as OcxProviderConfig & { fetch?: typeof globalThis.fetch }).fetch = async () => { + const index = httpCalls; + if (adapter === "test-http") httpCalls += 1; + return new Response("", { headers: { "x-fixture-attempt": String(index) } }); + }; if (adapter === "test-passthrough") { (result.providers.fixture as OcxProviderConfig & { fetch?: typeof globalThis.fetch }).fetch = async () => { passthroughFetchCalls += 1; diff --git a/tests/server/server-combo-failover-e2e.test.ts b/tests/server/server-combo-failover-e2e.test.ts index 041ed847c04..1e99a091a7e 100644 --- a/tests/server/server-combo-failover-e2e.test.ts +++ b/tests/server/server-combo-failover-e2e.test.ts @@ -1,3 +1,4 @@ +import { comboProviderFactory } from "../helpers/combo-provider"; import { sessionLaneIdFromRequest } from "../../src/server/request-log-conversation"; import { afterEach, beforeEach, describe, expect, mock, setDefaultTimeout, test } from "bun:test"; import { logsFromApiBody } from "../helpers/logs-api"; @@ -58,6 +59,7 @@ const { createCursorAdapter } = await import("../../src/adapters/cursor"); import type { CursorTransportFactory } from "../../src/adapters/cursor/transport"; let customRunTurn: NonNullable | undefined; let customFetchResponse: NonNullable | undefined; +const provider = comboProviderFactory(() => customFetchResponse); let customTransientResponse: (() => Promise) | undefined; let customUsageEstimate: ((model: string) => number | undefined) | undefined; let customCursorTransportFactory: CursorTransportFactory | undefined; @@ -102,7 +104,8 @@ mock.module("../../src/server/adapter-resolve", () => ({ }, async fetchResponse(request, context) { if (!customFetchResponse) throw new Error("custom fetchResponse not installed"); - return customFetchResponse(request, context); + return context!.executor!(request.url, { method: request.method, headers: request.headers, + body: request.body, signal: context?.abortSignal }); }, }; } @@ -251,22 +254,6 @@ function responsesSuccess(text: string, model = "responses-model"): Record = {}, -): OcxProviderConfig { - return { - adapter, - baseUrl: url, - allowPrivateNetwork: url.includes("127.0.0.1"), - authMode: "key", - apiKey, - ...extra, - }; -} - function comboConfig( providers: OcxConfig["providers"], targets = Object.keys(providers).map((name, index) => ({ provider: name, model: `m${index + 1}` })), @@ -1834,7 +1821,7 @@ describe("server combo failover 030 activation matrix", () => { .toEqual({ inputTokens: 17, outputTokens: 3, totalTokens: 20 }); }); - test("provider-local retry keeps one attempt, two sends, recovery kind, and latest estimate", async () => { + test("provider-local key retry keeps separate attempts and the latest estimate on the selected key", async () => { const estimates = [10, 25]; customUsageEstimate = () => estimates.shift(); let calls = 0; @@ -1855,11 +1842,14 @@ describe("server combo failover 030 activation matrix", () => { const response = await postLogged(config); expect(response.status).toBe(200); await response.text(); - const attempt = (await latestAttemptReceipts(config)).usage.attempts?.[0]; + const attempts = (await latestAttemptReceipts(config)).usage.attempts; + expect(attempts).toHaveLength(2); + expect(attempts?.[0]).toMatchObject({ sendCount: 1, usageStatus: "unreported" }); + const attempt = attempts?.[1]; expect(attempt).toMatchObject({ provider: "a", model: "m1", - sendCount: 2, + sendCount: 1, inputTokenEstimate: 25, recoveryKinds: ["key-429"], }); diff --git a/tests/server/server-key-failover-e2e.test.ts b/tests/server/server-key-failover-e2e.test.ts index 418ef993ca7..4bf924c2492 100644 --- a/tests/server/server-key-failover-e2e.test.ts +++ b/tests/server/server-key-failover-e2e.test.ts @@ -1,7 +1,9 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { mkdtempSync} from "node:fs"; +import { mkdtempSync, readFileSync} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; +import { readUsageEntries, resetUsageReadCacheForTests } from "../../src/usage/log"; import { loadConfig, saveConfig } from "../../src/config"; import { clearKeyCooldowns, rotateKeyOn429 } from "../../src/providers/key-failover"; import { deriveXaiConvId } from "../../src/providers/xai-transport"; @@ -81,7 +83,8 @@ describe("server 429 key failover (end-to-end)", () => { expect(providerApiKeySelectionIsCurrent(config, "current", current)).toBe(true); }); - test("native Chat rebuilds a queued request after a manual key selection during pacing", async () => { + test.each(["responses", "chat/completions"])("%s logs only the key selected after pacing", async surface => { + resetUsageReadCacheForTests(); let now = 0; let resumePacing: (() => void) | undefined; const queued = Promise.withResolvers(); @@ -113,9 +116,10 @@ describe("server 429 key failover (end-to-end)", () => { const abort = new AbortController(); try { await waitForProviderRequestSlot("paced", config.providers.paced); - const pending = fetch(new URL("/v1/chat/completions", server.url), { + const pending = fetch(new URL(`/v1/${surface}`, server.url), { method: "POST", headers: { "content-type": "application/json" }, signal: abort.signal, - body: JSON.stringify({ model: "paced/test", stream: false, messages: [{ role: "user", content: "hello" }] }), + body: JSON.stringify({ model: "paced/test", stream: false, + ...(surface === "responses" ? { input: "hello" } : { messages: [{ role: "user", content: "hello" }] }) }), }); await queued.promise; expect(seen).toHaveLength(0); @@ -131,6 +135,11 @@ describe("server 429 key failover (end-to-end)", () => { expect(await response.text()).toContain("current selection"); expect(seen.map(headers => headers.get("authorization"))).toEqual(["Bearer synthetic-second"]); expect(seen[0]!.get("x-static-test")).toBe("retained"); + const rows = readUsageEntries(); + expect(rows).toHaveLength(1); + expect(rows[0].attempts).toHaveLength(1); + expect(rows[0].attempts?.[0]).toMatchObject({ sendCount: 1, + accountLogLabel: apiKeyAccountLogLabel("paced", { entryId: "second", reference: "synthetic-second" }) }); } finally { abort.abort(); await server.stop(true); @@ -333,17 +342,29 @@ describe("server 429 key failover (end-to-end)", () => { } }); - test("routed 429 rotates to the pool's next key and succeeds", async () => { + for (const surface of ["combo", "responses", "chat", "image"] as const) for (const meteredFailure of [false, true]) for (const streaming of [false, true]) { + if (surface === "image" && !streaming) continue; + test(`${surface} key rotation attributes each send (failed usage reported: ${meteredFailure}, streaming: ${streaming})`, async () => { + resetUsageReadCacheForTests(); const seenAuth: string[] = []; upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, - fetch(req) { + async fetch(req) { + const body = await req.json() as { stream?: boolean }; seenAuth.push(req.headers.get("authorization") ?? ""); if (seenAuth.length === 1) { - return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + return new Response(JSON.stringify({ error: { message: "rate limited" }, ...(meteredFailure ? { usage: { prompt_tokens: 10, completion_tokens: 4 } } : {}) }), { status: 429, headers: { "retry-after": "30", "content-type": "application/json" }, }); } + if (body.stream) { + const chunks = [ + { id: "chatcmpl-1", choices: [{ index: 0, delta: { role: "assistant", content: "ok after rotate" }, finish_reason: null }] }, + { id: "chatcmpl-1", choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage: { prompt_tokens: 3, completion_tokens: 2 } }, + ]; + return new Response(chunks.map(chunk => `data: ${JSON.stringify(chunk)}\n\n`).join("") + "data: [DONE]\n\n", + { headers: { "content-type": "text/event-stream" } }); + } return new Response(JSON.stringify({ id: "chatcmpl-1", object: "chat.completion", choices: [{ index: 0, message: { role: "assistant", content: "ok after rotate" }, finish_reason: "stop" }], @@ -353,9 +374,12 @@ describe("server 429 key failover (end-to-end)", () => { }); const config: OcxConfig = { port: 0, hostname: "127.0.0.1", defaultProvider: "pooled", + combos: { fixture: { strategy: "failover", targets: [{ provider: "pooled", model: "some-model" }] } }, + images: { bridgeEnabled: surface === "image" }, providers: { + xai: { adapter: "openai-chat", baseUrl: "https://api.x.ai/v1", authMode: "key", apiKey: "synthetic-unused-image-key" }, pooled: { - adapter: "openai-chat", + adapter: "openai-chat", ...(meteredFailure ? { authMode: "key" as const } : {}), baseUrl: `http://127.0.0.1:${upstream.port}/v1`, allowPrivateNetwork: true, apiKey: "key-alpha-000111222333", @@ -369,21 +393,108 @@ describe("server 429 key failover (end-to-end)", () => { saveConfig(config); const server = startServer(0); try { - const res = await fetch(new URL("/v1/responses", server.url), { + const res = await fetch(new URL(surface === "chat" ? "/v1/chat/completions" : "/v1/responses", server.url), { method: "POST", headers: { "content-type": "application/json" }, - body: JSON.stringify({ model: "pooled/some-model", input: "hello", stream: false }), + body: JSON.stringify(surface === "chat" + ? { model: "pooled/some-model", messages: [{ role: "user", content: "hello" }], stream: streaming } + : { model: surface === "combo" ? "combo/fixture" : "pooled/some-model", input: "hello", stream: streaming, + ...(surface === "image" ? { tools: [{ type: "image_generation" }] } : {}) }), }); expect(res.status).toBe(200); - const json = await res.json() as { output?: { type: string; content?: { text?: string }[] }[] }; - const message = json.output?.find(o => o.type === "message"); - expect(message?.content?.[0]?.text).toBe("ok after rotate"); + expect(await res.text()).toContain("ok after rotate"); expect(seenAuth[0]).toBe("Bearer key-alpha-000111222333"); expect(seenAuth[1]).toBe("Bearer key-beta-444555666777"); + expect(seenAuth).toHaveLength(2); + const rows = readUsageEntries(); + expect(rows).toHaveLength(1); + const attempts = rows[0].attempts!; + expect(attempts).toHaveLength(2); + expect(attempts[0]).toMatchObject({ ordinal: 1, provider: "pooled", model: "some-model", status: 429, + accountLogLabel: apiKeyAccountLogLabel("pooled", { entryId: "k1", reference: "key-alpha-000111222333" }), + usageStatus: meteredFailure ? "reported" : "unreported" }); + if (meteredFailure) expect(attempts[0].usage).toMatchObject({ inputTokens: 10, outputTokens: 4 }); + else expect(attempts[0].usage).toBeUndefined(); + expect(attempts[1]).toMatchObject({ ordinal: 2, provider: "pooled", model: "some-model", status: 200, + accountLogLabel: apiKeyAccountLogLabel("pooled", { entryId: "k2", reference: "key-beta-444555666777" }), + usage: { inputTokens: 3, outputTokens: 2 } }); + const raw = readFileSync(join(testDir, "usage.jsonl"), "utf8"); + expect(raw).not.toContain("key-alpha-000111222333"); + expect(raw).not.toContain("key-beta-444555666777"); } finally { await server.stop(true); } }); + } + + + test("Responses continuation keeps hidden successful A usage when a later 429 rotates to B", async () => { + resetUsageReadCacheForTests(); + const seen: string[] = []; + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization") ?? ""); + if (seen.length === 2) return Response.json({ error: { message: "rate limited" }, + usage: { prompt_tokens: 7, completion_tokens: 1 } }, { status: 429 }); + return Response.json({ id: "chatcmpl-hidden", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: seen.length === 1 ? "" : "recovered" }, finish_reason: "stop" }], + usage: { prompt_tokens: seen.length === 1 ? 100 : 200, completion_tokens: seen.length === 1 ? 10 : 20 } }); + } }); + saveConfig({ port: 0, hostname: "127.0.0.1", defaultProvider: "pooled", emptyCompletionRetry: true, + combos: { hidden: { strategy: "failover", targets: [{ provider: "pooled", model: "test" }] } }, + providers: { pooled: { adapter: "openai-chat", authMode: "key", + baseUrl: `http://127.0.0.1:${upstream.port}/v1`, allowPrivateNetwork: true, + apiKey: "synthetic-first", apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }] } }, + } as OcxConfig); + const server = startServer(0); + try { + const response = await fetch(new URL("/v1/responses", server.url), { method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "pooled/test", input: "hello", stream: false }) }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("recovered"); + expect(seen).toEqual(["Bearer synthetic-first", "Bearer synthetic-first", "Bearer synthetic-second"]); + const rows = readUsageEntries(); + expect(rows).toHaveLength(1); + expect(rows[0].attempts).toHaveLength(2); + expect(rows[0].attempts?.[0]).toMatchObject({ sendCount: 2, usage: { inputTokens: 107, outputTokens: 11 }, + accountLogLabel: apiKeyAccountLogLabel("pooled", { entryId: "first", reference: "synthetic-first" }) }); + expect(rows[0].attempts?.[1]).toMatchObject({ sendCount: 1, usage: { inputTokens: 200, outputTokens: 20 }, + accountLogLabel: apiKeyAccountLogLabel("pooled", { entryId: "second", reference: "synthetic-second" }) }); + expect(rows[0].attempts?.reduce((sum, attempt) => sum + (attempt.usage?.inputTokens ?? 0), 0)).toBe(307); + } finally { await server.stop(true); } + }); + + for (const adapter of ["command-code", "openai-chat"] as const) for (const error of [false, true]) { + test(`${adapter} records one usage observation for a nested parser or HTTP-200 error (${error})`, async () => { + resetUsageReadCacheForTests(); + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch() { + if (adapter === "command-code") return new Response([ + { type: "text-delta", text: "synthetic answer" }, + { type: "finish", finishReason: error ? "error" : "stop", totalUsage: { inputTokens: 100, outputTokens: 20 } }, + ].map(row => JSON.stringify(row) + "\n").join(""), { headers: { "content-type": "application/x-ndjson" } }); + return Response.json({ id: "chatcmpl-error", object: "chat.completion", + ...(error ? { error: { message: "synthetic failure", type: "server_error" } } + : { choices: [{ index: 0, message: { role: "assistant", content: "synthetic answer" }, finish_reason: "stop" }] }), + usage: { prompt_tokens: 100, completion_tokens: 20 } }); + } }); + saveConfig({ port: 0, hostname: "127.0.0.1", defaultProvider: "metered", providers: { + metered: { adapter, authMode: "key", apiKey: "synthetic-key", allowPrivateNetwork: true, + baseUrl: `http://127.0.0.1:${upstream.port}` }, + } } as OcxConfig); + const server = startServer(0); + try { + const response = await fetch(new URL("/v1/responses", server.url), { method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "metered/test", input: "hello", stream: false }) }); + await response.text(); + const rows = readUsageEntries(); + expect(rows).toHaveLength(1); + expect(rows[0].attempts).toHaveLength(1); + expect(rows[0].attempts?.[0]).toMatchObject({ usage: { inputTokens: 100, outputTokens: 20 }, + accountLogLabel: apiKeyAccountLogLabel("metered", { reference: "synthetic-key" }) }); + } finally { await server.stop(true); } + }); + } test("reasoning replay misses after a 429 rotates to a different physical key", async () => { const model = "reasoning-model"; @@ -787,3 +898,164 @@ describe("server 429 key failover (end-to-end)", () => { delete process.env.OCX_KEYFAIL_WARM; } }); + +test.each([false, true])("chat-native attributes same-key 429 usage then the rotated key (stream=%s)", async (streaming) => { + const { clearRequestLogsForTests, getRequestLogEntries } = await import("../../src/server/request-log"); + const { clearKeyCooldowns } = await import("../../src/providers/key-failover"); + clearRequestLogsForTests(); + clearKeyCooldowns("mock"); + const authorizations: Array = []; + const upstream = Bun.serve({ + port: 0, + fetch(req) { + authorizations.push(req.headers.get("authorization")); + if (authorizations.length === 1) { + return Response.json({ error: { message: "rate limited" }, usage: { prompt_tokens: 10, completion_tokens: 1 } }, { + status: 429, headers: { "retry-after": "0", "content-type": "application/json" }, + }); + } + if (authorizations.length === 2) { + return Response.json({ error: { message: "rate limited" }, usage: { prompt_tokens: 7, completion_tokens: 0 } }, { + status: 429, headers: { "retry-after": "0", "content-type": "application/json" }, + }); + } + if (streaming) { + const chunks = [ + { id: "chatcmpl-1", choices: [{ index: 0, delta: { role: "assistant", content: "ok" }, finish_reason: null }] }, + { id: "chatcmpl-1", choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage: { prompt_tokens: 3, completion_tokens: 2 } }, + ]; + return new Response(chunks.map(chunk => `data: ${JSON.stringify(chunk)}\n\n`).join("") + "data: [DONE]\n\n", { + headers: { "content-type": "text/event-stream" }, + }); + } + return Response.json({ + id: "chatcmpl-1", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + usage: { prompt_tokens: 3, completion_tokens: 2 }, + }); + }, + }); + saveConfig({ port: 0, hostname: "127.0.0.1", defaultProvider: "mock", providers: { mock: { + adapter: "openai-chat", allowPrivateNetwork: true, + baseUrl: `${upstream.url.toString().replace(/\/$/, "")}/v1`, + authMode: "key", + apiKey: "key-one", + apiKeyPool: [{ id: "one", key: "key-one" }, { id: "two", key: "key-two" }], + retryOn429: { attempts: 1, intervalMs: 100, maxIntervalMs: 100, respectRetryAfter: false }, + } } } as OcxConfig); + const server = startServer(0); + try { + const response = await fetch(new URL("/v1/chat/completions", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "mock/test-model", stream: streaming, messages: [{ role: "user", content: "hi" }] }), + }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("ok"); + expect(authorizations).toEqual(["Bearer key-one", "Bearer key-one", "Bearer key-two"]); + const entry = getRequestLogEntries().at(-1); + expect(entry?.attempts).toHaveLength(2); + expect(entry?.attempts?.[0]).toMatchObject({ sendCount: 2, usage: { inputTokens: 17, outputTokens: 1 } }); + expect(entry?.attempts?.[1]).toMatchObject({ sendCount: 1, usage: { inputTokens: 3, outputTokens: 2 } }); + } finally { + await server.stop(true); + upstream.stop(true); + clearKeyCooldowns("mock"); + } +}); + +test("chat-native preserves same-key retry, key rotation, usage, and request logging", async () => { + const { clearRequestLogsForTests, getRequestLogEntries } = await import("../../src/server/request-log"); + const { clearKeyCooldowns } = await import("../../src/providers/key-failover"); + clearRequestLogsForTests(); + clearKeyCooldowns("mock"); + const authorizations: Array = []; + const upstream = Bun.serve({ + port: 0, + fetch(req) { + authorizations.push(req.headers.get("authorization")); + if (authorizations.length < 3) { + return Response.json({ error: { message: "rate limited", type: "rate_limit_error" } }, { + status: 429, + headers: { "retry-after": "0" }, + }); + } + return Response.json({ + id: "chatcmpl_retry", + object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + usage: { prompt_tokens: 4, completion_tokens: 2 }, + }); + }, + }); + saveConfig({ port: 0, hostname: "127.0.0.1", defaultProvider: "mock", providers: { mock: { + adapter: "openai-chat", allowPrivateNetwork: true, + baseUrl: `${upstream.url.toString().replace(/\/$/, "")}/v1`, + authMode: "key", + apiKey: "key-one", + apiKeyPool: [{ id: "one", key: "key-one" }, { id: "two", key: "key-two" }], + retryOn429: { attempts: 1, intervalMs: 100, maxIntervalMs: 100, respectRetryAfter: false }, + } } } as OcxConfig); + const server = startServer(0); + try { + const response = await fetch(new URL("/v1/chat/completions", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "mock/test-model", stream: false, messages: [{ role: "user", content: "hi" }] }), + }); + expect(response.status).toBe(200); + await response.text(); + expect(authorizations).toEqual(["Bearer key-one", "Bearer key-one", "Bearer key-two"]); + const entry = getRequestLogEntries().at(-1); + expect(entry?.status).toBe(200); + expect(entry?.usage).toMatchObject({ inputTokens: 4, outputTokens: 2 }); + expect(entry?.attempts).toHaveLength(2); + expect(entry?.attempts?.[0]?.recoveryKinds).toEqual(["rate-limit-429"]); + expect(entry?.attempts?.[0]?.sendCount).toBe(2); + expect(entry?.attempts?.[1]?.recoveryKinds).toEqual(["key-429"]); + expect(entry?.attempts?.[1]?.sendCount).toBe(1); + } finally { + await server.stop(true); + upstream.stop(true); + clearKeyCooldowns("mock"); + } +}); + +test.each([false, true])("key refetch retains transient recovery metadata (stream=%s)", async stream => { + resetUsageReadCacheForTests(); + const seen: Array = []; + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization")); + if (seen.length < 3) return Response.json({ error: { message: seen.length === 1 ? "rate limited" : "temporarily unavailable" }, + usage: { prompt_tokens: seen.length, completion_tokens: 0 } }, { + status: seen.length === 1 ? 429 : 503, headers: { "retry-after": "0" }, + }); + const usage = { prompt_tokens: 10, completion_tokens: 2 }; + if (stream) return new Response([ + { id: "chatcmpl-refetch", choices: [{ index: 0, delta: { content: "recovered" }, finish_reason: null }] }, + { id: "chatcmpl-refetch", choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage }, + ].map(chunk => `data: ${JSON.stringify(chunk)}\n\n`).join("") + "data: [DONE]\n\n", { + headers: { "content-type": "text/event-stream" }, + }); + return Response.json({ id: "chatcmpl-refetch", object: "chat.completion", usage, + choices: [{ index: 0, message: { role: "assistant", content: "recovered" }, finish_reason: "stop" }] }); + } }); + saveConfig({ port: 0, hostname: "127.0.0.1", defaultProvider: "refetch", providers: { refetch: { + adapter: "openai-chat", authMode: "key", allowPrivateNetwork: true, baseUrl: `http://127.0.0.1:${upstream.port}/v1`, + apiKey: "synthetic-refetch-a", apiKeyPool: [{ id: "a", key: "synthetic-refetch-a" }, { id: "b", key: "synthetic-refetch-b" }], + transientRetryOn5xx: { attempts: 3 }, retryOn429: { attempts: 0 }, + } } } as OcxConfig); + const server = startServer(0); + try { + const response = await fetch(new URL("/v1/responses", server.url), { method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "refetch/test", input: "hello", stream }) }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("recovered"); + expect(seen).toEqual(["Bearer synthetic-refetch-a", "Bearer synthetic-refetch-b", "Bearer synthetic-refetch-b"]); + const attempts = readUsageEntries()[0]?.attempts; + expect(attempts).toHaveLength(2); + expect(attempts?.[0]).toMatchObject({ sendCount: 1, usage: { inputTokens: 1, outputTokens: 0 } }); + expect(attempts?.[1]).toMatchObject({ sendCount: 2, recoveryKinds: ["key-429", "transient-5xx"], + usage: { inputTokens: 12, outputTokens: 2 } }); + } finally { await server.stop(true); } +}); diff --git a/tests/server/server-xai-oauth-401-replay.test.ts b/tests/server/server-xai-oauth-401-replay.test.ts index aab488ccb5c..887415af39d 100644 --- a/tests/server/server-xai-oauth-401-replay.test.ts +++ b/tests/server/server-xai-oauth-401-replay.test.ts @@ -317,11 +317,13 @@ describe("xAI OAuth Responses opt-in upstream 401 replay", () => { expect(seenAuth).toEqual([`Bearer ${firstKey}`, `Bearer ${secondKey}`]); const entries = readUsageEntries(); expect(entries).toHaveLength(1); - const attempt = entries[0]?.attempts?.[0]; - expect(entries[0]?.attempts).toHaveLength(1); + const attempts = entries[0]?.attempts; + expect(attempts).toHaveLength(2); + for (const attempt of attempts ?? []) { expect(attempt?.credentialSource).toBe("xai-api-key"); - expect(attempt?.sendCount).toBe(2); + expect(attempt?.sendCount).toBe(1); expect(attempt?.adapter).toBe("openai-chat"); + } const persisted = readFileSync(usageLogPath(), "utf8"); expect(persisted).not.toContain(firstKey); expect(persisted).not.toContain(secondKey); diff --git a/tests/usage/key-attribution.test.ts b/tests/usage/key-attribution.test.ts new file mode 100644 index 00000000000..78359416e94 --- /dev/null +++ b/tests/usage/key-attribution.test.ts @@ -0,0 +1,115 @@ +import { describe, expect, test } from "bun:test"; +import { + addFinalRequestLog, beginRequestAttempt, finishRequestAttempt, noteProviderAttemptSend, + recordKeyAttemptFailure, recordKeyAttemptUsage, applyResponseLogMetadata, + inspectResponseLogSsePayload, type RequestLogContext, type RequestLogEntry, +} from "../../src/server/request-log"; +import { normalizeUsageEntryForTest } from "../../src/usage/log"; + +describe("key attempt accounting", () => { + test("a combo parent copied before streaming rotation cannot overwrite the final key's usage", () => { + const active = beginRequestAttempt(1, "test", "model", "openai-chat"); + const child: RequestLogContext = { provider: "test", model: "model", comboId: "stream", + activeAttempt: active, attempts: [active] }; + const key = (reference: string) => ({ adapter: "openai-chat" as const, authMode: "key" as const, + baseUrl: "https://example.test", _apiKeyAttempt: { reference } }); + noteProviderAttemptSend(child, "test", key("synthetic-a"), undefined); + recordKeyAttemptUsage(child, { inputTokens: 100, outputTokens: 10 }); + const parent = { ...child }; + noteProviderAttemptSend(child, "test", key("synthetic-b"), undefined, "key-429"); + recordKeyAttemptUsage(child, { inputTokens: 200, outputTokens: 20 }); + const rows: RequestLogEntry[] = []; + addFinalRequestLog("stream-key-switch", Date.now(), parent, 200, undefined, row => rows.push(row)); + expect(rows[0].attempts?.map(attempt => attempt.usage?.inputTokens)).toEqual([100, 200]); + expect(rows[0].usage).toMatchObject({ inputTokens: 300, outputTokens: 30 }); + }); + test("adding reported usage cannot upgrade an earlier estimate to a measurement", () => { + const active = beginRequestAttempt(1, "test", "model", "openai-chat"); + const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active }; + recordKeyAttemptUsage(ctx, { inputTokens: 100, outputTokens: 0, estimated: true }); + recordKeyAttemptUsage(ctx, { inputTokens: 10, outputTokens: 2 }); + finishRequestAttempt(active, 429, 1); + expect(active).toMatchObject({ usageStatus: "estimated", usage: { inputTokens: 110, outputTokens: 2, estimated: true } }); + }); + test.each(["synthetic-a", undefined])("a late unreported segment (%s) cannot reuse a stale parent total", reference => { + const active = beginRequestAttempt(1, "test", "model", "openai-chat"); + const child: RequestLogContext = { provider: "test", model: "model", comboId: "stream", + activeAttempt: active, attempts: [active] }; + const key = (value?: string) => ({ adapter: "openai-chat" as const, authMode: "key" as const, + baseUrl: "https://example.test", _apiKeyAttempt: value ? { reference: value } : undefined }); + noteProviderAttemptSend(child, "test", key("synthetic-a"), undefined); + recordKeyAttemptUsage(child, { inputTokens: 100, outputTokens: 10 }); + const parent = { ...child }; + noteProviderAttemptSend(child, "test", key("synthetic-b"), undefined, "key-429"); + noteProviderAttemptSend(child, "test", key(reference), undefined, "key-429"); + const rows: RequestLogEntry[] = []; + addFinalRequestLog("stale-parent", Date.now(), parent, 499, undefined, row => rows.push(row)); + expect(rows[0].attempts?.map(attempt => attempt.usage?.inputTokens)).toEqual([100, undefined, undefined]); + expect(rows[0].attempts?.[2].usageStatus).toBe("unreported"); + }); + test("rotation preserves reported failure usage and the stable active object; unknown stays unreported", async () => { + const provider = (reference?: string) => ({ adapter: "openai-chat" as const, authMode: "key" as const, + baseUrl: "https://example.test/v1", _apiKeyAttempt: reference ? { reference } : undefined }); + const active = beginRequestAttempt(1, "test-provider", "test-model", "openai-chat"); + const ctx: RequestLogContext = { provider: "test-provider", model: "test-model", comboId: "fixture", + activeAttempt: active, activeAttemptStartedAt: Date.now(), attempts: [active] }; + noteProviderAttemptSend(ctx, "test-provider", provider("test-key-a"), 99999); + const labelA = active.accountLogLabel; + const failed = Response.json({ usage: { prompt_tokens: 100, completion_tokens: 20 } }, { status: 429 }); + await recordKeyAttemptFailure(ctx, failed); + expect(await failed.json()).toEqual({ usage: { prompt_tokens: 100, completion_tokens: 20 } }); + noteProviderAttemptSend(ctx, "test-provider", provider("test-key-b"), 200, "rate-limit-429"); + expect(ctx.activeAttempt).toBe(active); + expect(ctx.attempts).toHaveLength(2); + expect(ctx.attempts?.[0]).toMatchObject({ accountLogLabel: labelA, status: 429, ordinal: 1, + usageStatus: "reported", usage: { inputTokens: 100, outputTokens: 20 }, sendCount: 1 }); + expect(active.accountLogLabel).not.toBe(labelA); + expect(active.usage).toBeUndefined(); + // The next failure reports no usage. It must not inherit the preceding account's usage or an estimate. + await recordKeyAttemptFailure(ctx, Response.json({ error: "synthetic" }, { status: 401 })); + noteProviderAttemptSend(ctx, "test-provider", provider(), 300, "key-401"); + expect(ctx.attempts?.[1]).toMatchObject({ status: 401, ordinal: 2, usageStatus: "unreported" }); + expect(ctx.attempts?.[1].usage).toBeUndefined(); + expect(active.accountLogLabel).toBeUndefined(); + recordKeyAttemptUsage(ctx, { inputTokens: 300, outputTokens: 40 }); + const rows: RequestLogEntry[] = []; + addFinalRequestLog("key-rotation", Date.now(), ctx, 200, undefined, row => rows.push(row)); + const roundTrip = normalizeUsageEntryForTest(JSON.parse(JSON.stringify(rows[0])))!; + expect(roundTrip.attempts).toHaveLength(3); + expect(roundTrip.attempts?.map(a => a.ordinal)).toEqual([1, 2, 3]); + expect(roundTrip.attempts?.[0].accountLogLabel).toBe(labelA); + expect(roundTrip.attempts?.[2].usage).toMatchObject({ inputTokens: 300, outputTokens: 40 }); + expect(roundTrip.usage).toMatchObject({ inputTokens: 400, outputTokens: 60 }); + expect(JSON.stringify(roundTrip)).not.toContain("test-key-"); + }); + test("wire snapshots replace the current send against a pre-send baseline", () => { + const active = beginRequestAttempt(1, "test", "model", "openai-chat"); + const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active, attempts: [active] }; + const key = { adapter: "openai-chat" as const, authMode: "key" as const, baseUrl: "https://example.test", _apiKeyAttempt: { reference: "synthetic-a" } }; + noteProviderAttemptSend(ctx, "test", key, undefined); + applyResponseLogMetadata(ctx, { usage: { prompt_tokens: 10, completion_tokens: 1 } }); + inspectResponseLogSsePayload(ctx, JSON.stringify({ usage: { prompt_tokens: 10, completion_tokens: 1 } })); + noteProviderAttemptSend(ctx, "test", key, undefined); + applyResponseLogMetadata(ctx, { usage: { prompt_tokens: 3, completion_tokens: 2 } }); + applyResponseLogMetadata(ctx, { usage: { prompt_tokens: 3, completion_tokens: 2 } }); + finishRequestAttempt(active, 200, 1); + expect(active.usage).toMatchObject({ inputTokens: 13, outputTokens: 3 }); + expect(active.usage?.estimated).toBeUndefined(); + }); + test("an estimated baseline plus repeated and progressive wire snapshots stay one current send", () => { + const active = beginRequestAttempt(1, "test", "model", "openai-chat"); + const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active, attempts: [active] }; + const key = { adapter: "openai-chat" as const, authMode: "key" as const, baseUrl: "https://example.test", _apiKeyAttempt: { reference: "synthetic-a" } }; + noteProviderAttemptSend(ctx, "test", key, undefined); + recordKeyAttemptUsage(ctx, { inputTokens: 10, outputTokens: 0, estimated: true }); + noteProviderAttemptSend(ctx, "test", key, undefined); + applyResponseLogMetadata(ctx, { usage: { prompt_tokens: 3, completion_tokens: 2 } }); + applyResponseLogMetadata(ctx, { usage: { prompt_tokens: 3, completion_tokens: 2 } }); + applyResponseLogMetadata(ctx, { usage: { prompt_tokens: 4, completion_tokens: 2 } }); + finishRequestAttempt(active, 200, 1); + expect(active).toMatchObject({ + usageStatus: "estimated", + usage: { inputTokens: 14, outputTokens: 2, estimated: true }, + }); + }); +});