diff --git a/docs-site/src/content/docs/guides/claude-code.md b/docs-site/src/content/docs/guides/claude-code.md index 14dbe92c81..a24be41ebf 100644 --- a/docs-site/src/content/docs/guides/claude-code.md +++ b/docs-site/src/content/docs/guides/claude-code.md @@ -27,9 +27,15 @@ rotation does not protect against provider enforcement. Operational contract when enabled: -- Upstream **429** cools that account using `Retry-After` when present (else a default backoff), - clears its affinities, and may rotate to another eligible account within the same request - (bounded). +- Upstream **429** cools that account, clears its affinities, and may rotate to another eligible + account within the same request (bounded). The cooldown uses `Retry-After` when present, + otherwise the reset time of whichever window Anthropic marks `rejected` — a drained 5-hour + window is honoured up to six hours rather than being retried every few minutes. A refusal that + states neither falls back to a default backoff. +- Responses report the serving account's 5-hour and weekly utilization, and whichever of those + two the response carries is recorded for that account — each window independently, and a + refusal counts as well as a success. Usage-aware selection works from ordinary traffic, + without waiting for a dashboard poll. - Affinity is **process-local** (lost on proxy restart). - **401/403** credential failures quarantine the account (`needsReauth`) so it is excluded from selection until re-authenticated. diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 85a9ba6903..9eb24561b4 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -430,14 +430,24 @@ rotation may trigger provider restrictions. | `anthropicAccountPool.enabled?` | `boolean` | `false` | Enable sticky session affinity and quota-ranked new-session selection. **429 failover is not gated here**: it activates whenever two or more usable accounts are stored, exactly like every other multi-credential provider, and cannot be switched off. | | `anthropicAccountPool.autoSwitchThreshold?` | `number` | `80` | For new sessions, when the active account reaches this threshold, choose the lowest known cached usage in the configured window; the account chosen does not itself have to be at or above the threshold. `0` disables **proactive** usage-based switching only — new-session selection and routing recovery after an eligible 429 still consult `quotaWindow`. | | `anthropicAccountPool.strategy?` | `"quota" \| "round-robin" \| "fill-first"` | `"quota"` | New-session strategy; `quota` ranks accounts by the window set by `quotaWindow`, and `fill-first` evaluates its drain threshold in that same window. | -| `anthropicAccountPool.quotaWindow?` | `"five-hour" \| "weekly" \| "max-utilization"` | `"five-hour"` | The cached provider-reported utilization bar used for usage-aware account selection. `five-hour` keeps the original behavior. `weekly` scores the weekly bar and skips accounts whose 5-hour bar is exhausted while another eligible account remains, but falls back to exhausted candidates when none do. `max-utilization` scores the highest known bar, so it can use 5-hour usage before weekly usage is available; if neither is known, the account follows unknown-usage ordering. Known usage ranks before unknown usage under the opt-in `weekly` and `max-utilization` windows only; an omitted or explicit `five-hour` preserves the legacy ordering. If every eligible account is unknown, selection still returns one in eligible order. After the documented lower-5-hour tie-break, exact ties preserve eligible order. A healthy affinity-bound session is not proactively rebalanced. For new-session assignment and routing recovery after an eligible 429 replacement, `quota` ranks eligible candidates directly with this window; `fill-first` advances in stable order using this window's threshold and exhaustion rules; `round-robin` ignores it. Cooldown, failover limits, and reauthentication eligibility remain separate local state. Per-account weekly bars are only known once the dashboard Providers page has polled them. | +| `anthropicAccountPool.quotaWindow?` | `"five-hour" \| "weekly" \| "max-utilization"` | `"five-hour"` | The cached provider-reported utilization bar used for usage-aware account selection. `five-hour` keeps the original behavior. `weekly` scores the weekly bar and skips accounts whose 5-hour bar is exhausted while another eligible account remains, but falls back to exhausted candidates when none do. `max-utilization` scores the highest known bar, so it can use 5-hour usage before weekly usage is available; if neither is known, the account follows unknown-usage ordering. Known usage ranks before unknown usage under the opt-in `weekly` and `max-utilization` windows only; an omitted or explicit `five-hour` preserves the legacy ordering. If every eligible account is unknown, selection still returns one in eligible order. After the documented lower-5-hour tie-break, exact ties preserve eligible order. A healthy affinity-bound session is not proactively rebalanced. For new-session assignment and routing recovery after an eligible 429 replacement, `quota` ranks eligible candidates directly with this window; `fill-first` advances in stable order using this window's threshold and exhaustion rules; `round-robin` ignores it. Cooldown, failover limits, and reauthentication eligibility remain separate local state. Per-account 5-hour and weekly bars come from the rate-limit headers on responses that account served; the model-scoped weekly bars are only known once the dashboard Providers page has polled them. | | `anthropicAccountPool.stickyLimit?` | `number` | `1` | Successful new-session binds retained on one round-robin selection. Range 1–100. | -When enabled, 429 records bounded cooldown from `Retry-After` or a default backoff and may rotate -within the request. Affinity is process-local and size-bounded. Credential 401/403 marks the account -as needing reauthentication. If all eligible accounts are cooling, clients receive 429 with +When enabled, 429 records bounded cooldown and may rotate within the request. The cooldown length +comes from `Retry-After` when the refusal carries one, otherwise from the reset time of whichever +rate-limit window Anthropic reports as `rejected`; a stated reset is honoured up to six hours, so a +drained 5-hour window is not re-offered every few minutes. A refusal that states neither falls back +to a short default backoff. Affinity is process-local and size-bounded. Credential 401/403 marks the +account as needing reauthentication. If all eligible accounts are cooling, clients receive 429 with `Retry-After` when known, not an authentication error. +Anthropic responses also report the serving account's 5-hour and weekly utilization, and whichever +of those two a given response carries is recorded against that account — each window independently, +on refusals as well as successes. Usage-aware selection therefore works from the accounts you +actually use, without waiting for the dashboard Providers page to poll them. These readings refresh +the existing row rather than replacing it, so the model-scoped weekly bars that only the usage +endpoint reports are preserved. + :::caution[Experimental] Leave this disabled unless you understand Anthropic account policy risk. Prefer manual `ocx account use anthropic ` switching when unsure. diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 776857ac80..96dc1561e3 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -204,6 +204,7 @@ "anthropic-image-retry.test.ts": "adapters/anthropic", "anthropic-pool-toggle-copy.test.ts": "adapters/anthropic", "anthropic-quorum-cache.test.ts": "routing", + "anthropic-ratelimit-headers.test.ts": "adapters/anthropic", "anthropic-reasoning.test.ts": "adapters/anthropic", "anthropic-sidecar-account-failover.test.ts": "adapters/anthropic", "anthropic-stream-hardening.test.ts": "adapters/anthropic", diff --git a/src/images/loop.ts b/src/images/loop.ts index e3a7f8252f..040a033db1 100644 --- a/src/images/loop.ts +++ b/src/images/loop.ts @@ -263,8 +263,26 @@ export interface ImageBridgeDeps { * Optional 429 failover for the routed (non-xAI) model. Return a rebuilt adapter for the * rotated credential, or null when the pool is exhausted. Async hooks support OAuth refresh; * existing synchronous key-pool hooks remain valid. + * + * `responseHeaders` carries the whole refusal, not just Retry-After, because an Anthropic + * 429 states the window's reset epoch even when it omits Retry-After -- and a rotation that + * cannot see it cools the drained account for the short default instead of until the window + * actually reopens. Optional so existing callers keep compiling. */ - on429?: (retryAfterHeader: string | null) => ProviderAdapter | null | Promise; + on429?: ( + retryAfterHeader: string | null, + responseHeaders?: Headers, + ) => ProviderAdapter | null | Promise; + /** + * Called with the headers of each upstream response the loop accepts. + * + * A bridge iteration is a billed provider call like any other, and Anthropic reports the + * serving account's remaining headroom on every one of them. Without this seam the caller + * only ever hears about the refusals (`on429`), so a workload that runs mostly through the + * bridge contributes no quota measurement at all -- the divergence between sidecar and main + * path that the 429 arm already had to be fixed for once. + */ + onUpstreamResponse?: (responseHeaders: Headers) => void; /** Opt-in same-target 429 policy (key-auth providers). When present, 429 replays on the SAME key before on429 rotation. */ retryOn429Policy?: Required | null; /** Called when the bridged Responses stream completes (parity with runTurn / routed paths). */ @@ -579,7 +597,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise {}); } catch { /* already closed */ } adapter = rotated; @@ -610,6 +628,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise; + const PROVIDER = "anthropic"; +/** + * The cooldown for a refusal that states nothing about when it ends. + * + * A fixed short backoff and no ceiling of its own: with no upstream evidence there is + * nothing to bound. The old 15-minute MAX_COOLDOWN_MS capped the STATED durations instead + * and was removed with that clamp -- keeping it would have described a limit that no + * remaining path applies. `generic-account-failover.ts` still has its own copy for the + * providers it serves. + */ const DEFAULT_COOLDOWN_MS = 60_000; -const MAX_COOLDOWN_MS = 15 * 60_000; +/** + * Ceiling for a cooldown derived from an upstream-stated reset time. + * + * Anthropic's five-hour window answers a drained account with `Retry-After: 7999` and a + * matching `anthropic-ratelimit-unified-5h-reset`. Clamping that to 15 minutes does not + * shorten the ban -- upstream keeps refusing -- it only makes the pool re-offer the same + * exhausted account every quarter hour and hand the client another 429. + * + * Six hours covers the five-hour window with margin. It does NOT cover the seven-day one, + * and that is the deliberate half of this number: an account whose WEEKLY window is spent + * states a reset days away, and this ceiling re-offers it every six hours until then. The + * alternative -- honouring a multi-day reset -- benches an account for days on a single + * refusal, and nothing here can distinguish a genuinely drained week from a reset the + * operator has since topped up, changed plan on, or that upstream revised. Six hours is + * the cost of being wrong about that, paid once per six hours instead of once per fifteen + * minutes. Anything beyond the ceiling is treated as a bound, not as a fact about the + * account. + */ +const MAX_MEASURED_COOLDOWN_MS = 6 * 60 * 60_000; const AFFINITY_IDLE_TTL_MS = 24 * 60 * 60_000; const MAX_AFFINITY_ENTRIES = 2_000; const MAX_AFFINITY_COMPONENT_BYTES = 512; @@ -58,9 +93,19 @@ export interface AnthropicAccountPoolConfig { quotaWindow?: OcxAccountPoolQuotaWindow; } +/** + * Where a cooldown's length came from. Same vocabulary as `CodexCooldownSource`, because it + * answers the same question for the same reason: `retry-after` is upstream answering THIS + * refusal, `reset-derived` is upstream stating when the spent window reopens, and `default` + * is our own guess. The dashboard renders the first as a rate limit and the rest as quota, + * which is exactly the distinction a reset-derived cooldown carries -- collapsing it into + * `retry-after` would report a drained five-hour window as request-rate throttling. + */ +type AnthropicCooldownSource = "retry-after" | "reset-derived" | "default"; + interface AccountHealth { cooldownUntil: number; - cooldownSource: "retry-after" | "default"; + cooldownSource: AnthropicCooldownSource; } interface AffinityEntry { @@ -112,19 +157,59 @@ export function anthropicQuotaWindow(config: AnthropicAccountPoolConfig): OcxAcc return normalizeAccountPoolQuotaWindow(config.quotaWindow); } +/** + * A Retry-After header as a cooldown length in ms. + * + * Both forms the RFC allows are a STATEMENT by upstream about when it will serve this + * account again, so both are bounded by MAX_MEASURED_COOLDOWN_MS rather than by the + * 15-minute ceiling that used to apply here. Truncating a stated 2h13m to 15 minutes was + * the bug: it does not shorten the ban, it just re-offers the exhausted account four times + * an hour, and each attempt spends a real request to earn another 429. + */ function parseRetryAfterMs(value: string | null | undefined, now: number): number | undefined { const text = value?.trim(); if (!text) return undefined; if (/^\d+(?:\.\d+)?$/.test(text)) { const seconds = Number(text); if (Number.isFinite(seconds) && seconds > 0) { - return Math.min(Math.max(Math.ceil(seconds * 1000), 1), MAX_COOLDOWN_MS); + return Math.min(Math.max(Math.ceil(seconds * 1000), 1), MAX_MEASURED_COOLDOWN_MS); } } const timestamp = Date.parse(text); if (!Number.isFinite(timestamp)) return undefined; const delay = timestamp - now; - return delay > 0 ? Math.min(delay, MAX_COOLDOWN_MS) : undefined; + return delay > 0 ? Math.min(delay, MAX_MEASURED_COOLDOWN_MS) : undefined; +} + +/** + * The cooldown implied by the account's own rate-limit headers, when they say a window is + * spent. + * + * Retry-After is not always present on an Anthropic 429, but + * `anthropic-ratelimit-unified-*-status: rejected` plus a `-reset` epoch always is, and it + * names the exact second the window reopens. Only a REJECTED window counts: an `allowed` + * window's reset is just the end of the current period, and cooling an account until then + * would bench a healthy account for hours. + * + * With more than one window rejected the answer is the LATEST reset, not the earliest. The + * limiter is AND-composed -- upstream refuses while ANY window rejects -- so an account whose + * 5-hour bucket rolls in three minutes is still refused for the five days its weekly window + * needs. Cooling to the earliest reset would re-offer that account every three minutes for + * five days, which is the re-offer loop this whole path exists to end. + */ +function parseRateLimitResetMs(headers: AnthropicRateLimitHeaders | null | undefined, now: number): number | undefined { + if (!headers) return undefined; + let latest: number | undefined; + for (const window of ["5h", "7d"] as const) { + if (headers.get(`anthropic-ratelimit-unified-${window}-status`)?.trim() !== "rejected") continue; + const resetSeconds = Number(headers.get(`anthropic-ratelimit-unified-${window}-reset`)?.trim()); + if (!Number.isFinite(resetSeconds) || resetSeconds <= 0) continue; + const resetAt = resetSeconds * 1000; + if (resetAt <= now) continue; + if (latest === undefined || resetAt > latest) latest = resetAt; + } + if (latest === undefined) return undefined; + return Math.min(latest - now, MAX_MEASURED_COOLDOWN_MS); } export function getAnthropicAccountHealthSnapshot( @@ -669,6 +754,7 @@ export function rotateAnthropicAccountOn429( retryAfterHeader: string | null | undefined, sessionKey?: string | null, now = Date.now(), + rateLimitHeaders?: AnthropicRateLimitHeaders | null, ): string | null { // Reactive 429 failover is NOT gated on the pool flag. That flag buys PROACTIVE routing -- // session affinity, quota-ranked new-session selection, autoSwitchThreshold, strategy -- all @@ -678,11 +764,18 @@ export function rotateAnthropicAccountOn429( // Presence is the activation rule, the same one an apiKeyPool of two keys already uses. if (!isAnthropicAccountPoolEnabled(config) && !hasAnthropicFailoverQuorum(now)) return null; + // Retry-After first: it is the header written FOR this decision. The rejected window's + // reset is the fallback, because a 429 that omits Retry-After still carries it -- and + // without that fallback such a refusal cools for the 60s default and the exhausted + // account is back in the rotation a minute later. const parsedRetry = parseRetryAfterMs(retryAfterHeader, now); - const cooldownMs = parsedRetry ?? DEFAULT_COOLDOWN_MS; + const resetDerived = parsedRetry === undefined ? parseRateLimitResetMs(rateLimitHeaders, now) : undefined; + const cooldownMs = parsedRetry ?? resetDerived ?? DEFAULT_COOLDOWN_MS; upstreamHealth.set(failedAccountId, { cooldownUntil: now + cooldownMs, - cooldownSource: parsedRetry ? "retry-after" : "default", + cooldownSource: parsedRetry !== undefined + ? "retry-after" + : resetDerived !== undefined ? "reset-derived" : "default", }); sweepExpiredOnWrite(now); clearAnthropicSessionAffinityForAccount(failedAccountId); diff --git a/src/oauth/health.ts b/src/oauth/health.ts index 4c997c47cc..011ebd8f41 100644 --- a/src/oauth/health.ts +++ b/src/oauth/health.ts @@ -184,6 +184,9 @@ export function projectStoredOAuthAccountHealth( needsReauth: account.needsReauth === true, reauthReason: account.needsReauth === true ? "refresh_failed" : undefined, cooldownUntilMs: anthropicSnap?.cooldownUntil, + // Same mapping as the Codex pool's `cooldownReasonFromSource`: only a Retry-After is + // request-rate throttling. A reset-derived cooldown means a usage window is spent, which + // is quota, and reporting it as a rate limit would tell the operator to retry shortly. cooldownReason: anthropicSnap?.cooldownSource === "retry-after" ? "rate_limit" : anthropicSnap ? "quota" : undefined, warningReason: detectOAuthWarning(provider, account, opts.observeOnly === true, now), now, diff --git a/src/providers/quota.ts b/src/providers/quota.ts index 7136cd3c70..f039442f6a 100644 --- a/src/providers/quota.ts +++ b/src/providers/quota.ts @@ -1642,6 +1642,108 @@ export function setCachedProviderAccountQuotaForTests( accountQuotaCache.set(key, { ts: Date.now(), quota }); } +/** + * Anthropic's unified rate-limit headers, present on EVERY `/v1/messages` response. + * + * The account's own five-hour and seven-day headroom rides along with the answer, so the + * pool can be told what a turn cost without spending a probe on `/api/oauth/usage`. Two + * wire details differ from every other reader here and are the reason this parser exists + * rather than reusing one: + * + * - `utilization` is a FRACTION (`0.74`), while `ProviderQuota.*Percent` is 0..100 and + * `fetchAnthropicUsageQuota` already reads the probe's `74.0` as a percent. Passing the + * header value through `normalizePercent` unscaled would file a 74%-spent account as + * 0.74% and make the emptiest account look like the freshest. + * - `reset` is epoch SECONDS; `normalizeResetAt` promotes a sub-1e10 number to + * milliseconds, so it is fed the raw value on purpose. + * + * Returns null when neither window parses, so a header set that Anthropic renames or drops + * degrades to "no measurement" — the same state as before the observation existed. + */ +export function parseAnthropicRateLimitHeaders(headers: Headers): ProviderQuota | null { + const fiveHourPercent = normalizeUtilizationFraction(headers.get("anthropic-ratelimit-unified-5h-utilization")); + const weeklyPercent = normalizeUtilizationFraction(headers.get("anthropic-ratelimit-unified-7d-utilization")); + if (fiveHourPercent === undefined && weeklyPercent === undefined) return null; + const fiveHourResetAt = normalizeResetAt(headers.get("anthropic-ratelimit-unified-5h-reset")); + const weeklyResetAt = normalizeResetAt(headers.get("anthropic-ratelimit-unified-7d-reset")); + return { + ...(fiveHourPercent !== undefined ? { fiveHourPercent } : {}), + ...(fiveHourResetAt !== undefined ? { fiveHourResetAt } : {}), + ...(weeklyPercent !== undefined ? { weeklyPercent } : {}), + ...(weeklyResetAt !== undefined ? { weeklyResetAt } : {}), + updatedAt: Date.now(), + }; +} + +/** + * A 0..1 utilization fraction as a 0..100 percent. + * + * Anthropic sends `1.0` for a fully drained window, so the guard is `> 1` rather than + * `>= 1`: a value above one is a wire change (or a percent that leaked into a fraction + * field), and inventing `100` from it would cool an account on a misread. Reject instead -- + * an unmeasured account already has a defined meaning here. + * + * Rounded to two decimals because `fraction * 100` is lossy in binary floating point: 0.29 + * yields 28.999999999999996, and the CLI renderers interpolate the percent raw + * (`account.ts`, `account-extended.ts`) rather than rounding at the edge like the GUI does. + * The value is also persisted, so an artifact would survive restarts. + */ +function normalizeUtilizationFraction(value: string | null): number | undefined { + const numeric = toFiniteNumber(value); + if (numeric === undefined || numeric < 0 || numeric > 1) return undefined; + return Math.round(numeric * 10_000) / 100; +} + +/** + * File rate-limit headers observed on a live Anthropic turn against the account that served it. + * + * Modelled on Codex's `applyAccountQuotaFromUpstreamHeaders`: a pure side effect, fail-soft on + * anything unparseable, and fenced by a `writerGeneration` the CALLER captured when it resolved + * the serving credential — not here at write time, because a streaming turn is a long await and + * a generation read at the end cannot see an account change from the start of the same turn. + * + * Unlike `recordPassiveAccountQuota` this is not gated on `hasPassiveAccountQuota`: Anthropic + * does publish a usage endpoint and stays a probe provider. The headers are a free refresh + * BETWEEN probes, never a replacement, and two rules keep it that way: + * + * - The write MERGES over the existing row. The headers report exactly two windows; the probe + * also returns `customWindows` with the model-scoped weekly limits (Opus, Sonnet, Fable), + * which nothing on the wire carries. A wholesale replace would delete those bars, and they + * are read for real: by the manual-preference exhaustion check in `anthropic-routing.ts` and + * by `headroomOf` in `account-quota-rank.ts`, not only by the dashboard. + * - The entry's `ts` is NOT advanced. `fetchAccountQuota` gates re-probing on that timestamp + * (`Date.now() - cached.ts < ACCOUNT_QUOTA_TTL_MS`), so refreshing it on every turn would + * silence the probe entirely for any account used more than once per ten minutes: the + * observation would both narrow the row and disable the only thing that could widen it again. + * The quota's own `updatedAt` still moves, because the numbers it carries really are fresh. + */ +export function recordAnthropicAccountQuotaFromHeaders( + accountId: string, + headers: Headers, + writerGeneration: number, +): void { + if (!accountId) return; + const observed = parseAnthropicRateLimitHeaders(headers); + if (!observed) return; + const key = accountCacheKey("anthropic", accountId); + if (!mayCommitAccountQuotaKey(key, writerGeneration)) return; + // Hydrate before writing, for the same reason `recordPassiveAccountQuota` does: this write + // arrives unprompted from the request path, and `persistAccountQuotaCache` serializes the + // whole map. Landing before any reader has hydrated would persist this single row and erase + // every other provider's saved row. + hydrateAccountQuotaCache(); + const previous = accountQuotaCache.get(key); + accountQuotaCache.set(key, { + ...previous, + // A row that held only a failed probe (`quota: null`, `unavailable`) is now measured, so + // that failure flag must not survive the observation which replaced it. + unavailable: undefined, + ts: previous?.ts ?? Date.now(), + quota: { ...(previous?.quota ?? {}), ...observed }, + }); + persistAccountQuotaCache(); +} + /** * Providers whose per-account quota is OBSERVED in-band, never probed. * diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index e87136b67a..47861f1e6d 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -230,7 +230,7 @@ import { } from "../../providers/request-pacing"; import { slugsEquivalent } from "../../providers/slug-codec"; import { isMuseSubscriptionUsagePayload, parseMuseSubscriptionUsage } from "../../providers/muse-subscription-usage"; -import { hasPassiveAccountQuota, recordPassiveAccountQuota } from "../../providers/quota"; +import { hasPassiveAccountQuota, recordAnthropicAccountQuotaFromHeaders, recordPassiveAccountQuota } from "../../providers/quota"; import { captureConfigGeneration } from "../../lib/state-store-sweeper"; import { applyOpenAiVirtualModel, resolveOpenAiCompactModel } from "../../providers/openai-virtual-models"; import { isUsageDebugEnabled } from "../../usage/debug"; @@ -3697,6 +3697,16 @@ async function handleResponsesInner( let replayOAuthCredentialSnapshot: Pick | undefined; let anthropicPoolAccountId: string | null = null; let anthropicPoolFailovers = 0; + /** + * Generation fence for the Anthropic rate-limit headers observed on this turn. + * + * Captured where the serving credential is RESOLVED, for the same reason + * `passiveQuotaWriterGeneration` is: a turn is a long await, and a generation read at + * write time cannot see a config or account change that happened earlier in it. Stays 0 + * for every provider that is not Anthropic OAuth, which the observer treats as "no + * account to attribute" and skips. + */ + let anthropicQuotaWriterGeneration = 0; // Generic OAuth rotation (#2568) for providers with no pool of their own. Bound to the account // the request actually used, so a concurrent rotation cannot cool an innocent replacement. let genericFailoverAccountId: string | null = null; @@ -3838,6 +3848,7 @@ async function handleResponsesInner( stampOAuthAccountLabel(logCtx, route.providerName, route.provider, snapshot.accountId); if (route.providerName === "anthropic") { anthropicPoolAccountId = snapshot.accountId; + anthropicQuotaWriterGeneration = captureConfigGeneration(); logCtx.provider = formatAnthropicProviderForLog("anthropic", snapshot.accountId, config); } else { genericFailoverAccountId = snapshot.accountId; @@ -3979,6 +3990,20 @@ async function handleResponsesInner( throw new Error("OAuth account selection changed repeatedly before dispatch"); }; }; + /** + * File Anthropic's in-band rate-limit headers against the account that served the turn. + * + * A no-op without an account id, which is the honest answer for an API-key provider or a + * single-account OAuth install below failover quorum: the headers describe a subscription + * the pool has no row for. Never throws -- a quota measurement must not be able to fail a + * turn that already succeeded. + */ + const observeAnthropicRateLimitHeaders = (accountId: string | null, headers: Headers): void => { + if (route.providerName !== "anthropic" || !accountId) return; + try { + recordAnthropicAccountQuotaFromHeaders(accountId, headers, anthropicQuotaWriterGeneration); + } catch { /* best-effort observation */ } + }; const anthropicSessionKey = route.providerName === "anthropic" && route.provider.authMode === "oauth" ? anthropicSessionKeyFromParts({ sessionIdHeader: sessionIdHeaderFromRequest(req.headers), @@ -4007,6 +4032,7 @@ async function handleResponsesInner( const admitted = await commitResolvedOAuthSelection(await getAnthropicPoolAccessSnapshot(selection.accountId), true, selection.reason); if (!admitted) return formatErrorResponse(409, "conflict_error", "OAuth account selection changed; retry the request"); anthropicPoolAccountId = admitted.accountId; + anthropicQuotaWriterGeneration = captureConfigGeneration(); route.provider = { ...route.provider, apiKey: admitted.accessToken }; logCtx.provider = formatAnthropicProviderForLog("anthropic", admitted.accountId, config); } else { @@ -4083,6 +4109,7 @@ async function handleResponsesInner( // are proactive and stay behind anthropicAccountPool.enabled. if (route.providerName === "anthropic" && hasAnthropicFailoverQuorum()) { anthropicPoolAccountId = resolved.accountId; + anthropicQuotaWriterGeneration = captureConfigGeneration(); } // Captured beside the account it fences, so the two can never disagree. if (hasPassiveAccountQuota(route.providerName)) { @@ -5960,7 +5987,15 @@ async function handleResponsesInner( const imgPlan = !routedCompaction ? await planImageBridge(config, parsed, route.provider) : undefined; const vidPlan = !routedCompaction ? await planVideoBridge(config, parsed, route.provider) : undefined; const canRunWebSearch = !!wsPlan && !adapter.runTurn; - const rotateSidecarProviderOn429 = async (retryAfter: string | null): Promise => { + const rotateSidecarProviderOn429 = async ( + retryAfter: string | null, + responseHeaders?: Headers, + ): Promise => { + // Read BEFORE the rotation rebinds the account: a 429 reports the refusing account's + // utilization too, and `onUpstreamResponse` cannot deliver it because both sidecar loops + // throw on a non-OK response before reaching their success hook. Without this the one + // reading that matters most -- the window that just hit 100% -- is the one never recorded. + if (responseHeaders) observeAnthropicRateLimitHeaders(anthropicPoolAccountId, responseHeaders); const rotated = rotateProviderTransportOn429(config, route.providerName, route.provider, { retryAfter, now: Date.now(), @@ -6004,6 +6039,8 @@ async function handleResponsesInner( anthropicPoolAccountId, retryAfter, anthropicSessionKey, + Date.now(), + responseHeaders, ); if (!nextAccountId) return null; try { @@ -6015,6 +6052,8 @@ async function handleResponsesInner( const admitted = await commitResolvedOAuthSelection(await getAnthropicPoolAccessSnapshot(nextAccountId)); if (!admitted) throw new Error("OAuth selection changed during recovery"); anthropicPoolAccountId = admitted.accountId; + // Re-captured with the rebind, for the same reason as the main recovery loop. + anthropicQuotaWriterGeneration = captureConfigGeneration(); anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: admitted.accessToken }; logCtx.provider = formatAnthropicProviderForLog("anthropic", admitted.accountId, config); @@ -6136,6 +6175,10 @@ async function handleResponsesInner( } }, on429: rotateSidecarProviderOn429, + // The sidecar's accepted responses carry the same headroom headers as the main path. + // `anthropicPoolAccountId` is read here, at event time, because the sidecar's own 429 arm + // rebinds it mid-loop. + onUpstreamResponse: headers => observeAnthropicRateLimitHeaders(anthropicPoolAccountId, headers), retryOn429Policy: rateLimitRetryPolicyFor(route.provider), ...(options.onFirstOutput ? { onFirstOutput: options.onFirstOutput } : {}), ...(options.forceEmptyResponseId ? { forceEmptyResponseId: true } : {}), @@ -6221,6 +6264,10 @@ async function handleResponsesInner( stallTimeoutSec: wsPlan.stallTimeoutSec, streamRoutedModelOutput: wsPlan.streamRoutedModelOutput, on429: rotateSidecarProviderOn429, + // The sidecar's accepted responses carry the same headroom headers as the main path. + // `anthropicPoolAccountId` is read here, at event time, because the sidecar's own 429 arm + // rebinds it mid-loop. + onUpstreamResponse: headers => observeAnthropicRateLimitHeaders(anthropicPoolAccountId, headers), retryOn429Policy: rateLimitRetryPolicyFor(route.provider), onCompletedResponse: commitReasoningReplayServingRoute, }); @@ -7036,6 +7083,8 @@ async function handleResponsesInner( anthropicPoolAccountId, upstreamResponse.headers.get("retry-after"), anthropicSessionKey, + Date.now(), + upstreamResponse.headers, ); if (!nextAccountId) break; try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } @@ -7043,6 +7092,11 @@ async function handleResponsesInner( const admitted = await commitResolvedOAuthSelection(await getAnthropicPoolAccessSnapshot(nextAccountId)); if (!admitted) throw new Error("OAuth selection changed during recovery"); anthropicPoolAccountId = admitted.accountId; + // Re-captured with the rebind, never left at the value taken for the account that + // just 429'd: the fence answers "is this measurement still current for THIS + // account", so carrying a stale generation past a rotation can refuse the very + // observation the rotation exists to produce. + anthropicQuotaWriterGeneration = captureConfigGeneration(); anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: admitted.accessToken }; invalidateSameTargetRequest(); @@ -7224,6 +7278,15 @@ async function handleResponsesInner( cancelBodyOnAbort(upstreamResponse.body, upstream.signal); + // Anthropic reports the serving account's five-hour and seven-day headroom on EVERY + // `/v1/messages` response, so the only quota measurement the pool has stops being a + // periodic `/api/oauth/usage` probe of whichever account happens to be active. Read here, + // after every recovery arm has settled, so the numbers are attributed to the account that + // actually served the turn rather than one the request rotated away from. Reading + // `anthropicPoolAccountId` live (not a value captured before dispatch) is load-bearing for + // the same reason: all three 429 arms rebind it mid-request. + observeAnthropicRateLimitHeaders(anthropicPoolAccountId, upstreamResponse.headers); + // One bounded internal continuation re-ask for clean end_turn turns that announced an edit // without emitting a tool call. Anthropic gets this by default; openai-chat providers opt in // per-provider via `terminalContinuationGuard` (the heuristic was tuned on Anthropic turns, @@ -7449,6 +7512,8 @@ async function handleResponsesInner( anthropicPoolAccountId, response.headers.get("retry-after"), anthropicSessionKey, + Date.now(), + response.headers, ); if (nextAccountId) { try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ } @@ -7456,6 +7521,8 @@ async function handleResponsesInner( const admitted = await commitResolvedOAuthSelection(await getAnthropicPoolAccessSnapshot(nextAccountId)); if (!admitted) throw new Error("OAuth selection changed during recovery"); anthropicPoolAccountId = admitted.accountId; + // Re-captured with the rebind, for the same reason as the main recovery loop. + anthropicQuotaWriterGeneration = captureConfigGeneration(); anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: admitted.accessToken }; invalidateSameTargetRequest(); @@ -7551,6 +7618,12 @@ async function handleResponsesInner( return; } + // A continuation is a second billed Anthropic call with its own fresh headers, and it + // never passes the observation above. Without this the numbers recorded for a + // terminal-guard turn are those of the first leg only -- and the guard is on by default + // for Anthropic, so that is the common case, not an edge one. + observeAnthropicRateLimitHeaders(anthropicPoolAccountId, response.headers); + try { // Protect the continuation body against a client abort landing between fetch resolution and // reader attach, exactly as the initial response is guarded above (#390/366e3053). Without diff --git a/src/web-search/loop.ts b/src/web-search/loop.ts index 3a2c5e99b4..2688e07429 100644 --- a/src/web-search/loop.ts +++ b/src/web-search/loop.ts @@ -309,8 +309,26 @@ export interface WebSearchLoopDeps { * 429 failover hook: rotate the provider's active credential and return a rebuilt adapter, * or null when the pool is exhausted. Async hooks support OAuth refresh; existing synchronous * key-pool hooks remain valid. + * + * `responseHeaders` carries the whole refusal, not just Retry-After, because an Anthropic + * 429 states the window's reset epoch even when it omits Retry-After -- and a rotation that + * cannot see it cools the drained account for the short default instead of until the window + * actually reopens. Optional so existing callers keep compiling. */ - on429?: (retryAfterHeader: string | null) => ProviderAdapter | null | Promise; + on429?: ( + retryAfterHeader: string | null, + responseHeaders?: Headers, + ) => ProviderAdapter | null | Promise; + /** + * Called with the headers of each upstream response the loop accepts. + * + * A sidecar iteration is a billed provider call like any other, and Anthropic reports the + * serving account's remaining headroom on every one of them. Without this seam the caller + * only ever hears about the refusals (`on429`), so a workload that runs mostly through the + * sidecar contributes no quota measurement at all -- the divergence between sidecar and main + * path that the 429 arm already had to be fixed for once. + */ + onUpstreamResponse?: (responseHeaders: Headers) => void; /** Opt-in same-target 429 policy (key-auth providers). When present, 429 replays on the SAME key before on429 rotation. */ retryOn429Policy?: Required | null; /** Called only when the final bridged Responses stream reaches completed or incomplete. */ @@ -521,7 +539,7 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise { + home = mkdtempSync(join(tmpdir(), "ocx-anthropic-ratelimit-")); + process.env.OPENCODEX_HOME = home; + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + clearAccountQuotaCache(); + // `lastReconciledGeneration` is module-global and survives a cache clear, so the fence case + // below would otherwise raise the floor for every test that runs after it in this file. + resetProviderQuotaReconcileStateForTests(); + forgetAnthropicFailoverQuorum(); +}); + +afterEach(() => { + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + // The argument-less form, deliberately: only it calls cancelPendingAccountQuotaPersist. + // The observer ends in a 250ms-debounced write that resolves OPENCODEX_HOME at fire time, + // so a provider-scoped clear would leave that write to land in whatever home is current a + // quarter second later — the next test's sandbox, or the developer's real one. + clearAccountQuotaCache(); + resetProviderQuotaReconcileStateForTests(); + forgetAnthropicFailoverQuorum(); + if (originalHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = originalHome; + removeTreeWithRetry(home); +}); + +/** The store assigns its own slot ids, so the seeded `accountId` is never the cache key. */ +async function seed(count: number): Promise { + for (let i = 0; i < count; i++) { + await saveCredential("anthropic", { + access: `access-${i}`, + refresh: `refresh-${i}`, + expires: Date.now() + 3_600_000, + accountId: `uuid-${i}`, + email: `user${i}@example.test`, + } as never); + } + return getAccountSet("anthropic")?.accounts.map(a => a.id) ?? []; +} + +function poolEnabled(): OcxConfig { + return { + port: 0, + defaultProvider: "anthropic", + providers: { + anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth" }, + }, + anthropicAccountPool: { enabled: true }, + } as OcxConfig; +} + +/** A real 429 from a drained five-hour window, captured from api.anthropic.com. */ +function drainedFiveHour(resetEpochSeconds: number): Headers { + return new Headers({ + "anthropic-ratelimit-unified-status": "rejected", + "anthropic-ratelimit-unified-5h-status": "rejected", + "anthropic-ratelimit-unified-5h-reset": String(resetEpochSeconds), + "anthropic-ratelimit-unified-5h-utilization": "1.0", + "anthropic-ratelimit-unified-7d-status": "allowed", + "anthropic-ratelimit-unified-7d-reset": String(resetEpochSeconds + 86_400), + "anthropic-ratelimit-unified-7d-utilization": "0.36", + }); +} + +describe("Anthropic cooldown honours the stated window", () => { + test("a multi-hour Retry-After is not truncated to the guessed-backoff ceiling", async () => { + const start = Date.now(); + const ids = await seed(2); + // 7999s is what a drained five-hour window actually answers; the old 15-minute clamp + // turned a single refusal into sixteen wasted retries before the window reopened. + rotateAnthropicAccountOn429(poolEnabled(), ids[0]!, "7999", null, start); + const health = getAnthropicAccountHealthSnapshot(ids[0]!, start); + expect(health?.cooldownUntil).toBe(start + 7_999_000); + expect(health?.cooldownSource).toBe("retry-after"); + }); + + test("an absurd Retry-After is still bounded", async () => { + const start = Date.now(); + const ids = await seed(2); + // The ceiling did not disappear, it moved: a stated reset is trusted up to the longest + // window Anthropic publishes, so a wire anomaly cannot bench an account for a week. + rotateAnthropicAccountOn429(poolEnabled(), ids[0]!, "604800", null, start); + expect(getAnthropicAccountHealthSnapshot(ids[0]!, start)?.cooldownUntil) + .toBe(start + 6 * 60 * 60_000); + }); + + test("an HTTP-date Retry-After is honoured, and bounded by the same ceiling", async () => { + const start = Date.now(); + const ids = await seed(2); + // RFC 9110 allows either form, and both are upstream STATING when it will serve again -- + // the date branch had its own clamp and would have kept the 15-minute truncation. + rotateAnthropicAccountOn429(poolEnabled(), ids[0]!, new Date(start + 2 * 60 * 60_000).toUTCString(), null, start); + const cooldown = getAnthropicAccountHealthSnapshot(ids[0]!, start)?.cooldownUntil; + // toUTCString drops sub-second precision, so the deadline lands within a second of target. + expect(cooldown).toBeGreaterThan(start + 2 * 60 * 60_000 - 1_000); + expect(cooldown).toBeLessThanOrEqual(start + 2 * 60 * 60_000); + + rotateAnthropicAccountOn429(poolEnabled(), ids[1]!, new Date(start + 48 * 60 * 60_000).toUTCString(), null, start); + expect(getAnthropicAccountHealthSnapshot(ids[1]!, start)?.cooldownUntil).toBe(start + 6 * 60 * 60_000); + }); + + test("a 429 without Retry-After cools until the rejected window reopens", async () => { + const start = Date.now(); + const ids = await seed(2); + // The wire carries whole seconds, so the reset is built from an epoch second and the + // expectation is derived from the same value rather than from `start + 90min` — an + // assertion on the un-truncated millisecond would be testing the fixture, not the code. + const resetEpochSeconds = Math.floor((start + 90 * 60_000) / 1000); + // Retry-After is not guaranteed on an Anthropic 429; the rejected window's reset is. + // Without reading it this refusal cooled for the 60s default and the drained account + // was back in the rotation a minute later. + rotateAnthropicAccountOn429(poolEnabled(), ids[0]!, null, null, start, drainedFiveHour(resetEpochSeconds)); + const health = getAnthropicAccountHealthSnapshot(ids[0]!, start); + expect(health?.cooldownUntil).toBe(resetEpochSeconds * 1000); + // Its own source, not "retry-after": the dashboard renders that one as request-rate + // throttling, and a spent five-hour window is quota. Same vocabulary the Codex pool uses. + expect(health?.cooldownSource).toBe("reset-derived"); + }); + + test("an ALLOWED window's reset never cools the account", async () => { + const start = Date.now(); + const ids = await seed(2); + // Every response names when the current period ends, including a healthy one. Treating + // that as a cooldown would bench an account with 4% used for the rest of its window. + const healthy = new Headers({ + "anthropic-ratelimit-unified-status": "allowed", + "anthropic-ratelimit-unified-5h-status": "allowed", + "anthropic-ratelimit-unified-5h-reset": String(Math.floor((start + 3 * 60 * 60_000) / 1000)), + "anthropic-ratelimit-unified-5h-utilization": "0.04", + }); + rotateAnthropicAccountOn429(poolEnabled(), ids[0]!, null, null, start, healthy); + const health = getAnthropicAccountHealthSnapshot(ids[0]!, start); + expect(health?.cooldownUntil).toBe(start + 60_000); + expect(health?.cooldownSource).toBe("default"); + }); + + test("both windows rejected cools until the LAST one reopens", async () => { + const start = Date.now(); + const ids = await seed(2); + // The limiter is AND-composed: upstream refuses while ANY window rejects. An account whose + // 5-hour bucket rolls in three minutes is still refused for the days its weekly window + // needs, so cooling to the earliest reset would re-offer it every three minutes until the + // weekly window finally reopens -- the exact loop this path exists to end. + const fiveHourReset = Math.floor((start + 3 * 60_000) / 1000); + const weeklyReset = Math.floor((start + 5 * 60 * 60_000) / 1000); + const bothDrained = new Headers({ + "anthropic-ratelimit-unified-status": "rejected", + "anthropic-ratelimit-unified-5h-status": "rejected", + "anthropic-ratelimit-unified-5h-reset": String(fiveHourReset), + "anthropic-ratelimit-unified-7d-status": "rejected", + "anthropic-ratelimit-unified-7d-reset": String(weeklyReset), + }); + rotateAnthropicAccountOn429(poolEnabled(), ids[0]!, null, null, start, bothDrained); + expect(getAnthropicAccountHealthSnapshot(ids[0]!, start)?.cooldownUntil).toBe(weeklyReset * 1000); + }); + + test("a reset-derived cooldown surfaces as quota, a Retry-After as a rate limit", async () => { + const start = Date.now(); + const ids = await seed(2); + const account = getAccountSet("anthropic")!.accounts.find(a => a.id === ids[0]!)!; + // The distinction is not cosmetic: the dashboard tells an operator to wait out a rate + // limit and to switch accounts on spent quota. A drained five-hour window is the second. + rotateAnthropicAccountOn429( + poolEnabled(), + ids[0]!, + null, + null, + start, + drainedFiveHour(Math.floor((start + 90 * 60_000) / 1000)), + ); + expect(projectStoredOAuthAccountHealth("anthropic", account, start)).toMatchObject({ + status: "cooldown", + reason: "quota", + }); + + clearAnthropicAccountCooldown(ids[0]!); + rotateAnthropicAccountOn429(poolEnabled(), ids[0]!, "300", null, start); + expect(projectStoredOAuthAccountHealth("anthropic", account, start)).toMatchObject({ + status: "cooldown", + reason: "rate_limit", + }); + }); + + test("Retry-After wins over the header reset", async () => { + const start = Date.now(); + const ids = await seed(2); + // Retry-After is written for this decision; the reset epoch is a fallback for the + // refusals that omit it. A disagreement must not silently prefer the fallback. + rotateAnthropicAccountOn429( + poolEnabled(), + ids[0]!, + "120", + null, + start, + drainedFiveHour(Math.floor((start + 4 * 60 * 60_000) / 1000)), + ); + expect(getAnthropicAccountHealthSnapshot(ids[0]!, start)?.cooldownUntil).toBe(start + 120_000); + }); +}); + +describe("Anthropic rate-limit headers feed the routing cache", () => { + test("utilization is read as a fraction, not as a percent", () => { + // The header sends 0.74 for a 74%-spent window while the probe endpoint sends 74.0 for + // the same account. Passing the header value through unscaled would file the emptiest + // account as the freshest and route every new session straight at it. + const quota = parseAnthropicRateLimitHeaders(new Headers({ + "anthropic-ratelimit-unified-5h-utilization": "0.42", + "anthropic-ratelimit-unified-7d-utilization": "0.74", + })); + expect(quota?.fiveHourPercent).toBe(42); + expect(quota?.weeklyPercent).toBe(74); + }); + + test("reset epochs are promoted from seconds to milliseconds", () => { + const quota = parseAnthropicRateLimitHeaders(new Headers({ + "anthropic-ratelimit-unified-5h-utilization": "0.5", + "anthropic-ratelimit-unified-5h-reset": "1788717000", + })); + expect(quota?.fiveHourResetAt).toBe(1_788_717_000_000); + }); + + test("a header set with no utilization yields no measurement", () => { + // A renamed or dropped header must degrade to "unmeasured", which the router already + // has a defined behaviour for -- never to a fabricated zero, which reads as a fresh + // account and would pull traffic toward whichever account stopped reporting. + expect(parseAnthropicRateLimitHeaders(new Headers({ + "anthropic-ratelimit-unified-5h-reset": "1788717000", + }))).toBeNull(); + }); + + test("a utilization above 1 is rejected rather than clamped", () => { + // Above one is a wire change, not a full window. Inventing 100 from it would cool a + // healthy account on a misread. + expect(parseAnthropicRateLimitHeaders(new Headers({ + "anthropic-ratelimit-unified-5h-utilization": "42", + }))).toBeNull(); + }); + + test("an observed turn makes the serving account's usage known to the router", async () => { + const ids = await seed(2); + // Before the observation the account has no reading at all, which is what left a + // two-account pool scoring both at UNKNOWN_USAGE_SCORE and picking between them blind. + expect(getCachedProviderAccountQuota("anthropic", ids[0]!)).toBeNull(); + recordAnthropicAccountQuotaFromHeaders(ids[0]!, drainedFiveHour(Math.floor(Date.now() / 1000) + 3600), 0); + expect(getCachedProviderAccountQuota("anthropic", ids[0]!)?.fiveHourPercent).toBe(100); + // The other account stays unmeasured: an observation is attributed to the account that + // served the turn, never spread across the roster. + expect(getCachedProviderAccountQuota("anthropic", ids[1]!)).toBeNull(); + }); + + test("headers with nothing parseable leave the previous reading intact", async () => { + const ids = await seed(1); + recordAnthropicAccountQuotaFromHeaders(ids[0]!, new Headers({ + "anthropic-ratelimit-unified-5h-utilization": "0.25", + }), 0); + recordAnthropicAccountQuotaFromHeaders(ids[0]!, new Headers({ "content-type": "application/json" }), 0); + // A response that says nothing about quota is not evidence that the quota is gone. + expect(getCachedProviderAccountQuota("anthropic", ids[0]!)?.fiveHourPercent).toBe(25); + }); + + test("an empty account id writes nothing", () => { + // API-key providers and single-account installs below failover quorum reach the observer + // with no account to attribute; that is an ordinary state, not an error. Asserting only + // that it does not throw would pass with the guard deleted -- an empty-string cache key + // is perfectly writable -- so this asserts the absence of the row instead. + recordAnthropicAccountQuotaFromHeaders("", drainedFiveHour(Math.floor(Date.now() / 1000) + 3600), 0); + expect(getCachedProviderAccountQuota("anthropic", "")).toBeNull(); + }); + + test("a stale writer generation is refused", async () => { + const ids = await seed(1); + // The fence exists because a turn is a long await: an account or config change that lands + // mid-turn must not be overwritten by a measurement taken before it. Every other test here + // passes 0, which a fresh worker always accepts, so without this case the parameter is + // carried but never actually exercised as a fence. + reconcileProviderAccountQuotaRows({ + generation: 5, + providerNames: new Set(), + comboIds: new Set(), + comboTargets: new Set(), + codexAccountIds: new Set(), + oauthAccountKeys: new Set(), + configRoots: new Set(), + }); + recordAnthropicAccountQuotaFromHeaders(ids[0]!, new Headers({ + "anthropic-ratelimit-unified-5h-utilization": "0.5", + }), 1); + expect(getCachedProviderAccountQuota("anthropic", ids[0]!)).toBeNull(); + }); + + test("an observation keeps the model-scoped bars the probe filled", async () => { + const ids = await seed(1); + // The probe reports per-model weekly limits (Opus, Sonnet, Fable) that no header carries. + // They are read by the manual-preference exhaustion check and by `headroomOf`, so a + // wholesale replace would not merely blank the dashboard: it would route an Opus request + // to an account whose Opus allowance is spent. + setCachedProviderAccountQuotaForTests("anthropic", ids[0]!, { + fiveHourPercent: 10, + weeklyPercent: 20, + customWindows: [{ label: "Opus", percent: 96 }], + updatedAt: Date.now(), + }); + recordAnthropicAccountQuotaFromHeaders(ids[0]!, new Headers({ + "anthropic-ratelimit-unified-5h-utilization": "0.41", + }), 0); + const quota = getCachedProviderAccountQuota("anthropic", ids[0]!); + expect(quota?.fiveHourPercent).toBe(41); + // Untouched by this observation, not erased by it. + expect(quota?.weeklyPercent).toBe(20); + expect(quota?.customWindows).toEqual([{ label: "Opus", percent: 96 }]); + }); + + test("a percent that is not exactly representable is rounded, not left as an artifact", () => { + // `0.29 * 100` is 28.999999999999996 in binary floating point, and the CLI interpolates the + // percent raw. A user reading `5h 28.999999999999996%` would reasonably file a bug. + expect(parseAnthropicRateLimitHeaders(new Headers({ + "anthropic-ratelimit-unified-5h-utilization": "0.29", + }))?.fiveHourPercent).toBe(29); + }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 75542ebac2..84ffc907d8 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -39,6 +39,7 @@ "anthropic-image-retry.test.ts": "adapters/anthropic", "anthropic-pool-toggle-copy.test.ts": "adapters/anthropic", "anthropic-quorum-cache.test.ts": "routing", + "anthropic-ratelimit-headers.test.ts": "adapters/anthropic", "anthropic-reasoning.test.ts": "adapters/anthropic", "anthropic-sidecar-account-failover.test.ts": "adapters/anthropic", "anthropic-stream-hardening.test.ts": "adapters/anthropic", diff --git a/tests/update/update-stop-first.test.ts b/tests/update/update-stop-first.test.ts index f52c53a008..5dc299eade 100644 --- a/tests/update/update-stop-first.test.ts +++ b/tests/update/update-stop-first.test.ts @@ -630,7 +630,15 @@ describe("update stops the running proxy before replacing files", () => { // Bind the actual child environment, not merely HOME, to this case. expect(resolveCodexHomeDir({ env })).toBe(join(root, ".codex")); const port = await freePort(); - expect(existsSync(bundledBun)).toBe(true); + // A precondition, not the assertion: the recovery path under test hands the launcher + // the bundled Bun, so without it the case cannot run at all. `repoRoot` resolves to the + // checkout being tested, which in a git worktree is NOT the primary checkout -- a + // worktree that was never `bun install`ed fails here with a bare `expected true`, + // reading like a defect in the code under test rather than missing setup. + expect( + existsSync(bundledBun), + `${bundledBun} is missing: run \`bun install\` in this checkout (each git worktree needs its own).`, + ).toBe(true); mkdirSync(dirname(launcher), { recursive: true }); mkdirSync(join(packageRoot, "node_modules"), { recursive: true }); mkdirSync(opencodexHome, { recursive: true });