[Bug]: Reset-derived cooldowns can miss early recovery while another pool account remains eligible

[luvs01]

Client or integration

Codex App

Area

Authentication and account pool

Summary

A reset-derived Codex cooldown can remain after the account recovers early when another pool account remains eligible.

An actual upstream 429 with reset headers creates a non-persistent cooldown. The cooled account is excluded, and the current recovery probe is acquired only after routing selects that account. If another account can always serve traffic, the cooled account is never selected, so it never receives the bounded probe. A forced fresh WHAM read may show recovery, but updating quota cache does not clear the matching routing cooldown.

Persistently pausing an account at displayed 100% is not a valid workaround: OpenAI can reset earlier than the predicted timestamp, 100% can remain advisory while an in-progress task still succeeds, and pause can outlive the restriction.

Expected behavior:

• Create cooldown only after an actual upstream rejection; a WHAM 100% snapshot alone must not create cooldown, pause, or move an existing thread.
• Keep reset-derived exclusion non-persistent.
• At a bounded interval, permit one single-flight fresh quota refresh for a due reset-derived candidate even while another account serves requests.
• Fence by account identity, credential generation, quota scope, and cooldown generation.
• Clear only the matching reset-derived scope/generation when a complete fresh authoritative snapshot proves recovery.
• Retain cooldown on still-100%, incomplete, stale, unavailable, unrelated-scope, or replaced-credential results.
• Do not clear explicit Retry-After or headerless/default cooldowns through this path.
• Fail closed without delaying ordinary traffic on the healthy alternate.

Relevant source:

• src/codex/routing.ts: reset-derived health, generations, scope, selection, and probe leases.
• src/codex/auth-context.ts: probe admission currently occurs only after selection.
• src/codex/auth-api.ts: main/pool forced refresh and existing single-flight.
• src/codex/quota.ts: freshness and authoritative exhaustion.

Required tests: early recovery with another eligible account; still-exhausted/unknown/fetch-failure cases; concurrent single-flight; stale-generation rejection; main and pool accounts; shared vs Spark scope separation; explicit Retry-After untouched; all pool strategies re-include the recovered account; paused unchanged; and no cooldown/rebinding from a standalone 100% snapshot.

Related: #433 (bounded probe/cap), #590/#599 (scoped cooldowns), #584/#585 (same-request failover after real rejection), and #657/#866 (separate reset-credit semantics).

Reproduction

1. Configure accounts A and B.
2. Let A receive an actual upstream 429 with reset headers.
3. Keep B eligible so traffic continues through B.
4. Let OpenAI reset A earlier than the recorded deadline.
5. Force a fresh authoritative quota read while continuing traffic through B.

Current: A's quota cache updates, but its routing cooldown remains because A is never selected for the existing probe.

Expected: one bounded refresh clears only A's matching reset-derived generation, A becomes eligible immediately, and no persistent pause or percentage-only thread move occurs.

Version

2.10.0; the relevant routing and quota-refresh behavior remained on dev at 075c2f34c060d068ed56985e76924a9f23aca39d when checked on 2026-08-03.

Operating system

Windows 11 Pro 10.0.26200 (platform-independent routing gap).

Provider and model

OpenAI (Codex login / forward account pool); native Codex models and confirmed quota scopes.

Logs or error output

Selected Codex account is cooling down

Screenshots and supporting files

None. Exact quota values, reset times, and account identifiers are omitted.

Redacted configuration

{
  "providers": {
    "openai": {
      "adapter": "openai-responses",
      "authMode": "forward",
      "codexAccountMode": "pool"
    }
  },
  "accountPoolStrategy": "fill-first",
  "autoSwitchThreshold": 100,
  "activeCodexAccountId": "__main__"
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[lidge-jun]

Triaged against current dev. Real, and it needs more than a guard — recording the analysis so the fix does not start from scratch.

The mechanism you describe is exact. Cooled accounts are filtered out before all strategy selection (src/codex/routing.ts:734-760), and the alternate selectors only ever see the eligible list (:927-965). Probe eligibility does exist and is correctly limited to non-Retry-After cooldowns with one lease per interval (:369-423) — but resolveCodexAuthContext() selects the account first (src/codex/auth-context.ts:214-226) and only then checks that selected account's lease (:237-253). So with B eligible, A is never selected and never reaches the lease code. Selection-before-probe, exactly as you put it.

Your point about WHAM is right too: pool results call setAccountQuotaFromParsed() (src/codex/auth-api.ts:590-626), which writes the quota cache (src/codex/quota.ts:134-179) and has no authority over routing cooldown generations.

Why this is scheduled rather than patched now: the fix is a background recovery worker with a claim/settle contract fenced on cooldown generation, quota scope, and credential generation — not a change to account selection. Two constraints already established, so nobody rediscovers them:

• clearCodexAccountCooldown() must not be used; it clears every scope and carries no credential fence.
• The per-account quota refresh at src/codex/auth-api.ts:646-688 is already generation-aware and single-flight, so it is the thing to join rather than duplicate.

One qualification on the report: "a 100% WHAM snapshot must never move an existing thread under any strategy" is a separate policy change. The quota strategy deliberately rebinds an over-threshold thread (src/codex/routing.ts:1179-1200) and has tests for it. Fill-first and round-robin already preserve affinity. Worth deciding on its own rather than bundling into the recovery fix.

Written up in devlog/_plan/260803_cooldown_recovery_probe/000_plan.md on dev.