Client or integration
Codex App / Codex CLI
Area
Authentication and account pool
Summary
A weekly-exhausted ChatGPT pool account is never rotated away from. Every request fails and the client surfaces stream disconnected before completion: Incomplete response returned, reason: adapter_eof. The proxy is healthy, auto-switch is enabled at threshold 99, and other accounts have full headroom — routing simply never leaves the drained account.
This is the same root cause as #3769. I am filing it separately because I have field evidence from a production pool that (a) answers one of the maintainer's open questions on that PR, and (b) shows a second blast radius — already-running sessions — that the PR's tests do not currently pin.
Three behaviors compose into a state the pool cannot leave:
applyQuotaAutoSwitch → computeCodexUsageScore (src/codex/routing.ts) reads only the persisted quota cache via getAccountQuota. That cache advances on an explicit refresh/probe, never on a serving failure.- An upstream-exhausted account does not answer 429. It opens the stream and terminates mid-stream with
The usage limit has been reached, recorded as terminalStatus: "incomplete".
codexForwardTerminalOutcomeRecorder (src/server/responses/core.ts) treats every incomplete terminal as normal and records outcome 200 — "the account served the request. Don't penalize account health."
The only evidence that the account is drained is discarded at step 3, the cache stays stale at step 1, so hasCodexQuotaHeadroom keeps returning true and applyQuotaAutoSwitch returns active unchanged. No cooldown, soft-avoid, or failover is ever armed.
This differs from #3029 / #3425, which concern the five-hour window and were addressed via isTerminalShortWindow. That path scores an account exhausted from a full short window. The weekly/monthly case has no equivalent: when knownLong is present but stale, computeCodexUsageScore returns the cached value and the terminal evidence never corrects it.
Reproduction
- Run 2.43.0 with several ChatGPT accounts in the Codex pool; enable auto-switch (
quota strategy, threshold 99).
- Drain the active account's weekly window to 100% through normal use.
- Do not run
ocx account refresh — let the cached quota go stale.
- Send any Codex request.
Every request fails and the pool never rotates, even though other accounts have full headroom.
Field evidence for the open question on #3769
The review on #3769 asks whether the incomplete quota signal always carries a message string, or whether incomplete_details.reason: usage_limit_reached can arrive with an empty message — which would slip past the isRateLimitOrQuotaFailureMessage "usage limit" match.
I scanned the recent tail of a production usage.jsonl for every terminalStatus: "incomplete" row carrying a usage-limit signal:
total usage-limit incompletes: 77
message form ("The usage limit has been reached"): 77
reason-only form (empty message): 0
All 77 carried the message string, so the message-based detection in #3769 covers 100% of what this pool observed. That is positive evidence for the current approach rather than a counterexample. It is one pool on canonical ChatGPT forward, so it does not prove the reason-only shape never occurs; if a defensive incomplete_details.reason check is added, this data suggests it would be belt-and-braces rather than the primary path.
Every one of those rows also had usageStatus: "unreported", which may be a useful secondary signal: a terminal that reports no usage at all is a plausible cross-check for "the account did not actually serve this request."
Second blast radius: already-running sessions
The pool wedged again on 2026-09-07 (07:44 and 07:53 KST). This time the recovery action was not the problem — an external watchdog detected the drained account and switched away within seconds, verified. Sessions kept dying anyway.
The reason is thread affinity, and I think it matters for #3769. ocx states the constraint in its own help text:
A selection-order change applies from the next unbound request and never moves a bound thread.
Every already-running session holds an affinity binding to the drained account. ocx account use and ocx account priority act on future unbound selection, so they rescue new sessions while every in-flight session keeps hitting the dead account until it fails.
Two vendor paths would retire such a binding — isCodexAccountSelectable in the affinity branch of resolveCodexAccountForThreadDetailed, and clearThreadAccountMapForAccount in recordCodexUpstreamOutcome — but neither can fire here, for the same root cause: the exhaustion terminal is recorded as outcome 200, so no failure evidence is produced. reevaluateAffinityQuota cannot help either, because it scores the binding from getAccountQuota, the same stale cache.
So the defect has two distinct blast radii:
|
New sessions |
Already-running sessions |
| Before #3769 |
wedged |
wedged |
| With #3769 (429 recorded) |
recovered via cooldown/failover |
recovered if the quota branch also clears affinity |
Reading the quota branch of recordCodexUpstreamOutcome, #3769 does appear to cover this: the quota class reaches clearThreadAccountMapForAccount. I am flagging it explicitly because it is the part that determines whether a user's current work survives, and the PR's new test file covers the 429 capture and cooldown trip rather than the affinity release. A regression case asserting "a thread bound to the drained account rebinds on its next request" would pin the behavior that actually saves in-flight sessions.
Suggestion
#3769 already fixes the core defect and I would rather see it land than see a competing change. Two things seem worth considering on top:
- A quota reading that fails a request should be able to invalidate its own cache entry. Even with the 429 recording, the cached
weeklyPercent stays stale until the next explicit refresh, so hasCodexQuotaHeadroom still reports headroom for a provably drained account. Marking the cached entry stale on a quota terminal would make recovery independent of cooldown timing.
- Operator visibility. In this incident no surface reported the problem:
ocx status was green and the auto-switch setting was correct. Surfacing "active account returned a quota terminal but cached quota shows headroom" in ocx status or the dashboard would turn a multi-hour diagnosis into a glance.
For the released version, the only lever that reaches a bound thread is ocx account pause, because isCodexAccountSelectable checks isCodexAccountPaused first, and the pause route additionally runs clearThreadAccountMapForAccount + selectFallbackAfterPause. That is what I automated locally: pause on detection, resume automatically once the quota window recovers. It covers the native main account too — cmdPause maps the literal main to MAIN_CODEX_ACCOUNT_ID and the route admits it explicitly (verified on 2.43.0). That is a workaround for the released version, not a proposed design.
Version
2.43.0
Operating system
macOS 26.5.2 (arm64)
Provider and model
OpenAI / canonical ChatGPT forward, 5-account Codex pool. Observed on gpt-6-astra, gpt-5.6-sol, and gpt-5.6-luna; the defect is account-pool routing rather than model-specific.
Logs or error output
provider: openai-<label>
model: gpt-6-astra
status: 502
errorCode: upstream_server_error
upstreamError: "The usage limit has been reached"
terminalStatus: incomplete
closeReason: terminal
usageStatus: unreported
43 of these in 15 minutes on one account with no rotation, and 30 more in a later recurrence. ocx status reported a healthy proxy throughout and ocx account auto-switch openai status reported on (threshold 99%).
Checks
Client or integration
Codex App / Codex CLI
Area
Authentication and account pool
Summary
A weekly-exhausted ChatGPT pool account is never rotated away from. Every request fails and the client surfaces
stream disconnected before completion: Incomplete response returned, reason: adapter_eof. The proxy is healthy, auto-switch is enabled at threshold 99, and other accounts have full headroom — routing simply never leaves the drained account.This is the same root cause as #3769. I am filing it separately because I have field evidence from a production pool that (a) answers one of the maintainer's open questions on that PR, and (b) shows a second blast radius — already-running sessions — that the PR's tests do not currently pin.
Three behaviors compose into a state the pool cannot leave:
applyQuotaAutoSwitch→computeCodexUsageScore(src/codex/routing.ts) reads only the persisted quota cache viagetAccountQuota. That cache advances on an explicit refresh/probe, never on a serving failure.The usage limit has been reached, recorded asterminalStatus: "incomplete".codexForwardTerminalOutcomeRecorder(src/server/responses/core.ts) treats everyincompleteterminal as normal and records outcome 200 — "the account served the request. Don't penalize account health."The only evidence that the account is drained is discarded at step 3, the cache stays stale at step 1, so
hasCodexQuotaHeadroomkeeps returning true andapplyQuotaAutoSwitchreturnsactiveunchanged. No cooldown, soft-avoid, or failover is ever armed.This differs from #3029 / #3425, which concern the five-hour window and were addressed via
isTerminalShortWindow. That path scores an account exhausted from a full short window. The weekly/monthly case has no equivalent: whenknownLongis present but stale,computeCodexUsageScorereturns the cached value and the terminal evidence never corrects it.Reproduction
quotastrategy, threshold 99).ocx account refresh— let the cached quota go stale.Every request fails and the pool never rotates, even though other accounts have full headroom.
Field evidence for the open question on #3769
The review on #3769 asks whether the incomplete quota signal always carries a
messagestring, or whetherincomplete_details.reason: usage_limit_reachedcan arrive with an empty message — which would slip past theisRateLimitOrQuotaFailureMessage"usage limit"match.I scanned the recent tail of a production
usage.jsonlfor everyterminalStatus: "incomplete"row carrying a usage-limit signal:All 77 carried the message string, so the message-based detection in #3769 covers 100% of what this pool observed. That is positive evidence for the current approach rather than a counterexample. It is one pool on canonical ChatGPT forward, so it does not prove the reason-only shape never occurs; if a defensive
incomplete_details.reasoncheck is added, this data suggests it would be belt-and-braces rather than the primary path.Every one of those rows also had
usageStatus: "unreported", which may be a useful secondary signal: a terminal that reports no usage at all is a plausible cross-check for "the account did not actually serve this request."Second blast radius: already-running sessions
The pool wedged again on 2026-09-07 (07:44 and 07:53 KST). This time the recovery action was not the problem — an external watchdog detected the drained account and switched away within seconds, verified. Sessions kept dying anyway.
The reason is thread affinity, and I think it matters for #3769.
ocxstates the constraint in its own help text:Every already-running session holds an affinity binding to the drained account.
ocx account useandocx account priorityact on future unbound selection, so they rescue new sessions while every in-flight session keeps hitting the dead account until it fails.Two vendor paths would retire such a binding —
isCodexAccountSelectablein the affinity branch ofresolveCodexAccountForThreadDetailed, andclearThreadAccountMapForAccountinrecordCodexUpstreamOutcome— but neither can fire here, for the same root cause: the exhaustion terminal is recorded as outcome 200, so no failure evidence is produced.reevaluateAffinityQuotacannot help either, because it scores the binding fromgetAccountQuota, the same stale cache.So the defect has two distinct blast radii:
Reading the quota branch of
recordCodexUpstreamOutcome, #3769 does appear to cover this: the quota class reachesclearThreadAccountMapForAccount. I am flagging it explicitly because it is the part that determines whether a user's current work survives, and the PR's new test file covers the 429 capture and cooldown trip rather than the affinity release. A regression case asserting "a thread bound to the drained account rebinds on its next request" would pin the behavior that actually saves in-flight sessions.Suggestion
#3769 already fixes the core defect and I would rather see it land than see a competing change. Two things seem worth considering on top:
weeklyPercentstays stale until the next explicit refresh, sohasCodexQuotaHeadroomstill reports headroom for a provably drained account. Marking the cached entry stale on a quota terminal would make recovery independent of cooldown timing.ocx statuswas green and the auto-switch setting was correct. Surfacing "active account returned a quota terminal but cached quota shows headroom" inocx statusor the dashboard would turn a multi-hour diagnosis into a glance.For the released version, the only lever that reaches a bound thread is
ocx account pause, becauseisCodexAccountSelectablechecksisCodexAccountPausedfirst, and the pause route additionally runsclearThreadAccountMapForAccount+selectFallbackAfterPause. That is what I automated locally: pause on detection, resume automatically once the quota window recovers. It covers the native main account too —cmdPausemaps the literalmaintoMAIN_CODEX_ACCOUNT_IDand the route admits it explicitly (verified on 2.43.0). That is a workaround for the released version, not a proposed design.Version
2.43.0
Operating system
macOS 26.5.2 (arm64)
Provider and model
OpenAI / canonical ChatGPT forward, 5-account Codex pool. Observed on
gpt-6-astra,gpt-5.6-sol, andgpt-5.6-luna; the defect is account-pool routing rather than model-specific.Logs or error output
43 of these in 15 minutes on one account with no rotation, and 30 more in a later recurrence.
ocx statusreported a healthy proxy throughout andocx account auto-switch openai statusreportedon (threshold 99%).Checks