[Bug]: 2.41.0 keeps routing to a 5h-exhausted Codex account after repeated 502s

[h-dot-seo]

Client or integration

Codex App

Area

Authentication and account pool

Summary

OpenCodex 2.41.0 continued routing new Codex requests to a ChatGPT pool account after its five-hour window reached 100%, although auto-switch was enabled at an 80% threshold and a second account had only 3% usage. Requests recovered immediately after the exhausted account was manually paused.

This appears to be a recurrence or remaining transport-path gap related to #3029, #3110, and #3176, rather than a missing auto-switch setting.

Reproduction

1. Install and start @bitkyc08/opencodex 2.41.0 with two valid ChatGPT accounts in the Codex account pool.
2. In the OpenCodex dashboard, enable automatic account switching, select the quota strategy, set the threshold to 80%, and set the sticky request limit to 1.
3. Use account A until the dashboard quota refresh shows its five-hour/short window at 100%. Confirm account B remains available; in this occurrence it showed 3% five-hour usage.
4. With account A still active, submit new streaming turns from the Codex App through OpenCodex. Multiple concurrent and sequential turns were active during the occurrence.
5. Observe that new requests continue selecting account A and fail with HTTP 502 instead of switching to account B. In the local usage log, each failed attempt had sendCount: 1 and an empty recoveryKinds array.
6. In the OpenCodex dashboard, manually pause account A.
7. Submit another Codex turn. The next recorded request selects account B and returns HTTP 200.

Observed timeline on 2026-09-04 KST: account A produced 118 HTTP 502 attempts from 10:58:32 through 11:21:11. Account B returned HTTP 200 at 11:21:33, immediately after account A was manually paused.

Version

2.41.0

Operating system

Windows NT 10.0.26200.0

Provider and model

OpenAI ChatGPT account pool / Codex models used by Codex App

Logs or error output

Sanitized usage.jsonl aggregation, 2026-09-04 KST

provider/account: exhausted pool account (identifier removed)
HTTP 502 attempts: 118
first: 10:58:32
last: 11:21:11
sendCount min/max: 1/1
attempts with recoveryKinds: 0
attempts with streamAborted=true: 55
attempts with firstOutputMs: 52
attempts without streamAborted marker: 63
attempts with persisted upstreamError: 0

After manually pausing the exhausted account:
healthy pool account HTTP 200: 11:21:33

The subsequent quota refresh showed account A at five-hour/short 100% and weekly 49%, while account B was at five-hour/short 3% and weekly 18%.

The exact upstream 502 response body was not persisted. Therefore, this report does not claim that every unmarked 502 contained the canonical quota message.

Screenshots and supporting files

No screenshot is attached. The values above were aggregated directly from the local OpenCodex usage log with account identifiers, prompts, tokens, and response content excluded.

Relevant prior work: #3029 reported the same manual-pause recovery symptom; #3110 treated a live full burst window as exhausted; #3176 added rotation for quota messages wrapped in HTTP 502; #585 added same-request failover for pre-stream 429/402 responses.

The 2.41.0 main source at 4600f80198c8eb6cef1b9cf1338d51906100007e contains the bounded pre-stream 5xx quota-message classifier. It also explicitly leaves a quota failure terminal after SSE begins and records post-200 stream failure as synthetic HTTP 502 with streamAborted.

Suggested regression coverage:

• Account A has shortPercent: 100, account B is healthy, quota threshold is 80%, and multiple new/concurrent requests are admitted.
• A post-200 streamAborted request may remain terminal, but subsequent new admissions must avoid account A.
• A 502 without a persisted canonical message must not override a fresh quota snapshot that already marks account A at 100%.
• Affinity is cleared or re-evaluated so the next request reaches account B.

Redacted configuration

{
  "accountPool": {
    "accounts": 2,
    "autoSwitch": true,
    "strategy": "quota",
    "thresholdPercent": 80,
    "stickyRequestLimit": 1
  }
}

The JSON above is a redacted description of the effective dashboard settings, not a verbatim copy of the on-disk configuration schema.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

---

This issue or PR was generated by LazyCodex.
Tag: lazycodex-generated

[github-actions]

Issue reopened

The report now contains the information required by the automated check. Thanks for updating it.

[lidge-jun]

리뷰 · 우선순위 70 / 80

설명

이 이슈는 OpenCodex 2.41.0에서 Codex 계정 풀 자동 전환이 켜져 있고(quota 전략, 임계 80%, sticky 1) 계정 A의 5시간 창이 100%인데도, 새 Codex App 요청이 A로 계속 붙어 HTTP 502를 내고, A를 수동 pause한 뒤에야 계정 B(3%)로 200이 난 사고 보고입니다. 로그 요약이 설득력 있습니다. A로 118번의 502, sendCount 전부 1, recoveryKinds 비어 있음, 일부는 streamAborted, 업스트림 본문은 저장되지 않음. 수동 pause 직후 B 성공. 관련 선례로 #3029(수동 pause 회복), #3110(풀 버스트 창), #3176(502에 싸인 quota 메시지 로테이션), #585(스트림 전 429/402 페일오버)를 스스로 연결했습니다.

현재 dev(072df52eb, 패키지 2.42.0 라인) 기준으로 보면 “임계값 설정이 없다”는 문제는 아닙니다. src/codex/routing.ts의 hasCodexQuotaHeadroom/applyQuotaAutoSwitch는 autoSwitchThreshold(기본 80)와 computeCodexUsageScore로 active를 넘기고, src/codex/quota.ts의 isCodexQuotaExhausted는 shortPercent 100을 계획과 무관하게 소진으로 봅니다. pause-exhausted API(src/codex/auth-api.ts)도 스냅샷이 100이면 pause합니다. 그런데 보고는 대시보드 quota refresh에는 A가 100으로 보이는데도, 새 admission이 A를 계속 고르고, 실패가 same-request recovery로도 안 넘어간 상태입니다. 이슈가 짚은 가설이 핵심입니다. (1) 스트림이 시작된 뒤의 실패는 터미널로 두고, (2) 본문 없는 502가 신선한 100% 스냅샷을 덮어쓰거나 전환을 막으면 안 되며, (3) affinity/sticky가 A에 묶인 채 다음 요청도 A로 가면 안 됩니다. 릴레이 쪽(src/server/relay.ts)은 post-200 스트림 실패를 synthetic 502 + streamAborted로 남기는 경로가 있어, “502만 보고 quota 로테이션”과 “이미 소진으로 알려진 계정은 다음 admission에서 제외”를 섞으면 구멍이 납니다.

재현 제안(임계 80% + A short 100 + B 건강 + 동시/연속 새 요청, streamAborted는 터미널이어도 다음 admission은 A 회피, 본문 없는 502가 스냅샷을 이기지 못함, affinity 재평가)은 그대로 회귀 테스트 스펙으로 쓰면 됩니다. LazyCodex 생성 태그(#3425)가 있어도 로그 숫자·타임라인·설정 요약은 사람이 쓸 만한 수준입니다. types/config 분할로 닫을 이슈가 아닙니다.

우선순위 70은 풀 사용자에게 바로 체감되는 회귀/잔존 구멍이고, 수동 pause 워크어라운드만으로 버티게 두면 안 된다는 뜻입니다. 다만 업스트림 502 본문이 없어 “#3176 classifier miss”인지 “선택기가 stale active/affinity를 붙잡음”인지 한 번에 단정하진 마세요.

경로 src/codex/routing.ts applyQuotaAutoSwitch / hasCodexQuotaHeadroom - 임계 로직은 HEAD에 있습니다. 보고는 이 경로가 새 admission에 안 먹거나, 먹어도 affinity가 되돌리는지 확인이 필요합니다.
경로 src/codex/quota.ts isCodexQuotaExhausted - short 100이면 소진입니다. 대시보드 숫자와 선택기 입력이 같은 스냅샷인지 봐야 합니다.
경로 src/server/relay.ts streamAborted / synthetic 502 - 이슈의 55건과 맞을 수 있습니다. same-request recovery 없음(recoveryKinds 빈 배열)과 다음 요청 계정 선택이 따로 깨졌을 수 있습니다.
경로 #3029 / #3110 / #3176 / #585 - 관련 선례로 맞습니다. 중복 close보다는 “어느 가드가 2.41.0 경로에서 빠졌는지” 이슈로 유지하세요.

메인테이너의 판단이 필요한 지점

• 2.42.0 dev에서 동일 시나리오를 재현해 이미 고쳐졌는지, 아니면 2.41.0과 같은지
• 1차 가설을 “선택기/affinity가 100% 계정을 계속 고름”으로 둘지, “본문 없는 502 recovery 공백”으로 둘지
• stickyRequestLimit=1과 데스크톱 affinity 키가 충돌하는지
• 핫픽스 범위를 pause-exhausted 자동 호출로 좁힐지, admission 제외로 넓힐지

너의 추천
open bug로 두고, 이슈에 적힌 회귀 네 줄을 테스트로 먼저 고정하세요. 임시 완화는 사용자가 한 것처럼 소진 계정 수동 pause(또는 ocx account pause-exhausted)입니다. 본문 없는 502만으로 스냅샷 100%를 무시하지 않게 하는 쪽이 안전한 수정 방향입니다. 지금 close하지 마세요.

이 댓글은 grok-bot이 작성했습니다

[lidge-jun]

Investigated during the 260904 backlog review. I want to record what the code actually says, because it rules out the obvious fix.

It is not the quota selector. tests/codex-routing.test.ts:325 already pins "known 100% weekly usage is exhausted, not unknown, and switches accounts", and :308 pins isCodexQuotaExhausted on an explicit 100% window. A known 100% account already rotates away today. The unknown-usage passes at src/codex/routing.ts:1215 and :1655 are deliberate — the comment above them says unknown usage must not drain a tier that was simply never primed — so tightening them would not fix your case and would break accounts whose providers report no quota at all.

The suspect is that your failures were never recorded. recordCodexUpstreamOutcome begins with, at src/codex/routing.ts:2195:

if (writerGeneration < lastReconciledGeneration && !liveHealthAccountIds.has(accountId)) return;

If a writer's captured generation is older than the last config reconcile and the account is not in the live set, the outcome is dropped whole — no consecutiveFailures increment, no lastFailureAt, no soft-avoid. The transient branch at :2459 never runs, so upstreamFailoverThreshold (default 3) is never reached regardless of how many 502s arrive. That fits your report precisely: 118 failures, sendCount all 1, recoveryKinds empty, and rotation only after a manual pause, which takes a different path.

There is a second, independent amplifier at :2455. The streak resets to 1 when the previous failure is older than CODEX_FAILURE_WINDOW_MS (5 minutes). If your failing turns were more than five minutes apart — which hand-retried interactive work often is — the threshold of 3 is unreachable by construction, even when recording works perfectly.

What would tell these apart, and it is the one thing I cannot get from the log summary you posted: did anything reload configuration between the accounts being registered and the failures arriving — a dashboard edit, an ocx config set, an account add or pause, a service restart? A config reload bumps the generation, which is what arms the first path.

If you still have the run-request records, the useful fields are the timestamps of consecutive failures (to see whether they fall inside a 5-minute window) and whether any config write happened in that period. Either answer produces a different fix, and I would rather fix the one that actually fired than harden both and hope.

[lidge-jun]

Re-examined this against current dev, and the earlier "quota selector is at fault" theory is still wrong — but so is the follow-up suspicion, which is worth recording so nobody re-walks it.

Ruled out again: the quota selector. tests/codex-routing.test.ts:325 already proves a known-100% weekly account rotates away, and computeCodexUsageScore (src/codex/routing.ts:363-391) collapses short into long with Math.max when a known long window exists.

Newly ruled out: the generation guard. src/codex/routing.ts:2195 is an AND:

if (writerGeneration < lastReconciledGeneration && !liveHealthAccountIds.has(accountId)) return;

liveHealthAccountIds is filled at :331 from context.codexAccountIds, which listLiveCodexAccountIds (:282) builds from every id in config.codexAccounts. Account A was registered and live throughout, so the second condition is false and the early return never fires. This guard only reaches an account that was removed from config.

Also ruled out: the failure-window amplifier. :2455 treats a failure as stale past CODEX_FAILURE_WINDOW_MS (5 minutes, :116). Your timeline is 118 failures between 10:58:32 and 11:21:11 — roughly 12 seconds apart, so nothing ages out.

So the two questions worth your time are different from the ones asked earlier:

1. Was account A ever deleted and re-added during the incident, as opposed to the config merely being reloaded? Only removal empties it out of liveHealthAccountIds.
2. Is the Codex App pinning a specific account on these requests? src/codex/auth-context.ts:692,709 sets meta.fixedAccount = true whenever fixedAccountId is present, and routing.ts:2482/2487/2492 then skips both affinity clearing and failover promotion. If that flag is on, 118 attempts on one account is the system behaving as designed, and the fix is a product decision about what "fixed" should mean when the fixed account is failing — not a routing bug.

Separately, one real defect surfaced while checking this, and it is not yours: tests/codex-routing.test.ts:527 never calls reconcileCodexRoutingHealth, so it asserts against an empty liveHealthAccountIds — a state production never reaches. It pins an unreachable path green. It is not being changed in this pass because the fix belongs with whatever root cause this issue lands on, but it is recorded here so the next reader does not mistake it for coverage.

[lidge-jun]

Closing the demonstrated exhausted-account selection defect as implemented on dev in #3605 (commit e1b9ec851958c46ad6210a989b62c7b367edefee).

A freshly observed 100% short window now excludes the account even when the quota response omits its reset timestamp. Dedicated short-window observation time survives persistence and unrelated quota updates; stale, missing, or future observations remain unknown. The landed regression named “a 502 storm does not pin the pool to an exhausted account (#3425)” checks both new requests and an already-bound thread against a healthy alternate account.

This closes the reproduced selection mechanism, not a claim that every historical 502 had the same cause. Unmarked 502s remain transient and no blanket stream replay was introduced. If the original symptom recurs on a build containing this fix, reopen with the running revision and a sanitized current quota snapshot.

Verified that the landing commit is an ancestor of current dev af50c6d3451078a7d298b044c08fd2684c9e8eeb. Cross-platform CI passed for that exact dev commit (Linux and macOS test shards; the Windows full-suite job was skipped).