[Bug][Windows] Continuation spill failures accumulate behind healthy readiness after #3011 (observed on 2.39.0)

[stephen-drew]

Diagnostic follow-up landed; runtime recovery remains open

#3790 (c2b4dc0696) distinguishes fresh retry timeouts from remembered timeout refusals. The reported long-lived Windows process recovery failure is not declared fixed. The remaining acceptance is same-process evidence and a justified recovery remedy.

Verified against dev 5759d9ea2f1e7281cdc01eb9628f2e0a123fb59c. Original report by @stephen-drew. The attribution record was added in #3811.

Client or integration

Codex App through the local OpenCodex proxy.

Area

Windows response-state persistence, failure diagnostics and recovery.

Summary

During normal concurrent Codex App use on 1 September, an official OpenCodex 2.39.0 process stopped successfully publishing continuation spills while continuing to report healthy/ready. The successful spill count remained at 1,988 and the failure count eventually reached 11,365 in the same process.

This follows the asynchronous Windows spill work in #3011, #3018 and #3044. The new evidence concerns sustained persistence degradation after those changes. The saved measurements do not identify the exact exception thrown inside the affected process.

Reproduction

The incident was intermittent, under concurrent Codex App traffic. A deterministic production trigger has not been established.

1. Run the official 2.39.0 Windows package with multiple concurrent Codex tasks.
2. Compare successive supported ocx memory --json and health/readiness observations.
3. Observe a successful spill count that stops advancing while spill failures continue increasing for hours.
4. In a separate diagnostic process, verify that the same release's durable spill writer can publish, read back and clean up an isolated disposable spill.

Step 4 succeeded in the captured incident, making a persistent process-local failure a useful investigation direction.

Logs or error output

Saved incident evidence:

Installed/live version: 2.39.0
Official package parity: 923 / 923 files identical
Same process throughout incident: yes
Health/readiness: healthy
Successful spill publications: fixed at 1,988
Spill write failures: 1,582 -> 1,763 -> 3,439 -> 3,599 -> 3,714 -> 11,365
Spill read failures: 0
Earlier degraded-state snapshot: 946 tombstones, 0 spill stubs
Final sampled RSS: approximately 578 MB
Free disk space: approximately 229 GB
Final active turns: 7
Final draining state: false

The resident-memory budget was not exceeded. The spill directory retained a protected owner-only ACL. No ACL timeout override was found in the inspected service or environment configuration.

Separate-process checks of the exact release passed:

Single durable publication, read-back and cleanup: 305 ms
Five sequential publications: 232, 88, 92, 87, 85 ms
Leftover diagnostic files: 0

Source tests also demonstrated the bounded ACL-timeout recovery path and its exhausted-recovery error. This makes exhausted ACL recovery a hypothesis supported by the source and isolated tests. The live incident exposed aggregate counters only, so EACLRETRYEXHAUSTED is not claimed as a captured live exception.

Expected behaviour

Preserve required ACL checks and bounded waits. Sustained failure to persist continuations should be observable as a specific degraded capability, including a privacy-safe cause and an actionable recovery route.

A healthy routing endpoint can coexist with a persistence failure, but operators need to be able to distinguish those states without inferring the cause from thousands of failed saves.

Suggested investigation

• Expose a bounded, sanitised last spill error code, timestamp and failure streak alongside the existing counters.
• Distinguish an isolated failed response from a persistence path that can no longer publish successfully.
• Test an exhausted Windows ACL recovery followed by a healthy ACL runner, and document when publication can resume in the same process.
• If the failure requires a restart, expose that requirement explicitly and preserve the active-turn/drain boundary.
• Preserve fail-closed ACL behaviour, publication ordering and protection against automatic replay of side-effecting tool calls.

Version

Observed live: OpenCodex 2.39.0, official source commit af6113a0381d6fff2e4dce587652825c7eeb6423.

Installed package when reviewing this report on 4 September: 2.42.0. The incident has not been reproduced on 2.42.0. At the 4 September review snapshot, the process had approximately 19 minutes of uptime and reported zero successful spills and zero spill failures, so it had not exercised the relevant path.

Operating system

Windows, x64. The current review host reports 25H2, build 26200.9168.

Provider and model

The suspected failure is in local continuation persistence. No provider-specific root cause is claimed.

Screenshots and supporting files

Evidence was recovered from the saved incident report, package-parity checks, successive runtime snapshots and isolated diagnostic results. Private prompts, conversation contents, usernames, paths and credentials are omitted.

Redacted configuration

{
  "client": "Codex App",
  "platform": "Windows x64",
  "serviceBackend": "Task Scheduler",
  "proxyBinding": "local loopback"
}

Checks

• Current open spill-related issues were searched before submission.
• Related reports [Bug][Windows] synchronous ACL hardening still stalls /healthz and poisons response spill writes after #612 #3011, fix(windows): keep ACL stalls off the response event loop #3018 and fix(windows): drain response spill publications before shutdown snapshot #3044 were reviewed.
• The recorded incident is distinguished from the current unexercised process.
• The proposed cause is labelled as an inference.
• Private task content and credentials are excluded.

[Ingwannu]

Confirmed as a real current observability gap, but not yet as a reproduced current-version persistence defect.

On current dev, every ordinary Windows async spill exception reaches the catch in runPendingResponseSpill, increments only spillWriteFailures, and is then discarded. responseStateMetrics() exposes cumulative writes/read failures/write failures, while authenticated /api/system/memory forwards that scalar snapshot and /api/system/health remains ordinary process liveness. There is no privacy-safe last error code or timestamp, no consecutive-failure streak, and no success marker that proves the same process recovered. Therefore the reported 1,988 fixed writes plus 11,365 failures cannot currently be distinguished from many isolated failures using supported diagnostics.

I am keeping two claims separate:

• The missing degraded-capability diagnostics still exists on current dev and is actionable.
• The 2.39.0 incident's exact cause is not established. The live evidence did not capture EACLRETRYEXHAUSTED, and the report has not exercised this path on 2.42+ yet, so I will not label the root cause or weaken/restart the ACL path speculatively.

The bounded fix I would accept is:

1. Track a scalar-only spill-write health snapshot: consecutive failures, last failure timestamp, a sanitized closed error-code class, and last successful publication timestamp. Never expose messages, paths, response ids, payloads, account data, or raw causes.
2. Reset the failure streak only after a durable publication succeeds; keep cumulative counters unchanged for compatibility.
3. Surface that snapshot through the existing authenticated memory diagnostics and CLI JSON contract. Do not put it on unauthenticated /healthz.
4. Add a same-process failure -> failure -> successful publication regression, plus an exhausted Windows ACL attempt followed by a healthy runner, proving publication resumes without weakening ACL enforcement.
5. Keep readiness semantics separate for now. A routing endpoint can remain live while the authenticated diagnostic reports continuation persistence as degraded; automatic restart/drain behavior needs separate evidence.

I will leave this open while the current-version reproduction remains unproven. A 2.42/2.43 exercised snapshot showing the counters and the first sanitized failure class after this instrumentation will decide whether a second runtime-recovery fix is needed.

[lidge-jun]

리뷰 · 우선순위 62 / 80

이 이슈는 Windows에서 Codex App이 동시에 여러 일을 돌릴 때, 이어쓰기(continuation)를 디스크에 저장하는 spill 쓰기가 계속 실패하는데도 프로세스는 계속 “정상/준비됨”으로 보이는 문제를 말합니다. 보고자는 공식 2.39.0에서 같은 프로세스 안에서 성공한 spill이 1,988에서 멈춘 채 쓰기 실패만 1,582 → 11,365까지 쌓였고, 디스크 여유·RSS·읽기 실패는 정상이었다고 적었습니다. 같은 바이너리로 별도 프로세스에서 한 번 쓰고 읽고 지우는 검사는 통과했기 때문에, “파일 시스템 전체가 죽었다”기보다 “그 프로세스 안의 저장 경로가 오래 막힌 상태”로 보는 편이 맞습니다. 관련 배경은 비동기 Windows spill 작업 #3011/#3018/#3044 이후의 지속 저하입니다.

지금 dev HEAD 3aab264e3(패키지 2.43.0, 방금 #3510으로 tests 레이아웃 일부 이동)를 보면, 이 진단 구멍은 그대로입니다. src/responses/state.ts의 runPendingResponseSpill은 ACL 타임아웃 한 번 재시도 뒤에도 실패하면 catch에서 spillCounters.writeFailures만 올리고 예외는 버립니다. responseStateMetrics()가 내보내는 값은 누적 spillWrites/spillWriteFailures/spillReadFailures와 개수·바이트 정도뿐이고, “마지막 실패가 언제·무슨 종류였는지”, “연속으로 몇 번 실패했는지”, “같은 프로세스에서 다시 성공했는지”는 없습니다. src/server/management/system-routes.ts의 인증된 /api/system/memory는 그 스냅샷을 그대로 실어 주고, /api/system/health와 비인증 /healthz는 프로세스 살아 있음 쪽에 가깝습니다. 그래서 운영자가 “라우팅은 되는데 이어쓰기 저장만 죽었다”를 숫자만으로 구분해 내기 어렵습니다. 메인테이너(Ingwannu) 댓글도 같은 선을 확인했고, 2.39.0 원인 단정과 ACL 약화는 보류하는 쪽입니다.

보고 버전은 2.39.0이고, 작성 시점의 2.42.0 호스트에서는 spill 경로를 아직 거의 안 탄 상태라 현재 버전 재현은 없습니다. 따라서 “지금 빌드가 같은 방식으로 고착된다”고 단정하면 안 되고, 먼저 안전한 관측(마지막 실패 코드 분류·시각·연속 실패·마지막 성공 시각)을 넣는 쪽이 맞습니다. Windows ACL 쪽 코드는 src/lib/windows-secret-acl.ts에 EACLRETRYEXHAUSTED가 있고 tests/windows/windows-secret-acl.test.ts에 고갈 경로 테스트가 있지만, 라이브 사고 로그에는 그 코드가 잡혀 있지 않습니다. 가설로는 두고, 계측 후에야 두 번째 런타임 복구(재시작·드레인)를 논의하는 게 안전합니다. 지금 dev의 메인 트랙은 #3497 tests 도메인 이동(#3511/#3513/#3516/#3518 스택)이라, 이 이슈는 그 열차와 겹치지 않는 platform 관측 작업으로 따로 잡으면 됩니다.

라인 / 심볼 기준으로 보면 아래가 핵심입니다.

src/responses/state.ts runPendingResponseSpill catch (대략 383–389) - 예외를 삼키고 writeFailures만 올려서, 연속 실패·마지막 원인 분류·마지막 성공 시각을 남기지 않음
src/responses/state.ts ResponseStateMetrics / responseStateMetrics() (대략 2099–2153) - 누적 카운터만 노출. streak·lastFailureAt·sanitized code·lastSuccessAt 필드 없음
src/server/management/system-routes.ts /api/system/memory - 위 스칼라만 전달. 인증된 진단에 spill health 스냅샷을 더할 자리
/api/system/health 및 /healthz - 라우팅/프로세스 생존과 continuation 저장 가능 여부를 구분하지 않음(의도적으로 분리 유지하는 편이 맞음)
src/lib/windows-secret-acl.ts EACLRETRYEXHAUSTED - 소스·단위 테스트에만 있고, 이 사고의 라이브 증거로는 아직 확정되지 않음

메인테이너의 판단이 필요한 지점

• 관측만 먼저 넣을지, 같은 PR에 “ACL 고갈 후 같은 프로세스에서 재개” 회귀까지 묶을지
• readiness/health를 바꾸지 않고 authenticated memory/CLI만 쓸지(메인테이너가 제안한 경계와 동일)
• 2.42/2.43에서 spill이 실제로 도는 스냅샷이 오기 전에 runtime 재시작/드레인 자동화를 열지 말지
• 이슈를 help wanted로 열어둘지, 메인테이너가 계측 PR을 직접 잡을지

너의 추천
Ingwannu가 적어 둔 범위 그대로 진행하는 게 맞습니다. (1) 개인정보 없는 spill-write health 스칼라(연속 실패, 마지막 실패 시각, 닫힌 오류 코드 분류, 마지막 성공 시각)를 responseStateMetrics()와 인증된 memory/CLI에만 추가하고, (2) 누적 카운터는 호환용으로 유지한 채 성공 시에만 streak을 리셋하고, (3) 같은 프로세스 failure→failure→success 회귀와 ACL 고갈 후 건강한 runner로 재개되는 테스트를 넣으세요. /healthz에는 넣지 마세요. ACL 강제나 자동 재시작은 현재 버전에서 카운터·첫 sanitized 실패 분류가 잡히기 전까지 손대지 마세요. 이슈는 열어 두되, 작업 범위는 “관측 먼저”로 잘라 주세요.

이 댓글은 grok-bot이 작성했습니다

[Ingwannu]

Implemented the bounded observability fix in #3525.

The PR keeps cumulative counters and adds only privacy-safe process-local state: initial / healthy / degraded, consecutive write failures, a closed last-error class, and last failure/success timestamps. A durable publication is the only event that clears the streak. Raw errors, paths, ids, payloads, and causes are not exposed, and neither /healthz nor readiness/restart behavior changes.

Focused regressions now prove both failure → failure → success and Windows ACL retry exhaustion → healthy-runner recovery in the same process without weakening ACL enforcement. I am leaving this issue open, as planned, until an exercised current-version host captures the new diagnostic snapshot and establishes whether a separate runtime-recovery defect still exists.

[stephen-drew]

Exercised Windows 2.43.0 follow-up for #3522, with the diagnostics carried in #3542 present.

Two authenticated memory snapshots from the same running process on 6 September 2026 show continuing failure:

UTC snapshot	Successful spill writes	Failed writes	Consecutive failures	Status	Last failure code
07:23:19.811	1,567	1,792	1,711	degraded	EACLRETRYEXHAUSTED
07:28:15.121	1,567	1,897	1,816	degraded	EACLRETRYEXHAUSTED

That is 105 additional failures in approximately five minutes with no successful publication. The last-success timestamp stayed unchanged while the last-failure timestamp advanced. Process health remained healthy; 10-12 turns were active, with no drain or restart during this observation. RSS remained below the 4 GiB warning threshold; spill read failures and app-owned over-budget bytes were zero.

Five sequential synthetic spill writes in a separate process, using the installed spill implementation and an isolated home, all passed write/read-back/cleanup in 124-300 ms. Read-only inspection of the live spill directory found protected inheritance and current-user-only FullControl, with no reparse point.

Source disclosure: all installed package files match the verified official 2.43.0 archive except a registered local first-response timeout change in src/server/responses/codex-ws-wire.ts (30 to 300 seconds). Spill-state and Windows ACL code are unmodified.

This captures the previously missing current-version failure class and confirms an ongoing degraded streak. It does not identify the original timeout stage or distinguish a remembered exhausted retry from a second actual timeout. The isolated success does not prove live same-process recovery.

Does this support a separate runtime-recovery investigation, and which additional privacy-safe diagnostic would best distinguish those two paths? No ACL weakening or automatic restart change has been applied.

[Ingwannu]

Yes, the same-process streak supports continued runtime-recovery investigation here. The separate-process success does not test the long-lived process's memoized timeout state, so it cannot establish recovery.

The current source confirms why the displayed code is ambiguous. windows-secret-acl.ts::previousTimeoutError(true) emits EACLRETRYEXHAUSTED for a previously consumed recovery. Separately, responses/state.ts sets exhaustedAclRetry when the second spill attempt returns ETIMEDOUT, then passes the same EACLRETRYEXHAUSTED override to noteSpillWriteFailure. Thus this field alone cannot count fresh ACL timeouts versus remembered refusals.

The smallest useful additional diagnostic would record a fixed origin alongside the existing code: retry_returned_timeout versus timeout_memo_refusal, with separate counters. At the ACL layer, an optional fixed step category and whether an OS command was actually launched would distinguish waiting for identity/compliance inspection from grant/inheritance/removal work. No raw paths, SIDs, usernames, stdout or credential material are needed.

Current async hardening also attempts existingAclAlreadyCompliantAsync before checking the timeout memo. Consequently, read-only inspection that the directory looks correct is not yet evidence that the live process's bounded compliance check completed successfully.

Keep the existing fail-closed ACL boundary and avoid automatic restart or memo clearing. The next recovery decision should be based on that same-process diagnostic, not on a fresh process passing five writes. No Windows recovery fix is claimed in this reply.