Skip to content

Fix app-server reader recovery for large messages - #36

Open
dimavrem22 wants to merge 3 commits into
mainfrom
fix/app-server-reader-framing
Open

Fix app-server reader recovery for large messages#36
dimavrem22 wants to merge 3 commits into
mainfrom
fix/app-server-reader-framing

Conversation

@dimavrem22

@dimavrem22 dimavrem22 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Executive Summary

Prevent bridged Codex turns from hanging when app-server emits large or early completion messages.

  • Raise the newline-delimited app-server message limit to a configurable 16 MiB default and bump the plugin to 0.2.9.
  • Fail pending work immediately and recycle unhealthy app-server clients when the stdout reader stops.
  • Buffer completion notifications that arrive before turn/start capture registration.

Description

This change adds CODEX_APP_SERVER_STREAM_LIMIT_BYTES, threads it through the subprocess stream reader, and documents its 16 MiB default. Reader-loop exceptions now produce an error log, fail every pending request and turn, terminate the child, and make client health false immediately. Session acquisition disconnects an unhealthy client and resumes the saved thread in a replacement process.

run_detailed now buffers agent-message, item-completed, and turn-completed notifications while the turn/start response is pending, then replays notifications for the returned turn after capture registration. Subprocess teardown is also bounded and awaits background task cleanup.

Reason

Python's default asyncio subprocess stream limit is 64 KiB. A larger app-server JSON line could stop the only stdout reader while turn futures remained pending until the outer turn timeout. Independently, app-server can emit completion notifications immediately after its turn/start response; the reader can process those lines before the awaiting coroutine registers its turn capture, dropping the completion and causing the same visible hang.

Decisions

  • Stream bound: Use a 16 MiB default with an environment override so operators can tune the maximum frame without a code change.
  • Reader health: Treat a missing or completed reader task as unhealthy even if the child process still appears alive.
  • Failure recovery: Settle all request and turn futures on reader failure, terminate the unreadable child, and let the owning session recreate it while resuming the saved thread.
  • Early notifications: Buffer only turn-result notification types during the narrow turn/start registration window and replay only the returned turn's events.
  • Teardown: Close a paused stdout transport after a line-limit failure and bound stdin/process shutdown waits.
  • Live voice evidence: Require exact caller intent in the fresh call transcript, a fresh open SMS action, successful reconciliation, and exactly one exact-body outbound SMS without requiring the hosted action prose to duplicate the transcript body.

Testing

  • Python 3.11 full offline suite: 442 passed, 24 skipped.
  • Python 3.12 full offline suite: 443 passed, 23 skipped.
  • Real Codex CLI 0.146.0 host contract: 5 passed.
  • Added regressions that assert item/completed and turn/completed fixtures exceed 64 KiB; verify successful parsing at the default limit; and verify low-limit failure logging, settlement of multiple pending requests and an active turn, child termination, unhealthy-client replacement, and early notification replay.
  • Hosted-voice live gate retains fresh-call correlation, exact transcript intent, open SMS action, successful reconciliation, and exactly-once exact-body delivery checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant