Proposal: recover stuck running turns when turn/completed is lost after long command execution

[cyppe]

Problem

We already have #911 for the Linux-specific case where the UI can get stuck in the running state because the final turn/completed JSON line never gets emitted by Node readline when stdout closes without a trailing newline.

I think there is a second, broader failure mode with a very similar user symptom:

• tool rows show as completed (MCP tool call complete, Command run complete)
• the UI still shows Working for ... indefinitely
• switching away and back can recover because the app resyncs from snapshot state

That makes this look less like a frontend rendering bug and more like a stranded turn lifecycle on the server side.

Why this looks server-side, not frontend-only

In the current UI, the loading state is driven from thread/session lifecycle state:

• apps/web/src/session-logic.ts -> derivePhase()
• apps/web/src/components/ChatView.tsx -> isWorking

The work log/tool rows can already be complete while the composer still remains blocked if the session never transitions from running back to ready.

So when the screenshot shows completed tool activity but the turn is still stuck, the most likely missing event is still turn/completed (or an equivalent lifecycle reconciliation), not the command tool UI itself.

Why long-running shell commands may make this easier to hit

Long-running / high-output command execution looks like a plausible trigger because it increases the chance that the provider finishes tool activity while the final turn lifecycle message is delayed or lost.

There is also a possibly related upstream Codex report:

• openai/codex#13821 - app-server command/exec can hang after outputBytesCap is reached due to back-pressure

I do not think T3 Code should depend on that being fixed upstream before handling this safely on our side.

Proposal

Keep the existing #911 style readline hardening, but add a more defensive recovery path in apps/server/src/codexAppServerManager.ts:

1. Keep the readline close residual flush so newline/buffering issues still get handled.
2. While a turn is marked running, start a lightweight recovery timer.
3. If the turn is still marked running after the grace period, call thread/read.
4. If thread/read shows the active turn already has a terminal status / completedAt, synthesize the equivalent recovered completion handling server-side and clear session state.
5. Ignore a later duplicate real turn/completed notification if it eventually arrives.

That gives T3 Code a provider-state reconciliation path instead of assuming realtime notifications are always sufficient.

Why this seems like the right layer

• preserves current frontend behavior
• makes the session lifecycle self-healing
• covers both the Linux newline-buffering case and other missing/delayed completion cases
• protects users even if Codex app-server has edge-case command/exec behavior under back-pressure or long output

Local evidence

I have a local patch with:

• readline crlfDelay: Infinity
• residual stdout flush on readline close
• turn completion recovery via thread/read
• duplicate completion suppression
• regression tests for recovery + dedupe

Local verification passed:

• bun fmt
• bun lint
• bun typecheck
• bun run test src/codexAppServerManager.test.ts (from apps/server)

Goal of this issue

This is intentionally filed as a proposal / design direction, not as "this exact patch must land unchanged".

The main point is:

• the stuck input/composer symptom is likely broader than the original Linux-only newline buffering bug
• the server should probably reconcile turn completion from provider snapshot state when realtime lifecycle messages are missing

If this direction sounds right, I can turn the patch into a cleaner PR or split it into smaller steps.

[cyppe]

Follow-up after testing a local fork build: I found a second contributor to the same stuck-composer symptom on top of the server-side turn/completed recovery problem.

There is also a frontend race in apps/web/src/components/ChatView.tsx around local sendPhase state:

• isWorking is derived from phase === \"running\" || isSendBusy || isConnecting || isRevertingCheckpoint
• isSendBusy stays true while sendPhase !== \"idle\"
• sendPhase was only being cleared after the UI observed one of:
  • phase === \"running\"
  • a pending approval
  • pending user input
  • a thread error

That leaves a race where a turn can start and complete quickly enough that the UI never renders the intermediate running phase. In that case:

• the latest turn may already exist and be completed
• the session may already be back to ready
• but local sendPhase never resets
• so isWorking remains true and the composer appears blocked/stuck

So the broader symptom seems to have at least two causes:

1. server-side lifecycle reconciliation is needed when turn/completed is lost/delayed
2. client-side send state should also self-clear once the UI has observed a newer latest turn for the current send

I patched this locally by resetting sendPhase when the current send has been superseded by an observed latest turn (based on latest turn timestamps vs sendStartedAt).

So I still think the server-side proposal in this issue is useful, but it likely needs to be paired with a small frontend guard as well to fully eliminate the stuck input/composer symptom.

[cheruvian]

Sanitized corroborating evidence from a Codex workload on T3 Code 0.0.32:

• Three projected provider sessions remained running with a non-null active turn even though the corresponding turn records already had completion timestamps and had produced no activity for roughly an hour.
• A fourth session remained starting with no active turn after its checkpoint was captured and then went idle.
• The corresponding provider processes were sleeping at approximately 0% CPU.
• The last lifecycle events still reported running / starting; no later ready or stopped transition was present.
• These were high-activity turns: individual sessions recorded 296–967 tool starts (with nearly balanced completions) and 10–46 task starts before becoming stale.
• Projection cursors were caught up, so this did not appear to be merely an unapplied event backlog.

This looks consistent with the proposed reconciliation path here: the provider completed or became idle, but the final lifecycle transition was lost or never ingested under load. A bounded reconciliation against provider state would recover these records without requiring direct database edits or terminating unrelated sessions.

Related pressure amplifier: #5719 documents per-delta projection work that can substantially increase event/transaction volume during these high-activity turns.

No thread IDs, message content, raw logs, usernames, or filesystem paths are included.