PTY delivery: clear blank-composer residue holding failed-draft latch; dead-letter deliveries stuck behind blocked_on_send

[khaliqgant]

Incident (2026-10-19)

Two broker PTY workers had relay messages stuck for days. desktop-upgrade-sf (sf-mini) held 3 deliveries and desktop-upgrade-finn (finn-mini) held 9, for 42-56h, while each Claude sat idle at an empty prompt.

• /api/status showed about 511-675 resends per delivery, with failed_attempts always 0 and no last_error.
• /api/node-delivery showed surfaced_and_acked: 0 since spawn, and the worker state was blocked_on_send.
• The operator flush (POST /api/spawned/<name>/flush) did nothing (flushed 0, held 0).
• The live snapshot had the cursor at [22,3] on a blank row directly under the prompt row ❯ .

Root cause

1. Prompt not recognised. Claude paints its prompt as ❯ + U+00A0. composer_candidates (broker/delivery_verification.rs is_prompt) only accepted ❯ or ❯ with an ASCII space. As a result current_composer returned None and composer_is_idle was false, even for a truly empty composer. failed_draft_released (pty_worker.rs latch release, ~3113) could therefore never succeed.
2. Blank-row residue. The failed injection left a line break in the composer, so the cursor sat one row below the prompt. That state is not idle either, and nothing would ever clear it.
3. Re-sends reset the failure count. Every 5 minutes (WAIT_DELIVERY_ACK_TIMEOUT) the broker re-sent each held delivery. The worker dropped it as a "duplicate pending delivery" (pty_worker.rs ~1557). Because the handoff succeeded, retry_pending_delivery (runtime/delivery.rs) reset failed_attempts to 0. Nothing ever dead-lettered.

Fix

• is_prompt treats U+00A0 as a space.
• New blank_composer_residue_rows / blank_composer_residue_cleanup. While a failed-draft latch is held and the composer is a bare prompt followed by only blank rows (at most 4, closed by the composer frame rule, nothing at or after the cursor), the worker sends one backspace (0x7f) per blank row. This is bounded to MAX_RESIDUE_CLEANUPS = 3 tries, spaced 2s apart. It never fires when any text exists in the composer, so a human draft is never erased.
• retry_pending_delivery: a successful handoff to a worker whose state is BlockedOnSend no longer resets failed_attempts. It increments the count and sets last_error = "worker blocked_on_send; delivery not injected", so MAX_DELIVERY_RETRIES dead-letters the delivery after about 10 resends. Working workers keep the old behaviour, so a long turn never dead-letters wait-mode messages.
• If residue cleanup is exhausted, the existing failed_draft_latched escalation fires. The latched delivery was already reported failed when it was latched, and the broker budget above then dead-letters the queued ones.

Tests

• blank_composer_residue_holds_the_failed_draft_latch_until_cleared: the real 24-row screen from the sf broker. With the residue it is not idle, residue = 1 row, cleanup = [0x7f]. Once cleared it is idle and the latch releases, which also covers the U+00A0 prompt.
• blank_composer_residue_never_matches_a_composer_holding_text: text on the cursor row, text on the prompt row, and text after the cursor all return None.
• residue_cleanup_is_bounded_and_spaced.
• resend_to_blocked_on_send_worker_counts_against_retry_budget_and_dead_letters and resend_to_working_worker_still_resets_failed_attempts.
• Targeted runs: 63 passed (delivery_verification, residue, retry, latch) and 104 passed / 3 ignored (pty_worker, runtime delivery, composer, injection). cargo fmt is clean.
• clippy -D warnings on toolchain 1.97 fails only on two pre-existing lints in node_control.rs (large_enum_variant, result_large_err), which this PR does not touch.

Open finding (not fixed here)

When the residue was cleared live by hand (POST /api/input/<name> with one \x7f), the latch released, and then every queued delivery (3 on sf, 9 on finn) failed with prompt_unproven. A fresh message afterwards worked.

• The suspected stale-deadline cause is not it. prompt_wait_started is set when the injection is popped (pty_worker.rs ~2327), and prompt_unproven has a single source (pty_worker.rs ~2413) that measures from it.
• The real cause is unconfirmed without the worker logs. Candidates:
  • The manual /api/input write raised the human-ownership / interactive-hold state while queued injections were popped. The window then elapsed while the hold-gated delay arm could not run.
  • A sticky auto_suggestion_visible sent Escape into Claude's empty composer, moving the cursor off the prompt row.
• Worth a follow-up with RUST_LOG=agent_relay::worker::pty=debug.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Agent Relay sessions

• claude session a6d60c2d · opened via gh pr create · last active 2026-10-20

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[khaliqgant]

CI note on the Targeted feature verification (Flows) failures

The workflow has been re-run 5 times on 16a9260 and failed a different shard almost every time. I checked whether this PR causes any of them:

• messaging-mcp-process-roundtrip (shard 1, 2 of 4 real runs): the timeout happens inside spawnPty (60.8s), while the test's cat agents register with Relaycast. This is before any message is sent, and none of this PR's paths (prompt detection, blank-composer residue cleanup, BlockedOnSend retry accounting) run then. The same step fails the same way on feat/workflow-file-surfaces, which has no broker changes (run 39369046593). Other branches fail it with "Relaycast publish timed out after 20000ms" (39361816288) and a 503 "database temporarily overloaded" (39096836282).
• Local repro of messaging.test.js with release builds: this branch passed 4 of 5 runs and main passed 3 of 4. The only failures were cast.agentrelay.com errors and publish timeouts, one on each side.
• Other shards: one run failed on a 503 from the Relaycast database, and one failed broker-local-only ("Absent local work must not be dead-lettered"). The latter goes through the local_* reconnect branch, which runs before this PR's change, and passed on the other runs of the same commit.

Conclusion: these are existing Relaycast-dependent flakes, not regressions from this PR. All Rust and unit checks pass.

🤖 Generated with Claude Code