[factory] Deliver briefs and messages over MCP instead of simulated keystrokes

[khaliqgant]

Summary

Agent briefs and agent-to-agent messages are delivered by simulating keystrokes into a PTY. That forces the broker to decide when a third-party TUI is "ready for typed input", which it can only do by scraping the vendor's screen. On 2026-08-15 that heuristic stopped matching and the entire fleet silently stopped accepting work for hours.

Deliver over the agent-relay MCP channel the agent already holds, and readiness detection stops being load-bearing.

What happened, so the motivation is concrete

claude_grid_ready (crates/relay-pty/src/readiness.rs) requires the literal string "Welcome back" or "Welcome to " on screen:

let has_welcome = grid.screen.contains("Welcome back") || grid.screen.contains("Welcome to ");
has_welcome && claude_prompt_row(grid)

Claude Code does not render that greeting on every launch. Three consecutive live launches on finn-mini showed a composer and no banner, so the gate returned false every time → worker_ready never fired → initial_tasks was never released.

The symptom is an agent that looks completely healthy and does nothing: it boots, registers, holds a brief it is never handed, sits at 0% CPU, and writes no session transcript. The objective was present and correct on the agent record the whole time — 7371 characters of it. Injected relay messages ride the same path, so agent-to-agent DMs died with it.

relay#1529 restores a bounded timeout so this degrades instead of hanging. That is a safety net, not the fix.

Why the current design cannot be made reliable

1. It depends on vendor cosmetics. Any upstream change to a greeting, a what's-new panel, or a composer glyph can strand every agent. We do not control, and are not notified of, those changes.
2. Failure is silent. The one diagnostic is tracing::warn!(target: "agent_relay::worker::pty", ...), and no broker log file exists with RUST_LOG unset on these nodes — so the outage produced no observable output anywhere.
3. The tests structurally cannot catch it. ->pty:ready short-circuits readiness before any heuristic runs. It appears three times in the repo: the check at readiness.rs:28, a unit test, and tests/e2e/fleet/nodes/stub-agent.cjs:70 which writes it. Nothing in production emits it — only the E2E test double. So every fleet E2E run takes the reliable path while production takes the fragile one, and claude_grid_ready is never exercised under test. "Fleet E2E is green" currently carries no information about readiness detection.
4. It is unevenly applied. claude and gemini early-return on a pure screen heuristic with no fallback; codex and the generic set fall through to total_bytes > 500, which is why they kept working. Gemini has the identical latent hazard.

Proposed direction

The broker already spawns each harness with --mcp-config pointing at its own agent-relay mcp server, and that server already exposes check_inbox. So a relay-owned, vendor-independent delivery channel exists and is connected on every spawn — it is simply not used for delivery.

Sketch, for the implementer to challenge rather than follow:

• On MCP initialize, have the server report to the broker that the harness is up. That is a real handshake rather than a string match. Note the honest limit: it proves the process booted and is loading config — roughly what harness_started already proves — not that a TUI accepts keystrokes. It is a better trigger, not a complete answer.
• Deliver the initial task and subsequent messages as MCP-visible inbox items rather than typed keystrokes, so delivery never depends on terminal state at all. This is the part that actually removes the failure class.
• Keep PTY injection as a fallback for harnesses without an MCP channel.
• Decide explicitly what happens for a harness that never connects MCP, and make that path loud.

Definition of done

• A brief reaches a spawned agent without any dependency on screen content. Demonstrate by making detect_cli_ready return false unconditionally and showing the agent still receives and acts on its task.
• Agent-to-agent DM delivery demonstrated the same way, measured at the recipient with a unique marker grepped from the recipient's own transcript — never from the sender's receipt. Sender-side receipts are exactly what hid relay#1523 for a day.
• A test that exercises the real path. If the fleet E2E stub keeps emitting ->pty:ready, add coverage that does not, or the suite keeps passing while production is broken.
• State plainly which harnesses are covered and which still fall back to PTY injection.

Notes for whoever picks this up

• Do not treat relay#1529 as having fixed this. It bounds the damage; the coupling remains.
• The ->pty:ready marker is a designed escape hatch that only a test stub uses. Either make it real or stop pretending it is a production path.
• Read relay#1529 and its review thread first — two reviewers caught a genuine hole in the first cut, where a timeout would have released a brief into an unanswered Codex trust prompt. Distinguishing "unrecognised prompt" from "deliberate veto" matters in any redesign.

[khaliqgant]

Fresh measurement, 2026-08-16 — the initial-task path worked and the DM path did not

Adding evidence rather than a new issue, because this is the same failure class and a second ticket would split it.

Context. relay PR #1536 sat with four unresolved bot review threads for 82 minutes. The lane that owned it had finished and gone idle; the reviews landed at 12:57:11Z and 13:00:38Z, minutes after it stopped. Nothing pushes a GitHub review into an agent session, so it never looked again — reasonable. What was not reasonable is that it could not be told.

What was measured.

• Two DMs to relay-1535-impl-sfmini-0816, one in wait mode (13:45Z) and one in steer mode (14:24Z). Both returned a real messageId at exit 0. Both stayed queued_or_unread.
• The agent was provably alive and provably idle throughout: a cross-node view attach from chief-broker rendered its live screen twice, ~35 minutes apart, showing it parked at a prompt after Worked for 1h 25m 31s. The two screens were byte-identical. So this is not "busy and hasn't got to it" — wait mode's documented semantics are inject at the next safe idle boundary, and the recipient was sitting on one.
• Across eight recipients this afternoon, exactly one read receipt came back (factory-lead, 12:57:43Z). The rest are still queued_or_unread, including a second resident on the same node as the one that read.
• The agent's own last screen line read: No merge performed. Chief's DM recipient resolved; reading remains unconfirmed. It knew its report might not have landed. Consistent with The PTY path's turn signal is self-declared "inferred": no proof a model consumed a delivered message #1475 — a resolved recipient is not a consumed message.

The control that makes this useful. The same brief, delivered as a fresh spawn's initial task instead of a DM, worked first time: relay-1536-threads-0816 came up in the right worktree on the right branch and shipped c73569e1c — all four review threads addressed, with a must-fire/must-not-fire test pair — inside about ten minutes.

So on this evidence the failure is specific to injecting into an already-running session, not to task delivery in general. That is a narrower and more testable claim than "delivery is broken", and it points at the same screen-scraped readiness gate this issue describes rather than at the queue.

What I am not claiming. queued_or_unread conflates undelivered with delivered but unread, so for the busy recipients this evidence cannot separate the two. The idle recipient is the discriminating case, and it is the one that should be reproduced first. Please treat the eight-recipient figure as a floor on a delivery problem, not as a measurement of one.

Operational cost, for prioritisation. With DMs unreliable, the working escalation path today is: view-attach to read an agent's actual screen, then replace the agent rather than message it. That works, and it throws away a live session's context every time — see #1387, which asks for exactly the injected-but-unprocessed signal that would have made this visible in seconds instead of 82 minutes.

[khaliqgant]

Owner assigned — relay-1531-mcp-delivery-0816

Recording ownership here rather than only in a DM, because a contract that lives in a mailbox dies with the mailbox — which is the same class of problem this issue is about.

Placed 2026-08-16 17:57Z, working in an isolated worktree at d0e2e7b75. It is a diagnose-and-design lane, not an implement lane: it reports back before writing code.

The defect is confirmed live on main

crates/relay-pty/src/readiness.rs:80 still reads:

let has_welcome = grid.screen.contains("Welcome back") || grid.screen.contains("Welcome to ");

Why it is not being handed straight to an implementer

Two reasons, both worth stating publicly.

1. The evidence does not all point one way, and the contradiction is the most informative thing available. Alongside the failure measurement posted earlier — two DMs to a live, idle agent, one in steer mode, both queued_or_unread for 40+ minutes with byte-identical screens 35 minutes apart — the same brief delivered as a fresh spawn's initial task worked first time, and later DMs to that newly spawned agent were read and answered. And a lane booting today rendered Claude Code v2.1.233 with neither Welcome back nor Welcome to on screen, yet still received its initial task.

So the failure is not uniform, and the screen-scrape gate may not be the only path. What distinguishes a session that receives an injected message from one that does not is a better first question than the rewrite, and the lane has been told to answer it before proposing anything.

2. This change has a bootstrapping hazard. It modifies the mechanism by which every agent receives work, using an agent that receives its work through that mechanism. If it is wrong, the fleet cannot be told to stop — and neither can the lane. The brief requires, before any code: a rollback path that does not require messaging the fleet, a verification that does not depend on the channel being changed, and staged exposure rather than a single flip.

Explicitly permitted outcome: recommend something smaller

The lane has been told to say plainly if it concludes the right first move is #1387's observability (surface last-activity and injected-but-unprocessed per agent) rather than the MCP rewrite. Being able to see injected-but-unprocessed would have turned today's 82-minute blind spot into a ten-second check. That may be worth more, sooner, than replacing the channel — and it should not lose to the option that happens to be in this issue's title.

Related and to be read together rather than as separate tickets: #1387, #1310 (delivery_ack sent after the first PTY write, before echo verification — a strong candidate for why a send receipt means nothing), #1475 (the PTY turn signal is self-declared "inferred"), #1263 (dedup cache does not protect pending-delivery replay).

[khaliqgant]

Fresh recurrence measured on 2026-09-06 during Relay/Fleet qualification.

Control 1:

• node finn-mini
• worker cloud-3363-consumer-opencode-0906
• spawn invocation inv_222271025070096384 confirmed and worker remained in the node heartbeat live-name set
• wait-mode checkpoints at 08:27Z, 08:39Z, and 08:48Z produced real message IDs but no recipient reply/read confirmation
• a steer checkpoint at 08:55Z likewise remained queued_unconfirmed
• the precisely owned worker was released at 08:56Z only after repeated checkpoints

Control 2, fresh replacement:

• same node, different cheap OpenCode model
• worker cloud-3363-consumer-mimo-0906b
• spawn invocation inv_222278955344232448 confirmed completed at 08:57:25Z
• wait DM 222279187227791360 at 08:58Z and steer DM 222280092344971264 at 09:01Z both remain queued_unconfirmed while the worker remains in the node heartbeat live-name set

This reproduces the existing distinction in this issue: a successful targeted spawn and a remotely live WorkerName do not make subsequent wait or steer injection consumable. This evidence does not claim whether the replacement received its initial spawn brief; only recipient-side ACK/read for both later delivery modes is absent. The campaign is monitoring GitHub branch/PR activity as an independent control.

[khaliqgant]

Additional 2026-09-06 reproduction on finn-mini: cloud-3363-consumer-mimo-0906b was spawned through agent-relay fleet spawn with OpenCode model openrouter/xiaomi/mimo-v2-flash:free. Spawn confirmation completed, but the agent never ACKed either wait-mode DM 222279187227791360 or steer-mode DM 222280092344971264. The DM conversation contains only the two supervisor messages. A subsequent targeted node presence query reported no agents on any reachable fleet node. This proves the second worker also disappeared without observable acknowledgement or release/lifecycle result; it does not prove that the initial spawn task was delivered. The campaign is continuing through a native Cloud worker so this failure does not block the wider qualification.

[khaliqgant]

Stronger contradictory presence evidence from the same workspace immediately after the prior comment: agent-relay fleet agent list --node finn-mini --pretty reported No agents on any reachable fleet node. In contrast, agent-relay fleet nodes --all reported finn-mini online/live with handlersLive=true, activeAgents=23, a fresh heartbeat, and relay:live-agents:v1 metadata.names explicitly containing cloud-3363-consumer-mimo-0906b. Thus targeted live-agent presence and node-advertised live-agent metadata disagree at the same time. The supervisor cannot tell whether the worker is alive, dead, orphaned, or merely undiscoverable, which blocks reliable release/lifecycle accounting.

[khaliqgant]

PR #1665 exact head 0e2aec13b1a0f8774eb6a547b72bc55212b1f800 now turns the contradictory presence observation into an explicit cleanroom qualification invariant. The targeted --node --pretty operation must reconcile the exact nonce-owned worker across heartbeat live names/count, targeted and unfiltered Fleet views, direct node process inventory, and roster state. Release checks require the same views to converge through live, roster-only where applicable, and fully absent states. A targeted-empty/heartbeat-live fixture fails as intended. This is harness enforcement, not proof that the product defect is fixed; live qualification remains required.

[khaliqgant]

Additional delivery anomaly from the Agent Relay merge-gate worker: one supervisor wait-mode DM produced two worker ACK DMs with materially identical full bodies, IDs 222290657660055552 and 222290678788112384, created five seconds apart in the same conversation dm_94a1a67ce46b25c50f13af1d. Only one supervisor checkpoint message (222289447329550336) was sent. This is separate from missing delivery: the message was processed, but the observable reply was duplicated. The campaign ledger will test at-most-once user-visible delivery alongside targeted presence/release reconciliation.

[khaliqgant]

Additional live release-settlement evidence (2026-09-06): agent-relay fleet release campaign-pr-merge-gate-0906 returned only a dispatched receipt (invocation inv_222296299978346496, chief-broker) and then hung without terminal output for more than 130 seconds until interrupted. Independent checks after interruption showed the release itself had succeeded: targeted fleet agent list --node chief-broker omitted the worker, and the workspace roster reported it offline with lastSeenAt 2026-09-06T10:06:14Z, about 7 seconds after dispatch. So provider/broker release settled, but the CLI wait/confirmation path never observed or returned that terminal state. This is a false-hang in release acknowledgement, distinct from an actual failed release.

[khaliqgant]

Duplicate delivery reproduced again with a newly spawned independent worker: relayfile-463-reviewfix-0906 sent one ACK payload, but the supervisor DM history contains two byte-identical messages one second apart (222296765283225600 at 10:07:58Z and 222296766667378688 at 10:07:59Z). This is the second fresh worker in this campaign to duplicate its initial ACK, so the behavior is not isolated to the Cloud consumer worker.