Subscription recipients spawned via `integration subscribe --spawn` never wake: deliveries never injected

[AgentRelayBot]

Summary

Verified against the published agent-relay@12.2.5 (post-merge of #1756) on production: a real GitHub → relayfile → subscription → channel chain delivers events to a spawned recipient's inbox, but the agent is never woken — zero delivery_injected, pending_messages stays 0, events sit unread indefinitely.

Repro

agent-relay integration subscribe github \
  --resource AgentWorkforce/software-garden \
  --to @ghsub-watcher-2 \
  --spawn claude \
  --broker-connection <node connection.json> \
  --task "<digest/ack task>"
# then: gh issue create / gh issue comment on the repo

Observed

• integration subscribe succeeds: relayfile binding, webhook wh_…, relay subscription sub_… all created; recipient spawns, reports ready, and stands by.
• Real GitHub events land in the agent's agent-events-<id> channel ~40s after the GitHub action (verified: evt_/rev_ relayfile metadata, issue docs: add bun install + markdown docs access (#385, #386) #529 payload present).
• Agent inbox (checked with its own at_live_ token): 3 unread messages — events ARE delivered to its queue.
• But pending_messages: 0 on the broker and no delivery_injected/delivery_queued events for the worker — the engine never pushes the deliveries to the node.

Root-cause chain (partially diagnosed)

1. Every spawn's node-control agent.register fails on this broker (bound HTTP-registered agent to node (via_node) after agent.register fallback in broker logs for ALL recent spawns — including non-subscription spawns, e.g. cloud-deploy-scope, since at least Sep 17).
2. The fallback path HTTP-registers the agent (providerName="default", locationType=via_node pointing at its direct-<id> pseudo-node) then calls bind_agent_to_node, which only writes agentNodeBindings — it does not update agents.locationNodeId/providerName.
3. Delivery routing (resolveLiveLocations) joins bindings only when agents.locationNodeId = bindings.node_id; engine-side this agent still points at the dead pseudo-node → deliveries never route to a live node.
4. On a non-enrolled node it's worse: the node's providerName differs, so inventory_sync is rejected with agent_provider_conflict and the node-control socket flaps every ~30s — degrading delivery for the whole node, not just the spawned agent.

Evidence

• Broker log (chief-linuyc.log.2026-09-19): bound HTTP-registered agent to node (via_node) after agent.register fallback worker=ghsub-watcher-2 at 00:31:46, then only failed to publish agent session event warnings; zero delivery events for the agent.
• /api/spawned on chief-linuyc: ghsub-watcher-2 idle, pending_messages: 0.
• Channel agent-events-226862793946365952 contains the real GitHub event messages; agent inbox shows 3 unread.
• On a fresh non-enrolled broker (ghsub-cli-e2e, own workspace): agent_provider_conflict → "Agent "ghsub-watcher-1" belongs to provider "default"" every ~30s; engine sees node offline.

Impact

The integration subscribe --spawn path (the documented way to wire a repo → agent wake) produces recipients that can never be woken. Since the register fallback applies to all broker spawns, new spawned agents generally may be undeliverable, not just subscription recipients. Long-standing (fallback present ≥Sep 17) — possibly an engine-side regression in agent.register/provider handling rather than a CLI regression.

Notes

• The same end-to-end chain DID deliver to a worker in the selfhost proof (ghsub-selfhost-e6a8fedbd7), but that worker was spawned via HarnessDriverClient.spawn directly against the fix: preserve broker spawn custody and dispatch compiled MCP calls once #1759 candidate broker — worth A/B-testing whether register_node_agent_token succeeds there.
• integration subscribe --spawn itself worked fine mechanically (bind, webhook, spawn, readiness) — the defect is in the spawn's registration path leaving the agent undeliverable.

[AgentRelayBot]

Update — root cause refined, fix in review

A Relayflows-authored fix is up at AgentWorkforce/relaycast#443.

Diagnosis correction: upsertAgentNodeBinding already moves locationType/locationNodeId onto the bound node as a side effect — the missing piece was providerName. The fallback-bound agent keeps providerName='default' while the node's live provider is e.g. broker, so sendToProvider(node, 'default') finds no connection and no deliver frame is ever sent. Verified by reverting: location/routing assertions pass pre-fix; provider-adoption assertions fail.

The fix (engine, bindAgentToNode):

• Provider adoption — precedence: served → sole → default → keep — plus originNodeId stamping
• Live-location guard → agent_location_conflict (409); implicit direct-* and dead locations are adopted, matching reconcileInventory semantics
• The binding row + location/provider/origin move + old-binding retirement + slot refund commit as one atomic unit (runAtomicWrites: interactive tx on Node, batch on D1, loud refusal otherwise)

Evidence: 61 conformance tests (17 new: provider table, guard boundaries, delivery push to the real provider, 2×4 failure matrix), full engine suite 1136 green, regression-verified by reverting.

Systemic follow-ups (pre-existing across every reserveNodeAgentSlot caller — register, recover, inventory — not specific to this path, flagged by the review loop):

• Reservation commits before the atomic move; a failure of the compensating release can leave capacity miscounted until reconciled.
• Concurrent binds to different nodes can both pass the live-location check (TOCTOU).

Once relaycast#443 merges and deploys, the live re-verification is: integration subscribe github --resource <repo> --to @agent --spawn claude + a real GitHub event → agent wakes and ACKs.

[miyaontherelay]

Additional reproduction from a different path — plain agent-to-agent DM, no subscription involved. This may widen the scope beyond integration subscribe --spawn.

Observed 2026-09-19 01:58–02:02Z, workspace rw_7ccfea89:

time	action	result
01:58Z	send_dm → relay-delivery-lead-0914, mode: wait	queued_unconfirmed, recipientMatched: true
02:01Z	same recipient, mode: steer	queued_unconfirmed, recipientMatched: true

Recipient state throughout: current_state=idle, last_activity_ms ≈ 2,347,063 (idle ~39 minutes, so there was no active work for a "safe idle boundary" to wait behind), and pending_messages=0.

Outcome: neither message injected. The unique markers never appeared in the recipient's own session file (~/.codex/sessions/…01a09dbd…), pending_messages stayed 0, and the agent was never woken. steer — documented as "request immediate injection, which may interrupt active work" — behaved identically to wait.

This matches the signature in the report: message accepted by the server, recipient never woken, counter never increments. The difference is that no subscription, webhook or relayfile binding is involved — just send_dm between two registered agents in the same workspace and channel.

Confound, stated explicitly: this broker/CLI is agent-relay 12.1.0, while the original report was verified against 12.2.5. So this is not a clean confirmation on the same version — it may be the same defect on an older broker, or a related one. Worth reproducing on 12.2.5 before drawing conclusions from it.

Non-symptom, for contrast: workspace write admission is healthy as of 01:41Z. A spawn probe at 01:46Z registered, bound and joined its channel cleanly after relaycast-cloud #173 / #178 / #179 deployed, and these DMs were accepted rather than refused with workspace_busy. The enqueue path works; the wake does not.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F4pDiY76Rd9RNCxenw1wnz

[AgentRelayBot]

Verified fixed on relaycast 8.11.3 (production)

Deploy chain: relaycast PR #443 merged (4a7c2a78) → published 8.11.3 → relaycast-cloud PR #180 merged (ec7592d) → sst deploy --stage production completed at 03:28 UTC.

Live re-verification on the enrolled chief node (same path that originally failed):

1. agent-relay node agent spawn claude --name ghsub-watcher-4 — spawned via the HTTP register + via_node fallback bind (the exact code path this fix rewrote).
2. agent-relay integration subscribe github --resource AgentWorkforce/software-garden --to @ghsub-watcher-4 — bound.
3. Real GitHub events: opened software-garden#530 + posted a comment.
4. Agent woke and acted: worker log shows it reading issues/530 + issues/_index.json, webhook delivery woke this agent, responded in-channel, 7 events, then Standing by for further deliveries. Broker reported pending_messages: 2 mid-processing, current_state: working.

End-to-end proof complete: GitHub webhook → Nango → relayfile → relaycast → node → PTY injection → agent wake + acknowledgement. This is the exact scenario that silently failed before the fix (events reached the inbox; agent never woke).

Cleanup: subscription removed, agent released, test issue closed.

Separate defect found during verification: all PTY spawns on the chief broker were failing with failed to spawn worker because the npm upgrade to 12.2.5 swapped the binary under the running 12.2.2 process (current_exe → deleted inode). Filed as #1798. This was likely a contributing factor to watcher-3's silent death during the original test run.