Broker manual_flush wedges a live worker behind a non-ACKable receipt gap

[kjgbot]

Observed production failure

On 2026-09-22, the live babysitter-factory-unblock worker on finn-mini became unreachable for new input while its PTY remained alive and view attach worked.

agent-relay node agent message flush babysitter-factory-unblock --state-dir .../finn-mini-node/state
{
  "flushed": 0,
  "deadLettered": 0,
  "held": 13,
  "blockedReason": "delivery sequence 90 for 'babysitter-factory-unblock' is not the next ACKable receipt"
}

message auto --node finn-mini returned mode: manual_flush, flushed: 0 (revisions 31 and 32). Both drive and passthrough attach failed with "broker remained in manual_flush mode while draining queued messages". Fleet still reported the worker live; no broker restart or agent release was performed. This blocks the operator from delivering the next Babysitter task even when the worker is idle.

Likely boundary to investigate

crates/broker/src/runtime/fleet.rs::flush_pending_relay_messages stops at the head receipt when FleetDeliveryBook::receipt_ackability returns Blocked. crates/broker/src/node_control.rs::receipt_ackability requires receipt.seq == acked_up_to_seq + 1 and seq <= received_up_to_seq. Determine whether sequence 89 was lost, awaiting a late ACK, or cursor/receipt state diverged. Do not simply skip or ACK sequence 90: preserve identity and cumulative-ACK safety.

Acceptance

• Deterministic regression for a held queue with a missing predecessor/receipt gap, including auto, flush, and attach behavior.
• Broker recovers after replay/cursor reconciliation, or gives an actionable terminal failure/DLQ path without keeping every later message held forever.
• Expose the current ACK cursor, head sequence, and gap reason in operator status without secrets.
• Verify a live worker can receive a new instruction after recovery, without restart or identity handover.

Related but not identical to #1559 and #1593: here the messages are visibly held and the flush endpoint reports an explicit ACKability gap.

[kjgbot]

Second live 12.4.0 repro on kjg-lap (2026-09-22): github-install-pending-rca is still a live worker, but its broker remains manual_flush revision 30. Local node agent message flush returned flushed:0, deadLettered:0, held:3, blocked on delivery sequence 79 not being the next ACKable receipt. Remote drive attach cannot become ready. This is in addition to finn-mini sequence 90 / 13 held. Do not restart either active broker or discard their pending messages; the same deterministic recovery must cover both.

[kjgbot]

Read-only correction at 2026-09-22 13:48 UTC: finn-mini broker PID 75086 and babysitter-factory-unblock PTY are alive. The current worker pending queue is empty by both /health pendingDeliveryCount=0 and the authenticated spawned/pending API; delivery cursor received=acked=106. There are zero dead letters for this worker. The earlier 13 held at sequence 90 was a historical snapshot, not a current backlog; held_for_manual_flush is cumulative. Worker mode remains manual_flush. Fleet node delivery is still offline because its WebSocket repeatedly reconnects and disconnects while heartbeat advances; that transport fault is tracked in #1591. This does not prove a safe recovery procedure for the original gap. Do not skip ACKs, flush, switch mode, or restart based on the stale seq-90 observation. Preserve state and investigate the WebSocket app-ack/reconnect cause before a controlled handoff.

[kjgbot]

Read-only kjg-lap correction at 2026-09-22 13:51 UTC: its fleet node is currently ONLINE with handlers live and two workers. github-install-pending-rca has pending=0 and cursor received=acked=277. babysitter-session-lineage-proposal has six current manual_flush pending messages, cursor received=157 and acked=151; neither worker has dead letters. The earlier 3-held/seq-79 snapshot was stale. Do not assume the generic /health pendingDeliveryCount=0 means the manual_flush queue is empty: the lineage worker still holds six. No mode, queue, ACK, or broker state was changed; leave the six deliveries for the lineage owner to inspect under a controlled procedure.