Relay silently drops fan-out frames under send-buffer backpressure; connected clients never recover the missed event #25

Summary

When a connection's outbound queue is full, the relay drops the fan-out frame for that subscriber and keeps the connection open. The event is already stored and the publisher has its OK, so nobody notices: the connected client never receives the event, and nothing tells it to fetch it.

This is independent of the background result delivery contract (#18). A canonical report that was accepted and stored can still be missed by a client that is connected at the time.

Mechanism (fork main @ 9b76dbeb)

1. state.rs:1120: try_send_ws_message handles TrySendError::Full by logging a warning and dropping the frame. The connection is cancelled only after BUZZ_SLOW_CLIENT_GRACE_LIMIT (default 15) consecutive failures, and any successful send resets the counter. A dropped frame is not kept for a later retry.
2. event.rs:373: OK means the event was durably accepted. Fan-out runs afterwards in a separate task, and no receipt from subscribers is involved.
3. ingest.rs:3505 and delivery.rs:421: re-publishing an already accepted event, or retrying messages deliver for a confirmed key, returns the stored result without a new fan-out. Retrying on the publisher side cannot repair the gap, and choosing a different report would break the canonical-report contract.
4. Recovery after reconnect is not guaranteed. Desktop replays channel subscriptions with a bounded lookback; mobile and buzz-acp replay from their last-seen cursor. Newer events that did arrive move that cursor past the gap.

The queue holds BUZZ_SEND_BUFFER frames per connection (default 1000), shared by all subscriptions on that connection. The queue fills when a client stops reading (a hung or sleeping app) or during bursts, such as concurrent history REQs on one connection (see block#8196).

Impact

• Humans (desktop, mobile): a message or agent report is missing from the timeline until a reload that happens to cover it.
• Agents (buzz-acp): a missed mention is silently not answered. The next turn in the thread fetches context over REST, so a later mention brings the missed message back into context, but the missed trigger itself starts no turn.

Related upstream work

• fix(relay): emit BUZZ_SYNC_REQUIRED gap signal on fan-out backpressure drops block/buzz#6090 (open) adds a ["BUZZ_SYNC_REQUIRED","backpressure"] signal and a dropped-frame metric on the relay side. It does not include client recovery. In this fork, desktop and mobile ignore unknown relay messages, buzz-acp logs a parse error and continues, and the shared Rust WS client returns UnexpectedMessage for an unknown head (message.rs:164). So porting the signal alone does not recover a single gap.
• REQ history stops partway with no EOSE or CLOSED when the per-connection send buffer fills (two concurrent REQs) block/buzz#8196 (open): the same queue and the same drop path cut REQ history short with no EOSE/CLOSED.

Proposed scope

A separate change, not bundled with the session-continuity work (#6/#7, PR #15).

1. Relay: detect a dropped frame per subscription and signal the client in a backward-compatible way. Count drops without sampling.
2. Clients (desktop, mobile, buzz-acp, shared Rust WS client):
   • handle the signal and keep a lower bound of the gap that newer events cannot move;
   • re-read the missed range from history under current authorization, deduplicated by signed event id;
   • coalesce repeated signals into one bounded repair with backoff;
   • if repair is impossible, show an explicit degraded or reconnect state rather than silent success.
3. The repair re-reads the same stored events (same ids, same completion keys). It must never re-publish, pick another report, or resume a provider session.
4. Interim option: close the connection on the first dropped frame, so existing reconnect replay kicks in. Only with documented limits, because replay windows are bounded.

Acceptance criteria

• Accepted canonical report → queue Full while the connection stays open → the client detects the gap and ends up with exactly one copy of the original event id.
• Newer events arriving after the gap do not erase its lower bound.
• Repeated signals, a full control queue, or an access change during repair cause no unbounded repair loop, no duplicates, and no data the client is not allowed to read.
• Old clients that do not understand the signal keep working (no protocol errors on subscribed connections).

Notes

The mechanism above was verified against the source and with an isolated local check of the unchanged queue function. Latency and drop numbers reported by third parties were not reproduced here.