Skip to content
This repository was archived by the owner on Jul 13, 2026. It is now read-only.
This repository was archived by the owner on Jul 13, 2026. It is now read-only.

Fresh workers can lose messages during boot (no replay in _watch_event_bus) #2

Description

@odfalik

Symptom

A freshly-spawned worker can be visible in who and yet silently lose messages sent to it shortly after spawn. The send tool returns "Message sent" but the recipient never sees the message — not delayed, lost.

Discovered while debugging an unrelated worker-spawn config issue; the spawn issue was the proximate cause in that case, but this race is independently real and would bite anyone doing rapid spawn-then-send.

Root cause

Three things happen in order during a worker's intercom MCP boot (intercom/server.py:_main):

  1. _acquire_lock() — worker now appears in who
  2. ServerSession enters context, _session is set
  3. _watch_event_bus task spawned via tg.start_soon(...)

Two problems:

(a) Visible-but-not-listening window between (1) and (3). A sender that polls who, sees the worker, and immediately sends will append to events.jsonl before the worker is tailing it.

(b) No replay. When _watch_event_bus finally starts, it initializes:

# server.py:244-248
try:
    pos = _EVENTS_FILE.stat().st_size
except OSError:
    pos = 0

i.e., it starts at the current end of the file. Anything appended during window (a) is permanently lost from that worker's perspective — there's no per-recipient inbox, just a shared events.jsonl tailed from end-of-file.

Bonus: _handle_send returns "Message sent to: <name>" (server.py:392-395) immediately after _append_event writes the line. That's a bus-append confirmation, not a delivery confirmation. Callers reasonably interpret it as the latter.

Possible fixes (in increasing order of work)

  1. who reports per-agent ready state. Surface whether the agent has completed all three boot steps, so senders can wait for "actually listening" before sending. Cheap, doesn't change protocol.
  2. Replay-from-position per agent. Each worker remembers (or rediscovers) its last-seen offset on startup and replays anything since. Solves the race cleanly; requires per-agent state file.
  3. Per-recipient inbox files with ack-based delivery. Bigger restructure but solves the race and lets send return a real delivery confirmation.

I'd lean toward (1) as a quick mitigation and (2) as the real fix.

Workaround for callers today

After spawn, send a ping and treat the worker as ready only after it replies via intercom (not just appearance in who). Retry until acked.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions