Skip to content

relayfile listen: event stream gets 429 workspace_busy and silently drops events under workspace load #496

Description

@khaliqgant

Summary

relayfile listen cannot hold an event stream when the workspace's durable
object is under load: the websocket handshake returns HTTP 429 and the
listener drops into an endless reconnect/backoff loop, so file.created (and
other) events are never delivered — the listen→--run automation silently
never fires. Reads on the same workspace also 429 intermittently.

This makes the headline "Linear issue → agent → PR" quick-start unreliable
whenever anything else is loading the workspace.

Evidence (listener log)

Listening on /linear/issues/by-uuid/** (file.created) — Ctrl+C to stop
listen: stream error (event stream error: failed to read JSON message: failed to get reader: failed to read frame header: EOF); reconnecting in 1s
listen: stream error (failed to WebSocket dial: expected handshake response status code 101 but got 429); reconnecting in 2s
listen: stream error (failed to WebSocket dial: expected handshake response status code 101 but got 429); reconnecting in 4s
... (exponential backoff continues; no events ever delivered)

Direct probe confirms the workspace-level cause:

$ relayfile status
error: http 429 workspace_busy: workspace durable object is busy; retry after the advertised delay

The create half works fine (a writeback push created a real Linear issue in
ready-for-agent, pending:0), so the failure is specifically the event
stream / read
path under contention, not auth or the write path.

Two distinct problems

  1. 429 fragility. The event-stream websocket doesn't tolerate a busy
    workspace DO — it can't get a handshake slot and never recovers on its own
    while load persists. A single busy workspace should not make the documented
    listen loop undeliverable.
  2. No visibility / no honest failure. From the user's side the loop just
    "doesn't fire." There's no surfaced "events are not being delivered because
    the workspace is rate-limiting you" state, and no --retry-after handling
    that actually succeeds once the advertised delay passes.

Root trigger in this instance was a runaway full-tree mount bootstrap saturating
the same workspace DO (see companion issue), but the listener's inability to
degrade gracefully / recover is its own defect: any concurrent load reproduces
it.

Expected / suggested direction

  • The listener should honor the server's advertised retry-after and reliably
    re-establish the stream once the DO is free, rather than backing off into a
    state where events are dropped.
  • Consider decoupling the event-delivery path from the same durable object
    that mount bootstraps saturate, or give event streams a reserved rate budget.
  • Emit a clear, user-visible diagnostic when the listener is up but not
    receiving events due to 429
    (distinct from "Listening on …"), so an operator
    knows the automation is not actually armed.

Acceptance criteria

  • With the workspace under moderate concurrent load, relayfile listen still
    delivers file.created events (possibly delayed) instead of silently dropping
    them.
  • When events cannot be delivered, the CLI says so explicitly.

Context

relayfile 0.10.57. Surfaced verifying the Lumina "Linear → agent → PR" quick-start end to end.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions