Skip to content

Full-tree mount (remoteRoot: /) never converges on large subtrees; stuck bootstrap saturates the workspace DO (429 for everyone) #495

Description

@khaliqgant

Summary

A full-tree relayfile mount (remoteRoot: "/") over a workspace that contains a
very large subtree never finishes its initial bootstrap. It gets stuck
enumerating the subtree, retries forever, freezes lastSuccessfulReconcileAt,
and — because it keeps hammering the workspace — saturates the workspace's
durable object, causing workspace-wide 429 workspace_busy
for every other
client (reads, other mounts, and relayfile listen event streams).

Observed on a real mount that had been stuck for 814 hours (since
2026-08-11).

Evidence

  • Mount config (.relay/state.json): {"workspaceId":"rw_7ccfea89","remoteRoot":"/","mode":"poll"}.
  • integrations-mount.err.log repeats identically on every restart:

    mount bootstrap terminal failure: bootstrap stalled for 31 consecutive checkpoint-stable cycles (limit 20, path "/github/repos/AgentWorkforce/relay/deployments/4952144869", … 5454 directories pending)

  • relayfile workspace status --json shows outboxPending: 3994 (the bootstrap's own never-completing mirror writes) while writeback status shows pending: 0.
  • The supervising launchd agent (com.agentworkforce.chief.integrations-mount → relayfile restart <ws> --foreground) re-enters the same loop on every restart; killing the process just respawns it, identical failure.
  • Side effect measured: with this mount running, relayfile read, relayfile status, and relayfile listen's websocket handshake all intermittently return http 429 workspace_busy: workspace durable object is busy. Stopping the bootstrap clears it. (See companion issue on the listen/429 fragility.)

Impact

  • The mount never serves fresh data (its local mirror is weeks stale).
  • It degrades the entire workspace for all other relayfile clients via 429s — not just itself.
  • No amount of restarting recovers it; there's no convergence.

Expected / suggested direction

  • Bootstrap of a large tree must converge (paginate/stream the big subtree, or checkpoint partial progress) instead of failing the whole mount after N stable-but-incomplete cycles.
  • A single mount's failing bootstrap should not be able to saturate the shared workspace durable object and 429 unrelated clients — bound its request rate / isolate it.
  • Provide a supported way to scope a mount (include/exclude globs, e.g. exclude /github/repos/*/deployments/**) so a full-tree mount can avoid pathologically large subtrees. remoteRoot: "/" with no exclusion is a foot-gun on big workspaces.
  • Surface the stuck state in relayfile status as an actionable error (it currently just says mount: stalled).

Acceptance criteria

  • A full-tree mount over a workspace containing a multi-thousand-dir subtree reaches a successful first reconcile (or degrades to a scoped mount) rather than looping forever.
  • A failing/bootstrapping mount cannot push the workspace into workspace-wide 429s for other clients.

Context

relayfile 0.10.57. Surfaced while verifying a Linear→agent→PR listen loop for a prospect; this mount's 429s blocked the loop's event stream.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions