Skip to content

[Bug] Streams reconnect loop: watchdog tears down a connected transport because its meta room is still joining #399

Description

@zxch3n

Affected area

Session / workspace / local sync

Installation method

Built from source

Lody version or commit

lody-oss @ 48249c9, CLI 0.89.3

Operating system

macOS (arm64)

Agent or runtime

N/A — Loro Streams transport layer

What happened?

Under a degraded network the daemon entered a sustained Streams reconnect loop: 3–11 transport reconnects per minute for over ten minutes, on a workspace that kept briefly reaching connected.

The loop is self-sustaining because the recovery action destroys the thing it is waiting for. The watchdog tears down a transport that is already connected because the meta room has not finished joining, and reconnecting the transport puts the meta room back into reconnecting:

13:09:24.700  Loro streams transport status: connected          <- already connected
13:09:25.419  Triggering Loro streams reconnect
                (reason=watchdog, health=recovering,
                 transport=connected,                           <- torn down anyway
                 metaRoom=reconnecting)                          <- only because the room is still joining
13:09:25.421  Scheduling Loro streams reconnect in 4654ms (attempt=3)
13:09:25.946  Loro streams transport status: connected
13:09:29.821  Loro meta room ready in 892ms                     <- the join needs ~900ms
13:09:30.075  Loro streams transport status: connecting         <- torn down again

Transport entries into connecting, per minute:

13:00   6     13:03   4     13:06   8     13:09   3
13:01  10     13:04   8     13:07   5
13:02  10     13:05  11     13:08   3

When the meta-room join takes longer than the scheduled backoff delay — i.e. exactly when the network is bad — the loop cannot escape on its own.

The observed attempt counter never rises past 2–3 and delays stay at ~2.1s / ~4.6s, so the flap-aware backoff documented in apps/cli/src/lib/loro/AGENTS.md is not biting on this shape:

health that does not survive LODY_LORO_HEALTH_STABILITY_WINDOW_MS (5s) counts as a failed recovery and charges the attempt counter instead of resetting it

resetReconnectBackoff (apps/cli/src/lib/loro/connection-recovery.ts:754) does implement the stability window and re-checks isStreamsHealthy() in its timer callback, so the defect looks to be one level up: the health signal that drives the reconnect decision aggregates the meta-room status, while the remedy acts on the transport.

Downstream this is what widens a brief outage into a long window in which anything requiring a confirmed Flock sync fails.

What did you expect?

A room that is still joining should be retried at the room level, with its own backoff. A transport that is already connected should not be torn down because a dependent room has not finished joining — that restarts the join and can prevent it from ever completing.

How can we reproduce it?

  1. Connect the daemon to a workspace.
  2. Introduce sustained packet loss / latency on the Streams connection such that a meta-room join takes longer than the reconnect backoff (observed join time was ~900ms on a healthy hop; degrade until it exceeds the ~2s first backoff).
  3. Watch ~/.lody/logs/<date>.log for Loro streams transport status transitions.
  4. Observe repeated Triggering Loro streams reconnect (… transport=connected, metaRoom=reconnecting) and a reconnect rate that does not decay.

How often does it happen?

Sometimes

Relevant log output

2026-09-04T13:09:23.242Z [DEBUG] [W:<ws>] Triggering Loro streams reconnect (reason=watchdog, health=recovering, transport=connecting, metaRoom=reconnecting)
2026-09-04T13:09:23.245Z [DEBUG] [W:<ws>] Scheduling Loro streams reconnect in 2172ms (attempt=2, reason=watchdog, health=recovering, transport=connecting, metaRoom=reconnecting)
2026-09-04T13:09:24.700Z [DEBUG] [W:<ws>] Loro streams transport status: connected
2026-09-04T13:09:25.419Z [DEBUG] [W:<ws>] Triggering Loro streams reconnect (reason=watchdog, health=recovering, transport=connected, metaRoom=reconnecting)
2026-09-04T13:09:25.421Z [DEBUG] [W:<ws>] Scheduling Loro streams reconnect in 4654ms (attempt=3, reason=watchdog, health=recovering, transport=connecting, metaRoom=reconnecting)
2026-09-04T13:09:29.821Z [DEBUG] [W:<ws>] Loro meta room ready in 892ms (reason=meta-room-joined)
2026-09-04T13:09:30.075Z [DEBUG] [W:<ws>] Loro streams transport status: connecting
2026-09-04T13:09:33.930Z [DEBUG] [W:<ws>] Resetting Loro streams reconnect backoff (reason=meta-room-joined, attempts=2)

The machine RPC read loop was also down for ~2 minutes in the same window:

2026-09-04T13:06:44.574Z [INFO] [W:<ws>] [rpc-server:<machine>] request loop recovered after 123415ms (consecutiveFailures=1 …)

Additional context

Existing regression coverage for the reconnect-storm class is apps/cli/tests/reconnect-storm-repro.test.ts; it does not appear to cover this shape, where the transport is healthy and a dependent room's join is what keeps aggregate health at recovering.

Suggested direction:

  1. Separate the remedy from the aggregate: reconnect the transport only for transport-level failure; retry a room join at the room level.
  2. Give the room join its own backoff, and do not let a scheduled transport reconnect fire while a join attempt is in flight and making progress.
  3. Extend the repro test with a "transport connected, room join slower than backoff" case.

Related: with this window open, session creation fails outright (filed separately), and the resulting errors are misclassified (also filed separately).

Before submitting

  • I searched the existing issues and did not find a duplicate.
  • This report concerns an open-source component in this repository, not a hosted service, Web or mobile app, account, or billing issue.
  • This is not a security vulnerability; security reports follow the repository's security policy.
  • I removed credentials, private source, conversations, prompts, personal data, and other sensitive information.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions