Skip to content

fix(remote-runtime): retry a pane attach that fails while the runtime is unreachable - #18496

Closed
luisgonzaleznf wants to merge 2 commits into
stablyai:mainfrom
luisgonzaleznf:luisgonzaleznf/debug-remote-server-disconnect
Closed

luisgonzaleznf wants to merge 2 commits into
stablyai:mainfrom
luisgonzaleznf:luisgonzaleznf/debug-remote-server-disconnect

Conversation

@luisgonzaleznf

Copy link
Copy Markdown

Fixes #18495.

The bug

A client that starts while its paired runtime is unreachable attaches to already-open panes rather than creating them, so the failure lands in attach() rather than connect().

attach() never sets connected, so the recoverable branch in handleRemoteTerminalError calls scheduleResubscribeAfterTransportClose(), which returns at its own if (destroyed || !connected || !handle) guard. The return after it skips connecting = false, emitRecoveryState() and surfaceErrorMessage(), so getRecoveryState() reports 'connecting' — which the banner filter hides.

Result: a mounted xterm bound to nothing. No text, no error, no retry, forever — including after the runtime returns. The pty keeps its host-side dimensions (~53 cols in the reported case) precisely because no client ever attached.

connect() handles the identical error class by arming scheduleConnectRetryAfterRecoverableFailure(). That asymmetry is the bug.

Changes

remote-runtime-pty-transport.ts

  1. Arm a retry when a recoverable error arrives with no live stream. Replays the entry point the way connect() does instead of latching silently. A live stream still goes through scheduleResubscribeAfterTransportClose().
  2. Don't let a replay wipe its own recovery window. attach() cancelled recovery on entry, so each replay restarted the 250ms..30s ladder and the deadline never latched — no 'disconnected' phase, so no Reconnect button and no parked retry. Now skipped when the attach is the replay.
  3. Don't cancel recovery on a recoverable attach failure. Same reason; connect() cancels only on the gone and fatal paths.

use-remote-runtime-recovery-triggers.ts

  1. Fire pane recovery when an environment becomes reachable. The existing online and system-resume triggers both miss a tailnet or VPN returning — navigator.onLine never went false because the LAN never dropped, and there is no power transition. The store already suppresses no-op re-probe writes, so this only fires on a real transition.

Net behaviour: the pane retries for the auto-recovery window, then latches to a visible disconnected with a working Reconnect button, and auto-revives when the environment returns.

Tests

New remote-runtime-attach-failure-recovery.test.ts, mirroring the existing remote-runtime-connect-failure-recovery.test.ts (which covers the connect() side of the same class, #12684):

  • arms a retry when attach fails against an unreachable runtime
  • leaves a parked retry that revives the pane when the runtime returns, and keeps the Reconnect button working
  • still surfaces a non-recoverable attach failure instead of retrying it (control)

Verified red/green — on unpatched code the first two fail with expected 'connecting' to be 'backoff' / 'disconnected', which is the bug reproduced exactly.

new test file          3/3 pass
terminal-pane suite    428 files, 4162 tests, all pass
tsc (web project)      exit 0
oxlint (changed files) clean

What is not covered

The unit tests prove the transport state machine. They do not prove the end-to-end render — the deliberate repro (runtime unreachable → launch client → restore reachability, against a build carrying this patch) has not been run. Worth doing before merge.

🤖 Generated with Claude Code

… is unreachable

A client that starts while its paired runtime is unreachable attaches to
already-open panes instead of creating them, so the failure lands in attach()
rather than connect(). attach() never sets `connected`, so the recoverable
branch's scheduleResubscribeAfterTransportClose() returned at its `!connected`
guard: nothing armed, `connecting` latched, and neither emitRecoveryState() nor
surfaceErrorMessage() reached. getRecoveryState() then reported 'connecting',
which the banner filter hides — the pane rendered as a mounted xterm bound to
nothing, with no text, no error and no retry, and stayed that way after the
runtime came back. connect() handles the identical error class by arming
scheduleConnectRetryAfterRecoverableFailure(); that asymmetry was the bug.

Arm the same retry when a recoverable error arrives with no live stream, and
stop the replay wiping its own recovery window: attach() cancelled recovery on
entry and its catch cancelled unconditionally, so every replay restarted the
250ms..30s ladder and the auto-recovery deadline never latched — no
'disconnected' phase, so no Reconnect button and no parked retry. connect()
cancels only on the gone and fatal paths, for the same reason.

Also fire pane recovery when a runtime environment becomes reachable. The
existing `online` and system-resume triggers both miss a tailnet or VPN
returning: navigator.onLine never went false because the LAN never dropped, and
no power transition occurs. A parked pane would otherwise wait for a manual
Reconnect indefinitely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: bfaa7c20-d825-475b-9b67-7c829f9967a7

📥 Commits

Reviewing files that changed from the base of the PR and between e39cfc8 and 07e1f22.

📒 Files selected for processing (3)
  • src/renderer/src/components/terminal-pane/remote-runtime-attach-failure-recovery.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
  • src/renderer/src/runtime/use-remote-runtime-recovery-triggers.ts
🚧 Files skipped from review as they are similar to previous changes (3)
  • src/renderer/src/components/terminal-pane/remote-runtime-attach-failure-recovery.test.ts
  • src/renderer/src/runtime/use-remote-runtime-recovery-triggers.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

Remote runtime attach recovery now distinguishes recovery replays from mount-initiated attaches. Recoverable attach failures retry through the transport entry point without resetting the recovery epoch. Non-recoverable errors retain direct error handling. The recovery trigger hook observes runtime status changes and retries parked recoveries when an environment becomes reachable. New tests cover retry scheduling, recovery revival, manual and global triggers, and non-recoverable failures.

Merge Risk: ⚪ Minimal · up to 07e1f

This change adds recovery for remote pane attachment failures and retries recovery when runtime reachability returns. No concrete unresolved risk is identified for the current change.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description clearly explains the bug, root cause, implementation, linked issue, tests, and known limitations. However, it omits the required Visual Proof section and does not provide the required … Add the required Visual Proof section with before-and-after screenshots or a video. If visual proof is not applicable, write N/A and explain why. Complete the relevant testing and checklist items, including manual-test status and validation…
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the primary change: retrying remote runtime pane attachments that fail when the runtime is unreachable.
Linked Issues check ✅ Passed The changes address all coding objectives in issue #18495 [#18495]: recoverable attach retry, preserved recovery windows, visible disconnected state with reconnect support, automatic recovery after re…
Out of Scope Changes check ✅ Passed The modified transport logic, reachability trigger, and recovery tests directly support issue #18495 [#18495]. The added documentation comments are related to the changed behavior. No unrelated code c…
Full details: Description check

Explanation

The description clearly explains the bug, root cause, implementation, linked issue, tests, and known limitations. However, it omits the required Visual Proof section and does not provide the required template checklist evidence for this behavior change.

Resolution

Add the required Visual Proof section with before-and-after screenshots or a video. If visual proof is not applicable, write N/A and explain why. Complete the relevant testing and checklist items, including manual-test status and validation commands.

Full details: Linked Issues check

Explanation

The changes address all coding objectives in issue #18495 [#18495]: recoverable attach retry, preserved recovery windows, visible disconnected state with reconnect support, automatic recovery after reachability returns, and non-recoverable error handling without retry.

Full details: Out of Scope Changes check

Explanation

The modified transport logic, reachability trigger, and recovery tests directly support issue #18495 [#18495]. The added documentation comments are related to the changed behavior. No unrelated code changes are identified.

  • Fix all pre-merge checks with AI

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…unctions

Satisfies the diff-scoped docstring coverage gate; no behaviour change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one coverage gap worth closing before merge.

Reviewed changes

  • Attach-failure recovery replay — handleRemoteTerminalError now routes a recoverable error with no live stream to scheduleConnectRetryAfterRecoverableFailure(), instead of a silent scheduleResubscribeAfterTransportClose() no-op that left connecting latched with nothing armed.
  • Replay no longer resets the recovery ladder — a replayingRecoveryAttach flag skips recovery.cancel() in attach() (and its async catch) so a replay continues the same epoch rather than re-arming the deadline every attempt.
  • Environment-reachable trigger — useRemoteRuntimeRecoveryTriggers now subscribes to runtimeStatusByEnvironmentId and fires retryAllRemoteRuntimePtyRecoveriesNow() when an environment transitions to reachable or bumps connectionGeneration.
  • Tests — new remote-runtime-attach-failure-recovery.test.ts mirrors the connect-side suite and pins backoff arming, parked revival, and the non-recoverable control.

ℹ️ New environment-reachable trigger has no test coverage

The transport changes are well covered, but the hasEnvironmentBecomeReachable predicate and the store subscription it feeds are pure logic with zero unit coverage — the existing use-remote-runtime-recovery-triggers.test.ts was not extended. A regression here (e.g. a false-positive transition that fires retryAllRemoteRuntimePtyRecoveriesNow() on every status write, or a missed connectionGeneration bump) would be invisible to the suite.

Technical details
# Test the environment-reachable trigger

## Affected sites
- src/renderer/src/runtime/use-remote-runtime-recovery-triggers.ts:11-31 — `hasEnvironmentBecomeReachable` (module-private, not exported)
- src/renderer/src/runtime/use-remote-runtime-recovery-triggers.ts:47-56 — store subscription wiring

## Required outcome
- A unit test that asserts `retryAllRemoteRuntimePtyRecoveriesNow` fires on a genuine reachability transition and does NOT fire on unrelated store changes.

## Suggested approach
- Either export `hasEnvironmentBecomeReachable` and test it directly (null → non-null status, absent → present, `connectionGeneration` increase, and the no-op cases: unchanged status, diagnostics-only overlay), or extend the existing `renderHook` test to drive a real `setRuntimeEnvironmentStatus` transition against the store and assert the mock fires.
- Note the store is a module singleton; reset it with `useAppStore.setState(useAppStore.getInitialState(), true)` around the test to avoid cross-test contamination.

ℹ️ Nitpicks

  • The environment-reachable trigger is global: any environment becoming reachable unlatches or advances backoffs for all parked panes, including those bound to still-unreachable environments (where the retry just fails and re-parks). This matches the existing online/systemResumed semantics, so it is consistent, but it is slightly over-broad.

Pullfrog  | Fix it ➔ | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Documentation only — the new commit adds JSDoc blocks to replayLastTransportEntryPoint, handleRemoteTerminalError, attach, hasEnvironmentBecomeReachable, useRemoteRuntimeRecoveryTriggers, and the test's installUnreachableRuntime. Descriptions accurately match the underlying behavior; no logic changed.

The prior review's coverage-gap note for the environment-reachable trigger still stands and is not addressed by this commit, but this delta introduces no new concerns.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@luisgonzaleznf

Copy link
Copy Markdown
Author

Closing: #18495 was fixed on main by #26483, through #26308, which makes a reattach that loses contact schedule a retry instead of hanging. I re-ran this PR's regression test, unchanged, against main at d4acd11, and it passes 3/3 without this change (it failed 2/3 before). Thanks to tian2-ai for the updater trace that showed how often this was hitting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote pane renders blank forever when the client starts while its paired runtime is unreachable

1 participant