[Bug]: Direct SSH: updating Orca strands all remote terminals and running agent sessions (relay version swap)

Operating system

macOS

Orca version

v1.4.180

Details

Short summary:
After an Orca app update, the first reconnect to a direct-SSH remote deploys the new version-pinned relay, and every terminal created by the previous relay is stranded — tabs come back as fresh empty shells, and running agent sessions are orphaned on the host.

What happened?

Orca deploys a version-pinned relay to the SSH host. After an app update, the first reconnect deploys the new relay, and every existing terminal still belongs to the old relay process. The result:

Each remote tab reconnects and gets a fresh empty shell instead of its previous terminal.
Running agent sessions (e.g. Claude Code) are left orphaned under the old relay, with no way to reach them from the UI — the user has to find the session IDs by hand and claude --resume them.
The old relay daemon and its shells are never cleaned up and keep running on the host indefinitely.

How can we reproduce it?

Connect to a direct-SSH remote and open a terminal running an agent session (e.g. Claude Code).
Update Orca to a release that bumps the relay version (observed on 1.4.174 → 1.4.180, relay 0.1.0+7175e0a40ea7 → 0.1.0+bd3ec370d21d).
Relaunch and reconnect.

Expected: remote terminals and agent sessions survive an app update, or at least agent sessions come back automatically.

Anything else that might help:

Two possible directions, either would solve it:

Seamless handover: during upgrade, the old relay passes its live pty fds and screen state to the new relay over a unix socket (SCM_RIGHTS), then exits. Nothing dies.
Cheaper fallback: on reconnect, when a tab's pty is unreachable because the relay version changed and the tab has a saved provider-session record, route it through the existing cold-restore agent-resume path (claude --resume) instead of spawning a bare shell — and reap the old relay after migration.

Environment: macOS client, Linux SSH host.

Cross-link: we hit the same 1.4.180 → relay 0.1.0+bd3ec370d21d swap on a macOS SSH host (Mac Mini, Homebrew Node 26).

On our side the first reconnect did not just strand old PTYs — Connect never left Deploying relay…. SSH Test succeeded. The new relay dir’s require("node-pty") / dlopen of pty.node hung with no stdout/stderr (same SHA-256 as the previous working relay’s binary; only the new path hung). Hard-linking the old natives into the new dir unblocked the probe and the new relay came up.

Wrote it up as #14830. Same trigger as this issue (version-pinned relay replace after update); different failure mode on Darwin remotes.

Root-caused while working #8585 / #13614 (PR #17821). Recording it because the mechanism is not what the title suggests, and the real fix is a wire change that should be decided deliberately rather than slipped into a bugfix.

Why reattach across an update is impossible by construction

Two independent facts combine:

The socket path moves with every build. sockFile = relayEndpointForHost(hostPlatform, remoteDir, sockName) (ssh-relay-deploy.ts:1385), where remoteDir is ~/.orca-remote/relay-<fullVersion> and fullVersion is a content hash of the build. The filename component is stable (relaySocketNameForInstanceId); the parent directory is not. So a new client looks for its relay at a path the old relay was never listening on.
Even at the same path, the handshake would refuse. relay-handshake.ts:127 compares msg.version !== launchVersion, where launchVersion is that same build content hash, and rejects with exit 42.

So the previous relay is not "orphaned by a bug" — it is unreachable by design, and its PTYs and running agent sessions go with it. remote-install-gc.ts:hasLiveRelaySocket then pins the whole old version directory forever, because its test -S sees a live socket.

What #17821 does and does not do

It stops the old relay being silently orphaned: the client no longer unlinks a socket it cannot prove is unheld, the incumbent is classified live / unverifiable / exited, one holding live work is retained rather than reaped, and a genuinely dead one stops pinning its directory.

That makes the sessions survivable and discoverable instead of silently abandoned — strictly better, but it does not give you your terminals back. Tabs still come back as fresh shells. I want to be blunt that the user-visible complaint in this issue is not fixed.

The actual fix

Two changes that have to land together:

A semantic protocol version in the handshake, not a build content hash. Today two builds whose relay protocol is byte-identical still refuse each other, purely because their bundles hash differently. Only an actual protocol change should refuse.
A version-independent socket path, so a new client can find the incumbent at all. Per docs/reference/remote-wire-compatibility.md this needs capability negotiation, since mixed client/relay versions become the normal state rather than an error — which is exactly the point.

With those, an update reconnects to the running relay and adopts its PTYs, and the reaping work in #17821 becomes the fallback for genuinely incompatible or dead relays rather than the primary path.

A third, larger option exists — SCM_RIGHTS fd handoff, so a new relay inherits the old one's PTY master fds — which would survive even a genuine protocol break. That is a substantial change across the relay's PTY layer and is worth costing separately.

Meanwhile

Relays orphaned before any of this ships already have unlinked sockets, so they are invisible to a path-based sweep and still need a one-time Reset Relay or manual kill.

Refs #17821, #13614, #8585.

Field measurement, and a consequence for how this gets fixed

A developer's live Ubuntu host, measured today:

relay directories: 11    (relay-0.1.0+<content-hash>, one per Orca build)
live sockets:      11    (exactly one per directory)
orphan relays:     19    processes, oldest 6.7 days
combined RSS:      839 MB

Eleven builds, eleven separate socket paths, eleven relays still running. That is this issue in production: each update moves the socket path, so the previous relay becomes unreachable by construction, and remote-install-gc.ts's hasLiveRelaySocket check then pins its whole directory from ever being collected — because the socket is live, just unreachable.

The part worth noting for the fix

While closing out a separate question — whether mixed client/relay versions need testing — I realised these are the same design decision seen from two sides.

relay-handshake.ts:126 refuses msg.version !== launchVersion and exits 42, which the client turns into RelayVersionMismatchError and a redeploy. Combined with the version-namespaced socket path, a new client cannot reach an old relay: different path, and refused even if it did. So mixed-version skew is not an untested risk on this transport — it is structurally impossible.

That impossibility is purchased with exactly the property this issue is about. The two are not separable: the same content-hash-in-the-path that guarantees no version skew is the thing that makes reattach across an update impossible.

So the fix has an ordering constraint. Separating endpoint identity (stable per host/user/target) from artifact identity (keep the content hash) would let a new client find the incumbent — and would, in the same stroke, create the version-skew exposure that does not exist today. Protocol negotiation is therefore not an optional extra to sequence afterwards; it is the thing that has to land with the endpoint change, or we trade a leak for a compatibility bug.

Concretely, the three identities to separate:

Artifact — which binary is installed. Keep the content hash; that part is right.
Endpoint — where a client rendezvouses. Stable per (host, user, target), not per build.
Protocol — what the two can speak. A small integer plus a negotiated extension list, bumped rarely, not derived from the build.

With those split, a version mismatch stops being impossible-by-construction and becomes a decision point where a handoff is expressible. Without the third, splitting the second is a regression.

A cheaper intermediate exists and is worth costing separately: preserve content rather than processes. Serialize each terminal's scrollback and seed a fresh shell with it on the new relay, so an update degrades to "your history is intact, the shell is new" instead of "your tab is gone." We already have pty.replay and a serialized buffer path, so this needs no handshake change at all — and it is what the state of the art actually delivers, since none of the comparable remote-development implementations preserve live remote PTY processes across a server update either.

Related: #8585 (the orphan mechanism), #8362 (193 leaked PTY master fds on the same host, which is why these orphans retain devices rather than sitting idle), and PR #17821 (stops the client unlinking a socket it cannot prove is unheld — prevents the next pile, does not clear this one).

Assessment: real, not fixable now, and the ordering is stricter than this issue states

Full analysis in the PR above (docs-only). Summary:

The mechanism is confirmed on current main, enforced at four independent points, not two: the hash-namespaced endpoint path (ssh-relay-versioned-install.ts:75), the handshake's exit-42 refusal (relay-handshake.ts:126), the PTY consumer grant re-check whose comment states the assumption outright — "client and relay ship in one build" (ssh-pty-consumer-session.ts:58-65) — and owner-claim recovery (ssh-relay-session.ts:1144).

Endpoint-first would be strictly worse than today. ssh-relay-deploy.ts:1439-1453 catches any --connect failure, including exit 42 — which is positive proof a healthy relay owns that socket — and rm -fs the socket before binding a fresh daemon. Post-update reconnects would unlink the live incumbent, turning a discoverable orphan into an invisible one; hasLiveRelaySocket then reads DEAD and GC deletes the tree under a running process with live PTYs. #17821 must merge first.

The sequencing in this issue's comment is not achievable. It says the two changes land together; they cannot. The relay that must tolerate a new client is an already-shipped artifact that will never be updated, so tolerant behavior must ship relay-side at least one release before any client uses it — and the hash path can never be removed.

Two corrections to the issue text

The relay is the only one of Orca's three compatibility regimes built this way. Runtime RPC uses RUNTIME_PROTOCOL_VERSION + capabilities; the terminal daemon uses PROTOCOL_VERSION = 36 with all 35 predecessors attachable.
The proposed "cheaper fallback" already existed in v1.4.180, the reported version (pty-connection.ts:8160 in that tag). The bare shell was not a missing path — it was buildColdRestoreAgentResumeStartup() returning null, one of seven conditions (cold-restore-resume-startup.ts:21-44). Which one is not determinable from source; the evidence that would settle it is the reporter's persisted sleepingAgents rows after the update.

Recommendation

The design this needs is already shipped in the consolidation target: daemon-v36.sock is stable across builds and namespaced by a semantic protocol version, and shouldPreserveDaemonWithLiveSessions preserves a daemon holding live sessions. orcad forks exactly that daemon. Retrofitting a second protocol-version scheme onto the relay produces the "two paths that drift" outcome this sweep hit nine times.

This is an argument for replacing the relay, not rebuilding it.

Seam if someone does take it: parseHandshakeMessage (protocol.ts:53-61) throws on an unknown type, so handshake evolution must use optional fields (precedent: endpointCredential). Order: (a) #17821, (b) relay binds an additional stable endpoint while keeping the in-dir socket, (c) client adoption one release later.

Loose thread: six source files cite docs/ssh-relay-versioned-install-dirs.md, which has never existed in git history.

Correction to my assessment above: the unlink hazard is broader than I said

While writing the design doc (#17975) I found my exit-42 framing was wrong in a way that understates the hazard.

Exit 42 is nearly unreachable. Because the socket lives inside the version dir, a daemon there was launched from that dir's relay.js and reads that dir's .version — so launchVersion always equals the client's. relay-handshake.ts:164 describes the check as defense-in-depth. The one genuinely reachable route is readLaunchVersion falling back to bare RELAY_VERSION (relay-handshake.ts:45) when .version is unreadable at launch, since 0.1.0 never equals 0.1.0+<hash>.

But the socket unlink at ssh-relay-deploy.ts:1439-1453 does not depend on exit 42 at all. The catch is untyped, so it also fires on the credential-rotation case — which involves no version change whatsoever. The hazard is therefore wider than the version-mismatch story I told, not narrower, and #17821 remains a hard prerequisite for any endpoint change.

A second detail that constrains the proposed fix

On Windows the pipe is not inside the version dir, so the relay writes .windows-active-pipe-* marker files into it purely so GC's "liveness must be discoverable from inside the directory" invariant still holds.

That makes the socket-placement ↔ GC coupling explicit rather than incidental — and it is the concrete thing that breaks if the endpoint moves to a stable path, which is what this issue proposes. Any implementation must keep liveness discoverable from inside the version dir, or gcOldRelayVersions will delete a tree under a running relay holding live PTYs.

The full scheme is now documented in #17975 (docs/ssh-relay-versioned-install-dirs.md), which six source files have been citing all along without it existing.

fix(ssh): stop orphaning live relays when an endpoint is taken over (#…

docs(ssh): record that an app update strands relay-backed terminals a…

fix(ssh): stop orphaning live relays when an endpoint is taken over (s…

Status: untouched. Documented, not fixed.

#17972 (merged today) added 8 lines to docs/reference/ssh-execution-boundary.md and zero code. It records that an app update strands relay-backed terminals as unverifiable — naming the failure, not preventing it.

Cross-version isolation is a deliberate, test-enforced invariant: ssh-relay-cross-version-isolation.test.ts asserts a v2 deploy never touches v1's install dir or socket. So the stranding follows from a designed property, not a bug.

Neither direction proposed here has shipped — no SCM_RIGHTS handoff, no version-swap-specific claude --resume routing. relay-handshake.ts:126 enforces exact version equality, so a handoff is currently impossible by construction.

One nuance worth correcting if it shaped expectations: the cold-restore resume mentioned in the docs is pre-existing and only fires when a resumable provider session was captured. Otherwise you get the bare shell this report describes.

Related and also partly open: #13614 covers the orphaned relay left behind by the same upgrade. Today's superseded-endpoint sweep now finds and logs that orphan, but deliberately will not reap one that still owns PTYs — killing it would destroy live work on a loss-of-contact inference.

docs(relay): say plainly that the protocol negotiation is not reachab…

Another data point, this time with a macOS SSH host (Apple silicon, Node 22 via nvm) and a macOS client. Seen several times in 1.4.19x–1.4.20x.

What it looks like from the user side:

After an app update that changes the relay content hash, the terminal count in the status bar drops sharply, and the agent tabs disappear from the UI or come back as empty shells.
On the host, the previous relay.js --detached --grace-time 0 from ~/.orca-remote/relay-0.1.0+<old-hash>/ is still running, with its relay.log stuck at clients=0. Its PTYs still hold live claude processes that are working normally. Nothing in Orca shows they exist.
The only ways out are both manual: find the old relay's pid and kill it (its agents die with it, and you continue them with claude -r <id>), or leave it running forever. On a laptop-class host, orphans also hold PTY slots and RAM.

Two things that would help even before the protocol / endpoint work lands:

Make retained-live-work visible outside the console. The superseded sweep already classifies the old relay correctly (ssh-relay-superseded-endpoints.ts#L107-L117), but it only reaches console.warn (L224–L227), which nobody sees in a packaged build. A line in the host panel would turn an invisible leak into an informed choice. For example: "A relay from a previous Orca build is still running N terminals (M agents) on this host — [Show] [Terminate]". Pulling pty.listProcesses from the old relay isn't possible across the handshake, but the sweep already has pid, age and child count from the incumbent probe.
Avoid a second live agent on the same transcript. When the cold-restore path resumes an agent onto the new relay while the old process is still alive under the stranded relay, you end up with two claude processes on one session. [Bug]: Updating Orca strands SSH relays and duplicates remote agents #21215 reports this; I hit it as well: a sidebar "sleeping agent" row with the same providerSession.id as a live tab. If the sweep found retained-live-work, cold-restore resume could say so ("the previous process may still be running on the host") instead of silently starting a duplicate.

Also a note for anyone searching: the post-update "Orca couldn't reattach to this pane's terminal session on the host" banner (terminal_pane_recovery_remount, reason: "reattach-unverifiable") is this same mechanism, not a separate bug. In orca terminal list --json, the affected tabs have a ptyId of the form ssh:…@@pty2:<relayInstanceId>:<n> with a relay instance id different from the current one and incarnationId: null.

We expect to fully fix this by Monday next week.

More PRs coming

Data point from a Windows 11 client with a WSL2 Ubuntu SSH target, on 1.4.220 → 1.4.221 (relay 0.1.0+885531a14396 → 0.1.0+dcbb9f311363).

The old relay logged Grace started timeoutMs=0, branch=configured, ptys=7 when the updater quit the app. About 70 seconds later the new build connected to a fresh relay, and all 7 SSH PTY leases were marked expired. Each tab showed [Orca skipped hidden terminal output because main recovery was unavailable.] The old relay kept all 7 shells and agent sessions running, unreachable from the UI, until the machine was rebooted 16 minutes later. On a second (macOS) SSH target, the old relay held 0 PTYs and exited cleanly.

So it still reproduces on 1.4.221 with a Windows client.

Status update: 1.4.221 predates the fix. On main (#24863, mainly #25124 and #25331), a reconnect after an update finds the previous build's relay on the host and reattaches its terminals through it instead of opening fresh shells, and the old relay exits once its last shell does. Managed-server hosts only update when no terminals are running (#25122). None of this is released yet.

Keeping this open for Windows SSH hosts: the old relay is detected and the tab is held instead of replaced, but it can't be bridged yet (src/main/ssh/ssh-previous-relay-terminals.ts:109).

Closing as obsolete after the merged SSH relay retirement in #27214 and #27215. This issue concerns the SSH relay PTY lifecycle, recovery, backpressure, or orphan cleanup. #27214 removed the desktop relay connection/fallback path; #27215 deleted the SSH relay server, deployment assets, and relay-only tests. SSH hosts now use the managed Orca server (orcad), and unsupported hosts fail explicitly, so this relay-specific work is no longer planned. Existing relays started by older Orca versions are not automatically stopped by these merges; their terminals must be closed in that older version or ended on the host before conversion.

Thanks for reporting this, and for the data points. On SSH hosts, terminals now live in Orca's managed server (orcad), which only updates itself when no terminals are running (#25122), so an app update no longer strands remote terminals or agent sessions. Please reopen if you see otherwise on 1.4.224 or later.

fix(ssh): stop orphaning live relays when an endpoint is taken over (s…

docs(ssh): record that an app update strands relay-backed terminals a…

Navigation Menu

Repository navigation

[Bug]: Direct SSH: updating Orca strands all remote terminals and running agent sessions (relay version swap) #13852

Operating system

Orca version

Details

Why reattach across an update is impossible by construction

What #17821 does and does not do

The actual fix

Meanwhile

Field measurement, and a consequence for how this gets fixed

The part worth noting for the fix

Assessment: real, not fixable now, and the ordering is stricter than this issue states

Two corrections to the issue text

Recommendation

Correction to my assessment above: the unlink hazard is broader than I said

A second detail that constrains the proposed fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Direct SSH: updating Orca strands all remote terminals and running agent sessions (relay version swap) #13852

Description

Operating system

Orca version

Details

Activity

domuk-k commented on Aug 16, 2026

nwparker commented on Sep 1, 2026

Why reattach across an update is impossible by construction

What #17821 does and does not do

The actual fix

Meanwhile

nwparker commented on Sep 1, 2026

Field measurement, and a consequence for how this gets fixed

The part worth noting for the fix

nwparker commented on Sep 1, 2026

Assessment: real, not fixable now, and the ordering is stricter than this issue states

Two corrections to the issue text

Recommendation

nwparker commented on Sep 1, 2026

Correction to my assessment above: the unlink hazard is broader than I said

A second detail that constrains the proposed fix

nwparker commented on Sep 3, 2026

tlgnkl commented on Sep 27, 2026

AmethystLiang commented on Oct 2, 2026

DaveDev42 commented on Oct 9, 2026

OrcaWin commented on Oct 10, 2026

OrcaWin commented on Oct 11, 2026

OrcaWin commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions