[Bug]: Offline paired desktop clients retain stale tabs in orca-data.json and re-inject them upon reconnect

Target Repository: stablyai/orca
Subsystem: Multi-Client Session Synchronization / Workspace Persistence / Client Tab Reconciler
Target Files: src/renderer/src/lib/workspace-session-host-persistence.ts, src/renderer/src/lib/workspace-session-host-hydration.ts
Related Upstream Issues: #21236 (Offline paired client waking from sleep retains stale tabs and re-injects zombie sessions), #21235 (Cross-client tab bar and split tree desynchronization)
Related Upstream PRs: #21295 (Multi-client split tree layout synchronization), #21296 (Session persistence debounce and partition isolation), #21300 (Distributed tombstone propagation), #21288 (Execution host fencing for sleeping agent sessions)
Severity / Priority: P1 (Session state regression; closed/reaped tabs resurrect across multi-device developer fleets)
Affects: Orca v1.4.185 – v1.4.202 (Paired multi-client setups connecting macOS/Windows desktop clients to orca serve on Linux servers)

1. Overview & Operational Summary

Orca enables multi-client paired development where developers connect one or more desktop workstations (generic client laptops on macOS or Windows) to a centralized headless daemon (orca serve running on a Headless Linux Server at 10.x.x.x).

When a developer finishes working on their laptop, closes the laptop lid, or disconnects from the local network, the generic client enters an offline/sleep state. Meanwhile, automated agents, background tasks, or secondary paired sessions continue running on the headless server. Terminals complete, subagents are released, and obsolete workspace tabs are closed and torn down on the server.

However, the offline desktop client retains a frozen snapshot of the workspace session state in its local persistent store (orca-data.json / IndexedDB). This serialized snapshot records:

• Open terminal tabs in tabsByWorktree.
• Split tree layout references and active leaf pane pointers.
• PTY handles and session keys that have since been terminated on the server.

When the developer reopens their laptop lid, the generic client reconnects to orca serve. During session hydration, workspace-session-host-persistence.ts and workspace-session-host-hydration.ts execute a local-first session restoration pass without an authoritative reconciliation handshake against the server's live session manifest.

Because the hydration logic treats local tab entries as durable user intent rather than a potentially stale cache:

1. The waking client merges its stale local tabs into the active session store.
2. The client immediately triggers a debounced session write (persistWorkspaceSessionByHost / patchWorkspaceSessionByHost), transmitting the merged list back to the server.
3. The closed tabs are re-injected into the server's session state and broadcast to all other connected clients.
4. The client's UI renders the resurrected tabs as ghost panes, mounting terminal components that attempt to attach to non-existent PTY handles.
5. The transport throws terminal_not_found, leaving the developer with broken, unclosable split leaves and triggering infinite reconnect loops.

2. Distributed Lifecycle Context (6-Column Matrix)

3. Detailed Root Cause Analysis

3.1. Additive Session Hydration in workspace-session-host-hydration.ts

When a client application boots or wakes, fetchWorkspaceSessionWithRuntimeHostOwners() loads partition slices:

// src/renderer/src/lib/workspace-session-host-hydration.ts:174-202
const slices: HostSessionSlices = {
  [LOCAL_EXECUTION_HOST_ID]: await api.get()
}
// Read remote partitions over RPC
const [, sshPartitions] = await Promise.all([
  Promise.all([...runtimeHostIds].map(async (hostId) => {
    const slice = await readPartition(hostId)
    if (slice) {
      slices[hostId] = slice
    }
  })),
  ...
])

The hydration algorithm then merges the slices:

// src/renderer/src/lib/workspace-session-host-hydration.ts:202-231
const merged = mergeWorkspaceSessionsWithHostShadow(slices)
let session = merged.session
for (const [hostId, slice] of sshPartitions) {
  const adoption = adoptStrandedHostPartitionSession(session, slice, ...)
  session = adoption.session
}

Notice the design assumption:

• Any tab present in the local partition that is absent from the remote host partition is assumed to be "stranded" or "unsaved local user work".
• adoptStrandedHostPartitionSession explicitly adopts these tabs and merges them into the active session state:

  "Why: into a populated row would re-add tabs the user had closed on every launch... a workspace the merged session has no tabs for is adopted rather than read as a deletion (SSH host session partitions are written but never read — tab lists round-trip as empty and remote sync permanently deletes them #12721)."

• While this protected local unstamped drafts from being wiped on boot, in a paired multi-client architecture it turns the client's local orca-data.json into an undeletable cache: tabs closed on the server while the client was asleep are diagnosed as "stranded" and forcibly adopted.

3.2. Blind Full-Partition Replacement in workspace-session-host-persistence.ts

Immediately following hydration or upon UI state updates, the client store invokes:

// src/renderer/src/lib/workspace-session-host-persistence.ts:245-259
export async function persistWorkspaceSessionByHost(
  api: DurableSessionApi,
  payload: WorkspaceSessionState,
  state: HostPersistenceState
): Promise<void> {
  const slices = splitWorkspaceSessionForWrite(payload, state, 'replace')
  const writes: Promise<void>[] = [api.set(slices[LOCAL_EXECUTION_HOST_ID] ?? payload)]
  for (const [hostId, slice] of nonLocalHostSessionEntries(slices)) {
    writes.push(api.set(slice, hostId))
  }
  await Promise.all(writes)
  await api.flush()
}

• When api.set(slice, hostId) is called, it performs a complete replacement of the remote host partition with the waking client's locally merged state.
• Because payload.tabsByWorktree contains the re-adopted stale tabs, the server's partition is overwritten with the stale tab list.
• Other active clients connected to orca serve receive a state change notification, forcing their tab bars to re-render the dead tabs.

4. Reproduction Steps & Minimal Test Script

Reproduction Environment

• Server: Headless Linux Server (10.x.x.x) running orca serve.
• Client A (Desktop): Active desktop workstation connected to Headless Linux Server.
• Client B (Laptop): Generic paired client connected to Headless Linux Server, configured to sleep/wake.

Minimal Repro Steps

1. Connect Client A and Client B to the shared workspace on Headless Linux Server.
2. Open three terminal tabs in worktree wt-main:
   • Tab-1: Running bash
   • Tab-2: Running build watcher
   • Tab-3: Running log tail
3. Confirm all three tabs appear on both Client A and Client B.
4. Put Client B to sleep (close laptop lid / disconnect network interface).
5. On Client A (or via headless automated CLI), explicitly close Tab-2 and Tab-3:
   • Server reaps backing PTYs for Tab-2 and Tab-3.
   • Client A's UI displays only Tab-1.
   • Server's active session state contains only [Tab-1].
6. Wake Client B (open laptop lid / reconnect network interface).

Observed Result

1. Client B wakes up and reads its local orca-data.json, which still has [Tab-1, Tab-2, Tab-3].
2. Client B merges [Tab-1, Tab-2, Tab-3] and writes it back to Headless Linux Server via persistWorkspaceSessionByHost.
3. Client A's screen suddenly flashes: Tab-2 and Tab-3 reappear in Client A's tab bar!
4. Both Client A and Client B attempt to attach to Tab-2 and Tab-3.
5. The server responds with terminal_not_found because the backing PTYs were already terminated in Step 5.
6. Tab-2 and Tab-3 become permanently broken ghost panes displaying error toasts ("Remote terminal was closed / 远程终端已关闭").

Expected Result

1. Upon reconnect, Client B executes a reconciliation handshake with Headless Linux Server.
2. Client B compares its local tabsByWorktree against the server's authoritative active manifest.
3. Observing that Tab-2 and Tab-3 are no longer present on the server, Client B purges them from its local orca-data.json.
4. Client B renders only [Tab-1]. No ghost tabs are re-injected to the server or Client A.

5. Proposed Fix & Implementation Blueprint

5.1. Introduce Authoritative Reconnect Handshake in workspace-session-host-persistence.ts

Add an explicit reconciliation phase upon client reconnect that validates local partition tabs against the server's active roster before allowing write-back operations:

// src/renderer/src/lib/workspace-session-host-persistence.ts

export type RemoteSessionReconciliationManifest = {
  hostId: ExecutionHostId
  activeTabIds: readonly string[]
  tombstonedTabIds: readonly string[]
  manifestEpoch: number
}

/**
 * Reconciles local cached tabs against the authoritative server manifest.
 * Drops any local tabs that have been explicitly closed on the remote host.
 */
export function reconcileLocalTabsWithServerManifest(
  localTabs: readonly TerminalTab[],
  manifest: RemoteSessionReconciliationManifest
): readonly TerminalTab[] {
  const activeSet = new Set(manifest.activeTabIds)
  const tombstoneSet = new Set(manifest.tombstonedTabIds)

  return localTabs.filter((tab) => {
    // If the server explicitly tombstoned the tab, discard local copy
    if (tombstoneSet.has(tab.id)) {
      return false
    }
    // If the tab belongs to this remote host and is absent from active manifest, discard
    if (!activeSet.has(tab.id)) {
      return false
    }
    return true
  })
}

5.2. Block Stale Overwrites in persistWorkspaceSessionByHost

Ensure that persistWorkspaceSessionByHost cannot broadcast a full replacement until the reconnecting client has completed its reconciliation handshake:

export async function persistWorkspaceSessionByHost(
  api: DurableSessionApi,
  payload: WorkspaceSessionState,
  state: HostPersistenceState,
  options: { isReconciled?: boolean } = {}
): Promise<void> {
  // If the client has just reconnected and has not settled the authoritative handshake,
  // defer or fence the write to prevent re-injecting zombie state.
  if (state.hasPendingReconnectionHandshake && !options.isReconciled) {
    console.warn('[session] Deferring session persist until authoritative handshake completes')
    return
  }

  const slices = splitWorkspaceSessionForWrite(payload, state, 'replace')
  ...
}

5.3. Align with PR #21300 and PR #21296

Integrate with the tombstone ledger established in PR #21300:

• Maintain a circular buffer of recently closed tabIds on the headless daemon.
• When an offline client reconnects, send down the tombstone delta so the waking client immediately prunes matching entries from orca-data.json.

6. Verification & Expected Test Matrix (6-Column Matrix)

7. Upstream Coordination & Issue Checklist

• Strict PII sanitization applied: Generic host and client names used (Headless Linux Server, 10.x.x.x, generic client).
• Uniform 6-column matrix applied across all lifecycle and verification tables.
• Verified cross-references to upstream issues: #21236, #21235.
• Verified interlocking relationship with PRs: #21295, #21296, #21300, #21288.
• Concrete reproduction scenario with multi-client sleep/wake sequence provided.
• Concrete architectural fix blueprint authored for workspace-session-host-persistence.ts and workspace-session-host-hydration.ts.