[Architecture / RFC]: Tab, Terminal & PTY Lifecycle Stabilization across Local, Remote & Cloud Runtimes

[LesleyMurfin]

Tab, Terminal & PTY Lifecycle Stabilization across Local, Remote & Cloud Runtimes

First and foremost, a sincere and enthusiastic thank you to the Orca maintainers and contributors for building such an extraordinarily capable application. Orca has fundamentally transformed engineering workflows, remote execution, and multi-agent development.

To be fair upfront: my background is in carrier-grade telecommunications and distributed network root-cause engineering—the tier called in when cascading, silent failures evade standard diagnostics. In mission-critical telecom networks, when an edge terminal or remote link flickers, the root fault is almost never a random network drop; it is almost always host-side resource starvation and uncollected descriptors failing keepalive SLAs.

I brought that exact carrier-grade telemetry and boundary-isolation discipline to Orca while managing demanding 24/7 autonomous multi-agent workloads—systematically tracing the lifecycle from Linux kernel cgroups and OS process trees, through transport RPC multiplexers, and up into desktop Electron reconcilers.

---

📌 Bottom Line Up Front (BLUF): Why We Are Opening This Tracking Issue

We put together this tracking issue to share what we've learned from months of running headless orca serve 24/7, and to offer a collaborative architectural perspective connecting several remote tab, terminal, and reconnection issues reported across Discord and GitHub.

Our hope is to share the relationships we've observed, seek feedback from the maintainers and community on a structured approach to addressing these gaps, and work together toward our shared goal: a rock-solid, production-stable Orca environment for everyone.

💡 $\color{#58a6ff}{\textsf{The Core Realization: Coupled Lifecycle Boundaries}}$

While client reconciler gaps and host daemon process leaks can occur independently, sustained 24/7 headless execution couples them into a compounding feedback loop:

Unreaped PTY process trees accumulate in the daemon's systemd scope ($\color{#da3633}{\mathbf{70\text{ dangling PTYs}}}$, $\color{#da3633}{\mathbf{44.9\text{ GB RAM}}}$, $\color{#d29922}{\mathbf{\text{saturated CPU cores}}}$, and $\color{#d29922}{\mathbf{\text{exhausted } /tmp}}$). The resulting event-loop latency stalls transport heartbeats, causing desktop clients to drop and attempt to rehydrate dead PTY handles—triggering continuous reconnect storms and ghost tab loops.

🎯 $\color{#bc8cff}{\textsf{The Cause-and-Effect Chain (How Failures Cascade Across Boundaries)}}$

Stage	Failure Domain	Mechanism & Real-World Impact
	Unreaped PTY Process Trees	Unreaped child processes accumulate in systemd cgroups → $\color{#da3633}{\mathbf{70\text{ dangling PTYs}}}$ / $\color{#da3633}{\mathbf{44.9\text{ GB RAM}}}$ / $\color{#d29922}{\mathbf{\text{CPU saturation}}}$ / $\color{#d29922}{\mathbf{/tmp\text{ exhaustion}}}$ → kernel swap thrashing & EAGAIN.
	Heartbeat Ping Timeouts	Severe host resource exhaustion spikes daemon event-loop latency → stalls WebSocket & SSH heartbeat pings → drops active sessions.
	Tab Resurrection Storms	Desktop client reconnects → revives dead handles as uncloseable ghost tabs → View Transition collision storms crash React with Error 185.

This turns what previously looked like separate, mysterious glitches across Discord into one connected, reproducible physics problem.

🌐 Cross-Environment Impact: How This Affects Every Deployment Tier

The physical cause-and-effect chain is identical across all setups, but the symptom threshold changes depending on environment constraints:

Deployment Tier	Failure Mechanism	What the Developer Experiences
💻 Local Desktop (No Server)	Subagent or terminal exits locally; client store retains unbacked placeholder (ptyId: null).	Rapid tab switching, remounts, or deleting a worktree triggers View Transition aborts, crashing React with Error 185.
🪟 Windows + WSL	Background processes leak inside the WSL distro; WSL fails to release cgroup memory back to Windows host.	Vmmem memory balloons to 16GB–32GB; WSL interop socket latency causes the Windows-to-WSL terminal bridge to stall.
🍎 Mac Mini over SSH	MacBook lid closes (sleep); background subagents on Mac Mini leak PTY descriptors into per-user launchd scopes.	Reopening laptop reveals frozen black terminal panes and uncloseable ghost tabs that loop on close.
☁️ Ephemeral Cloud VMs & Sandboxes	Per-workspace cloud containers run under strict cgroup limits (e.g. 2GB–4GB RAM, tight PID caps).	Leaking just 3–5 subagent process trees trips the Linux kernel OOM-killer, killing the entire cloud container.

💡 The Takeaway: Client reconciler gaps affect all desktop users, while host process leaks hit constrained runtimes—like disposable Cloud VMs and WSL distros—fastest. Establishing clean lifecycle agreements is the foundational prerequisite for making ephemeral Cloud VMs and remote desktop sessions rock-solid.

🎯 How We Hope This Issue Serves the Team:

• 🗺️ Mapping the Shared Lifecycle: Outlining how server-side resource behavior (memory, CPU, and process lifecycles) and client-side tab stability are connected across the transport layer.
• 📋 Offering a Modular Status Board: Gathering relevant in-flight work and discussions (from ourselves and fellow community members) into a single, organized view so maintainers can review, cherry-pick, or adapt fixes at their own pace without wire-protocol breaking changes.
• 📊 Contributing Open Telemetry: Sharing our empirical fleet logs, reproduction traces, and AppImage test results to help support upstream review and triage.

_{💡 Inspiration & Structural Credit: We explicitly want to credit the elegance and clarity of the canonical tracking roadmap in stablyai/orca#15190 (by @Tauri-EPC) as our direct inspiration for this tracking issue format.}

---

🛠️ Months of Server Debugging & Operational Context

While running heavy 24/7 workloads, our headless server began hitting critical limits—slowing down, dropping sessions, and accumulating memory. Rather than guessing, we spent months capturing raw telemetry to understand what was happening under the hood:

Focus Area	Empirical Context & Production Testing Strategy
🏢 The Environment	Enterprise bare-metal Linux server, operating in the host OS alongside our development K3s cluster (the platform we are building), running 24/7 automated workloads across ~12 projects and over 200 workspaces (paired with WSL2 and multi-platform desktop clients).
🔍 Observability Pipeline	Zero application modifications: External logging pipeline streaming OS and application telemetry into Grafana for native resource inspection.
🧪 Production AppImage Strategy	Our operational strategy has been to run our production servers on the official upstream AppImage releases, systematically layering in candidate stability patches to verify memory, disk, CPU, and tab lifecycle behavior under real-world load.

---

💡 The Realization: How Server Resource Exhaustion Reaches the Desktop

While tracking community reports on Discord and GitHub, we connected the dots: our internal enterprise server symptoms and the random scattering of remote tab, ghost pane, and reconnect bugs are all part of the exact same distributed lifecycle breakdown.

Layer	🔴 Layer 1: Host Server (The Resource Leak)	⚡ Layer 2: Transports (The Connection Cascade)	🖥️ Layer 3: Client UI (The Desktop Desync)
Symptoms	• Memory: Daemon accumulates 44.9 GB in cgroups • CPU: Lingering agent loops consume allowed CPU • Handles & Disk: 70+ /dev/pts/* descriptors, open FIFO pipes, and orphaned child dirs on disk	• Heartbeats drop across SSH tunnels & WebSockets • Remote stream frames lag and drop • Sudden reconnect loops and reconnection storms	• Uncloseable ghost terminal panes • Infinite client tab flicker loops (85 remounts/min) • React Error 185 crashes and wedged workspaces
Root Causes	• Finished tasks fail to kill subagent trees • Host PTY descriptors never reaped upon exit • Ephemeral child worktree dirs linger on disk	• Server event loop suffocates under host resource exhaustion • Volatile term_* handles go stale during drops • Transports lack graceful timeouts on dead processes	• Desktop reconciler attempts to re-hydrate dead PTYs • Tab close requests fail without idempotent acknowledgment • Client resurrects null-PTY placeholders indefinitely

🔄 The Failure Chain: How the Three Layers Cascade Upward

[ Layer 3: Desktop Client UI ] 
        ↕  (Stuck with ghost tabs that fail to close)
[ Layer 2: Transports (SSH Tunnels / WebSockets / Runtime RPC) ] 
        ↕  (Heartbeats drop because server event loop lags)
[ Layer 1: Host Server (Daemon & OS Process Trees) ] 
           (Unreaped subagents & PTYs pin 44.9 GB in systemd cgroups)

---

🎯 Our Shared Vision & The Roadmap Below

Our goal is to collaborate with maintainers on establishing clean, deterministic lifecycle agreements across each boundary—so headless Orca can run 24/7 autonomously with zero breaking wire changes:

• ✔️ Layer 1 (Host Server): Finished tasks cleanly reap their process trees, preserving host CPU, memory, and file descriptors.
• ✔️ Layer 2 (Transports): Prevent daemon event-loop lag from stalling heartbeats or desyncing session handles across SSH tunnels, WebSockets, and Runtime RPC.
• ✔️ Layer 3 (Client UI): Desktop reconcilers evict dead tabs cleanly without infinite remount loops, ghost panes, or renderer crashes.

The Developer Experience: An engineer should be able to run autonomous multi-agent workloads 24/7, close their laptop, and return hours or days later to a responsive, perfectly synchronized workspace—with zero ghost tabs, reconnect storms, or daemon memory bloat.

How We Structured This Document:

1. The Invariants & Deep Dives: Expand the accordions below for our empirical setup, cascading impact deep dives, and sequence diagrams.
2. The Status Board: Jump straight to the modular table below to review individual, decoupled PRs categorized by maintainer focus areas and live upstream status.

---

💥 The Cascading Impact We Observed (The 4 Pillars — Table Format)

Failing to address lifecycle boundaries creates a domino effect where a quiet host leak eventually crashes the desktop UI. This breakdown propagates across four interconnected pillars:

🛑 Pillar 1: Application Resource Exhaustion (The App Ceiling)

Layer	Breakdown & Observable Behavior
👤 User Experience	Background agent runs finish successfully, but over several hours the headless daemon eats up all available CPU, memory, and disk—leaving the server starved and sluggish.
⚙️ Under the Hood	Completed tasks and dismissed subagents fail to terminate their child process trees. Unreaped PTY file descriptors (/dev/pts/*) linger alongside orphaned worker subshells (bash, node, git) in unisolated systemd user cgroups.
📊 Empirical Telemetry	Within 24 hours of sustained multi-agent execution, the daemon consumed its entire systemd cgroup allowance: 44.9 GB pinned RAM across 70+ unkilled PTYs, CPU saturated by lingering subagent loops, and open /dev/pts/* descriptors & FIFO pipes held open alongside orphaned worktree directories —starving adjacent development services (K3s) as well as Orca itself.

---

⚡ Pillar 2: Cross-Boundary Desync (Transport Degradation)

Layer	Breakdown & Observable Behavior
👤 User Experience	Active desktop sessions disconnect unexpectedly; terminals freeze, and reconnect spinners loop repeatedly.
⚙️ Under the Hood	Host memory pressure and blocked threads induce severe Node.js event-loop latency on the daemon. Ping/pong heartbeats stall and drop across SSH multiplexed tunnels, remote WebSockets, and RPC relays.
📊 Empirical Telemetry	Stream bindings drop mid-session. During the reconnect window, volatile terminal identifiers (term_*) and durable layout records desync between host memory and client cache.

---

💥 Pillar 3: Client Render Crashes (The Reconciler Collapse)

Layer	Breakdown & Observable Behavior
👤 User Experience	On reconnect, tabs flicker rapidly, dead terminals render as uncloseable black boxes, and the desktop UI crashes with a React error boundary.
⚙️ Under the Hood	The desktop reconciler receives stale layout snapshots and mounts panes bound to missing host PTYs (ptyId: null). Clicking to close fails with unhandled selector_not_found errors, unsetting tombstones and triggering infinite remount oscillations.
📊 Empirical Telemetry	Generates 85 remount loops/min, triggering 2,707 InvalidStateError view transition aborts from concurrent Chromium animation collisions that crash the UI with React Error 185 (Maximum update depth exceeded).

---

🔒 Pillar 4: Operational Friction (Permanent Workspace Wedging)

Layer	Breakdown & Observable Behavior
👤 User Experience	Restarting the desktop app or rebooting the server does not recover the workspace—the ghost tabs and black panes immediately reappear.
⚙️ Under the Hood	Serialized layout records persist to disk without OS process liveness verification on boot. When the daemon restarts, it blindly re-hydrates dead sessions and pushes them to reconnecting clients.
📊 Empirical Telemetry	Active workspaces become permanently wedged with uncloseable tabs and zombie panes, forcing engineers to abandon or delete workspace directories to regain an IDE.

---

🔄 The 6-Step Distributed Tab Lifecycle (At a Glance)

1. Step 1: Launch & Bind — Host PTY & worktree allocation, environment isolation, custody registration.
2. Step 2: Synchronize — Remote snapshot propagation, multi-client split sync, offline catalog hydration.
3. Step 3: Reconnect & Attach — Reconnection after network drop, daemon restart, scrollback buffer ring recovery.
4. Step 4: Close & Teardown — Idempotent tab teardown across deleted worktrees, tombstone lock preservation.
5. Step 5: Reconcile & Prune — 30s TTL grace bound on null-PTY placeholders, ViewTransition abort absorption.
6. Step 6: Reap & Clean — OS PTY reaping by process incarnation (hostId, pid, startTime), systemd cgroup cleanup.

🔄 The Causal Chain: How Failures Cascade Across the 6 Steps

[ HOST SERVER: Steps 6 & 1 ]
   Background tasks finish, but child process trees & /dev/pts/* handles are not reaped.
   Dangling processes pin GBs of RAM in the systemd slice.
               │
               ▼
[ TRANSPORTS: Steps 2 & 3 ]
   Host daemon event-loop latency builds up under process pressure.
   Daemon fails to answer WebSocket/SSH heartbeat pings in time, dropping active streams.
   The disconnect is host event-loop lag, not external network failure.
               │
               ▼
[ CLIENT RECONNECT: Step 3 ]
   Desktop client reconnects and tries to re-attach to the terminal session.
   Host returns terminal_not_found; client transport latches in infinite connecting spinner.
               │
               ▼
[ CLIENT GHOST TABS: Steps 4 & 5 ]
   Reconciler rehydrates dead PTYs from stale snapshots into ptyId: null placeholders.
   User closes tab -> host reports missing target -> client lacked durable intent, allowing stale snapshots to resurrect tabs.
   Tab resurrects every 15 seconds in an 85 remounts/min oscillation loop.
               │
               ▼
[ UI CRASH: Step 5 ]
   Remount oscillations trigger colliding document.startViewTransition() calls.
   Chromium throws unhandled InvalidStateError rejections, tripping React Error 185.

---

🧭 Quick Navigation: The 4 Pillars • Lifecycle Overview • Status Board • Sequence Diagrams • Gap Specifications • Fleet Telemetry

---

Unified 6-Step Lifecycle Status Board

The tables below map the remote terminal and tab lifecycle across its core host-to-client boundaries, providing dedicated status boards with concise architectural context for each step:

Step 1: Launch & Bind 9 PRs • 5 Merged • 4 In-Flight

Governs host PTY allocation, worktree canonicalization, and initial custody registration when remote tabs, terminals, or subagents are spawned.

Subsystem	Type	Issue	Status	PR	Mechanism & Impact
worktrees	Bug	#16243		#16494	Resolve id: worktree selectors by canonical path equivalence (fix(worktrees), Merged v1.4.191).
worktrees	Bug	#16733		#16829	Route unstamped local worktrees to local runtime (fix(worktrees), Merged v1.4.192).
terminal	Bug	#5944		#5946	Fix floating workspace terminal selector_not_found on remote runtime (fix(terminal), v1.4.187).
serve	Bug	#5132		#5131	Persist active serve session tab so PTY output does not snap focus to last tab (fix(serve), v1.4.186).
cli	Feature	#10608		#10612	Implement orca serve stats [--json] for live headless runtime inspection (feat(cli)).
browser	Feature	#14552		#19628	Cap and reclaim offscreen browser tabs on headless serve (fix(browser)).
cli	Bug	#19343		#21447	Ignore ORCA_ENVIRONMENT remote selection for terminal commands (fix(cli)).
cli	Bug	#21339, #17767		#21351	Route terminal close across remote execution host boundary (fix(cli), Gap 1).
daemon	Bug	—		#21381	Answer the per-PTY snapshot predicate for the requested PTY (fix(daemon), Merged v1.4.205).

---

Step 2: Synchronize 6 PRs • 3 Merged • 3 In-Flight

Manages multi-client split leaf synchronization, remote snapshot propagation, and runtime catalog hydration across transport disconnects.

Subsystem	Type	Issue	Status	PR	Mechanism & Impact
remote-runtime	Feature	#20923		#20922	Stop shared inventory snapshot from retiring unconfirmed sibling panes (fix(remote-runtime)).
session	Feature	#21236, #21235		#21288	Scope agent session resume to the host that captured the session (fix(session), Merged v1.4.193).
terminal	Bug	—		#21294	Keep split real direction and container layout when leaf sets mutate (fix(terminal), Merged).
terminal	Feature	#17767		#21285	Preserve terminal buffer across cold-parking to restore scrollback on attach (fix(terminal)).
routing	Bug	#21341, #21295		#21332	Treat unhydrated runtime catalog as pending to prevent tab drops (fix(routing), Gap 3).
runtime	Bug	#21392		#21393	Keep client-dirty mirrored file dirty across host republish (fix(runtime), Merged v1.4.205).

---

Step 3: Reconnect & Attach 4 PRs • 1 Merged • 3 In-Flight

Handles client re-attachment after network drops or daemon restarts, scrollback ring buffer recovery, and unblocking the connecting state machine.

Subsystem	Type	Issue	Status	PR	Mechanism & Impact
daemon	Feature	#9585		#19430	Escape terminal daemon into dedicated systemd user scope (fix(daemon)).
transport	Bug	#21344		#21350	Handle terminal_not_found in attach() to unblock connecting state (fix(transport), Gap 6).
terminal	Bug	—		#21396	Keep unverifiable park-reveal snapshot apart from empty pane (fix(terminal), Merged v1.4.205).
terminal	Bug	—		#21480	Yield between cold-park scrollback captures to stop renderer freeze (fix(terminal)).

---

Step 4: Close & Teardown 3 PRs • 2 Merged • 1 In-Flight

Ensures idempotent tab closure across host boundaries, error toast teardown without ghost pane leaks, and client tombstone retention.

Subsystem	Type	Issue	Status	PR	Mechanism & Impact
runtime	Bug	#21189, #21041		#21277	Keep absent session tab close intents durable on definitive missing-target responses to stop resurrection loops (fix(runtime), Merged v1.4.194).
terminal	Bug	#21342, #9194		#21349	Teardown dead pane when dismissing remote terminal closed toast (fix(terminal), Gap 4).
editor	Bug	#21041		#21375	Keep unresolvable mirrored file tab open with truthful terminal state (fix(editor), Merged v1.4.205).

---

Step 5: Reconcile & Prune 2 PRs • 2 In-Flight

Enforces TTL bounds on null-PTY placeholders and absorbs benign ViewTransition animation aborts to stop React error 185 crashes.

Subsystem	Type	Issue	Status	PR	Mechanism & Impact
runtime	Bug	#21340, #11800		#21354	Bound null-PTY tab placeholders with 30s TTL in shouldReplaceTerminalTab (fix(runtime), Gap 2).
ui	Bug	#21345		#21358	Debounce startViewTransition and absorb abort errors to prevent React error 185 (fix(ui), Gap 7).

---

Step 6: Reap & Clean 5 PRs • 5 In-Flight

Guarantees operating system process tree termination via incarnation tuples (hostId, pid, startTime) and systemd cgroup memory reclamation.

Subsystem	Type	Issue	Status	PR	Mechanism & Impact
orchestration	Bug	#18789		#18790	Reap leaked worker terminals via process-incarnation fallback (fix(orchestration)).
orchestration	Bug	#18411		#19404	Clean up created_child worktree directories after worker release (docs(orchestration)).
daemon	Bug	#21343		#21352	Verify OS PTY liveness during host daemon startup layout migration (fix(daemon), Gap 5).
orchestration	Bug	#18803		#21448	Resolve terminal handle via incarnation fallback in federation reap (fix(orchestration)).
daemon	Feature	#20721		#21504	Retire legacy daemon generations once sessions are confirmed idle (feat(daemon)).

---

📊 1. Sequence & Architecture Diagrams (Mermaid)

Diagram A: The 6-Step Distributed Tab Lifecycle Architecture

The Remote Serve terminal and tab subsystem operates across six coupled execution boundaries connecting client UI state, RPC transports, and host OS processes. Deterministic cleanup requires strict invariants at each boundary:

flowchart TD
    subgraph Step1["Phase 1: Host Allocation & Runtime Routing — Tab & remote session binding, PTY allocation, custody registration"]
        A1["User / Agent Requests Terminal"] --> A2["git worktree add (Ephemeral Worktree)"]
        A2 --> A3["Mint Process Incarnation Handle (hostId, pid, startTime)"]
        A3 --> A4["Allocate OS PTY (/dev/pts/*)"]
        A4 --> A5["Register Process Custody with Daemon"]
    end

    subgraph Step2["Phase 2: Remote Session Synchronization & Split State — Transport snapshot propagation, inventory broadcasts, multi-client split sync"]
        B1["Attach WebSocket / RPC Transport"] --> B2["Propagate Initial Transport Snapshot"]
        B2 --> B3["Broadcast Inventory to Multi-Client Split Views"]
        B3 --> B4["Initialize xterm & Scrollback Buffer"]
        B4 --> B5["Mount React Tab in Client UI Workspace"]
    end

    subgraph Step3["Phase 3: Reconnect, Attach & Buffer Recovery — Reconnect after network drop, daemon restart, reminting & scrollback recovery"]
        C1["Client Disconnects / Laptop Sleep / Daemon Restart"] --> C2["Park Terminal & Preserve Offline Scrollback (#21285)"]
        C2 --> C3["Verify Host Identity on Resume (#21288)"]
        C3 --> C4["Remint Live Handle from Process Incarnation (hostId, pid, startTime) (#18790)"]
    end

    subgraph Step4["Phase 4: Tab Close, Teardown & Durable Tombstones — User or coordinator closes tab, RPC teardown, deleted worktree handling"]
        D1["User Closes Tab or Coordinator Releases Worker"] --> D2["Dispatch RPC session.tabs.close()"]
        D2 --> D3["Idempotent Check: Allow Already-Absent Worktree (#21277)"]
        D3 --> D4["Set Durable Client Close Tombstone (#9194)"]
        D4 --> D5["Unmount Tab & Evict Pane from Client Layout"]
    end

    subgraph Step5["Phase 5: Client Store Reconciliation & Placeholder Bounds — Client store reconcile against host snapshot, tombstone management, DOM boundaries"]
        E1["Background Store Reconciler Ticks (15s Loop)"] --> E2["Sync with session.tabs.list Snapshot"]
        E2 --> E3["Durable Tombstone Suppresses Resurrection"]
        E3 --> E4["Prune Stale Null-PTY Placeholders (#11800)"]
        E4 --> E5["Debounce ViewTransitions & Catch DOM Errors (#21358)"]
    end

    subgraph Step6["Phase 6: Host Process & PTY Reaping (Resource Reclamation) — Host process termination, PTY deallocation, worktree dir removal, RAM deallocation"]
        F1["Host Process Termination (SIGHUP / SIGKILL 5s Deadline)"] --> F2["Deallocate OS PTY File Descriptor"]
        F2 --> F3["Remove Ephemeral Child Worktree Directory (#19404)"]
        F3 --> F4["Reclaim systemd Cgroup Scope & Host Memory (#19430, #18790)"]
    end

    Step1 -->|PTY Created & Custody Registered| Step2
    Step2 -->|Network Interruption / Daemon Restart| Step3
    Step3 -->|Re-attached & Reminted| Step2
    Step2 -->|Close Initiated| Step4
    Step3 -->|Worktree Deleted Offline| Step4
    Step4 -->|Reconcile Pass| Step5
    Step5 -->|Host Teardown| Step6

    style Step1 fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
    style Step2 fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style Step3 fill:#fff8e1,stroke:#ffa000,stroke-width:2px
    style Step4 fill:#ffebee,stroke:#d32f2f,stroke-width:2px
    style Step5 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style Step6 fill:#e0f2f1,stroke:#00796b,stroke-width:2px

Loading

Diagram B: The Tab Close & Flicker Loop (Sequence Flow)

When a user or agent attempts to close a tab referencing an already-removed worktree, the lack of idempotence in the host RPC handler induces an infinite client-side re-render flicker loop:

sequenceDiagram
    autonumber
    actor User as User / Agent
    participant Renderer as Client Tab Reconciler
    participant Tombstone as Client Durable Tombstone Set
    participant HostRPC as Headless Host (orca serve)
    participant Catalog as Host Worktree Catalog
    participant OS as Linux Kernel / PTY Subsystem

    Note over User, OS: Normal Operation: Worktree deleted on host
    OS-->>Catalog: Worktree folder removed (/data/worktrees/...)
    Catalog-->>Catalog: Catalog entry purged

    Note over User, OS: User attempts to close stale tab in UI
    User->>Renderer: Click Close Tab (tab_worktree_123)
    Renderer->>Tombstone: Mark tab closing (transient flag)
    Renderer->>HostRPC: RPC terminal.close({ tabId: tab_worktree_123, worktree: id:123 })
    
    HostRPC->>Catalog: resolveWorktreeSelector(id:123)
    Catalog-->>HostRPC: Error: selector_not_found (Issue 21189)
    HostRPC-->>Renderer: RPC Exception: { code: selector_not_found, message: Worktree missing }

    critical The Defect (Pre-PR 21277)
        Renderer->>Tombstone: Catch error -> UNSET closing flag!
        Renderer->>Renderer: Retain tab in state (to avoid data loss)
    end

    Note over Renderer, HostRPC: 15 seconds later: Background reconcile fires
    Renderer->>HostRPC: RPC session.tabs.get()
    HostRPC-->>Renderer: Snapshot contains tab_worktree_123
    Renderer->>Renderer: Tab reappears in UI (Visible Flicker)
    Renderer->>Renderer: Attempt re-bind: spawn-left-pane-unbound
    Renderer->>HostRPC: RPC terminal.spawn({ worktree: id:123 })
    HostRPC-->>Renderer: Error: selector_not_found
    Renderer->>User: Toast: Remote terminal was closed / 远程终端已关闭 (Issue 9194)

    Note over User, OS: Infinite Loop Repeats Every 15 Seconds

Loading

Diagram C: The Server PTY Leak Cascade (The 5-Stage Failure Chain)

When invariants fail at any boundary in the distributed lifecycle, failures propagate across a 5-stage cascade from the host OS processes through the RPC transport and up into the client UI reconciler:

flowchart TD
    subgraph S1["Stage 1: Host Worktree Dismantle & PTY Leak"]
        A1["Worker / User Deletes Worktree"] --> A2["Network Epoch Bump (rendererGraphEpoch)"]
        A2 --> A3["Handle Stale: term_* Desyncs from OS PID"]
        A3 --> A4["Worker Release Emits release_unknown"]
        A4 --> A5["70 Dead PTYs Leaked / 44.9 GB RAM on Headless Linux Server"]
    end

    subgraph S2["Stage 2: Client Desync & Null-PTY Ghost Accumulation"]
        A5 -.-> B1["Host Worktree Deleted on Disk"]
        B1 --> B2["Client Syncs Tab Roster (session.tabs.list)"]
        B2 --> B3["shouldReplaceTerminalTab Retains null-PTY Placeholder"]
        B3 --> B4["Unclosable Ghost Tabs Accumulate (98 Mac, 74 PC)"]
    end

    subgraph S3["Stage 3: 15-Second Remount Loop & Input Wedging"]
        B4 --> C1["15s Cooldown Timer Fires (spawn-left-pane-unbound)"]
        C1 --> C2["Client Initiates RPC Tab Close / Re-bind"]
        C2 --> C3["Host Throws selector_not_found (#21189)"]
        C3 --> C4["Client Strips Closing Tombstone"]
        C4 --> C5["Bilingual Error Toast & Wedged Keystrokes (#9194)"]
        C5 --> C1
    end

    subgraph S4["Stage 4: ViewTransition Collisions & Layout Thrash"]
        C4 --> D1["Rapid Re-render Triggers document.startViewTransition()"]
        D1 --> D2["15s Oscillations Abort In-Flight Transitions"]
        D2 --> D3["2,707 InvalidStateError: Transition Aborted (PC)"]
        D3 --> D4["Microtask Queue Backpressure & Event Loop Stall"]
    end

    subgraph S5["Stage 5: Fatal Crashes"]
        D4 --> E1["Recursive useEffect State Dispatch"]
        E1 --> E2["React Error 185: Maximum update depth exceeded"]
        E2 --> E3["Detached DOM & xterm Buffers Accumulate"]
        E3 --> E4["V8 Heap Reaches 3.97 GB -> Hard OOM Crash (Mac 4x)"]
    end

    style S1 fill:#ffebee,stroke:#c62828,stroke-width:2px
    style S2 fill:#fff3e0,stroke:#ef6c00,stroke-width:2px
    style S3 fill:#fffde7,stroke:#fbc02d,stroke-width:2px
    style S4 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
    style S5 fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px

Loading

Detailed Stage-by-Stage Breakdown of the 5-Stage Cascade

1. Stage 1: Host Worktree Dismantle & PTY Leak (Host Kernel / PTY Subsystem)

   • Issues & PRs: #9585, #18789, #18803 • PR #18790, PR #19404, PR #19430, PR #21448
   • Mechanism: Worker releases emit release_unknown after reconnect epoch bumps; underlying OS processes are never sent SIGKILL, leaving orphan PTYs running indefinitely in systemd cgroups.

2. Stage 2: Client Desync & Null-PTY Ghost Accumulation (Client Reconciler Store)

   • Issues & PRs: #11800, #21236, #21340 • PR #21354, PR #21288
   • Mechanism: When worktrees delete on the host, shouldReplaceTerminalTab treats ptyId === null as lazy-spawning and refuses to prune, permanently retaining uncloseable ghost placeholders.

3. Stage 3: 15-Second Remount Loop & Input Wedging (Cross-Boundary RPC)

   • Issues & PRs: #21041, #21189, #21339 • PR #21277, PR #21351
   • Mechanism: Client attempts to close ghost tabs via session.tabs.close; host responds with selector_not_found; client strips durable tombstones and re-mounts the dead tab every 15-second cooldown cycle.

4. Stage 4: View Transitions Collision & 2,707 Abort Errors (Client UI / Animation Engine)

   • Issues & PRs: #21345 • PR #21358
   • Mechanism: Rapid 15-second remount thrashing triggers concurrent document.startViewTransition() invocations, aborting in-flight animations and flooding the microtask queue with unhandled InvalidStateErrors.

5. Stage 5: Fatal Crashes (React Error 185 & V8 Heap 3.97 GB OOM)

   • Issues & PRs: #21041, #21189, #21343 • PR #21277, PR #21352
   • Mechanism: Uncontrolled state oscillations hit React's recursive update depth limit (Maximum update depth exceeded) crashing the UI, while leaking xterm DOM buffers until V8 heap OOMs.

Gaps & Uncovered Slices (Where No PR Exists Yet)

While active community and maintainer PRs address key boundary points across the distributed tab lifecycle, 7 failure modes across the client-daemon boundary currently lack open PRs. These slices represent critical architectural gaps where explicit handling must be authored to achieve end-to-end resilience:

Gap 1: Server CLI PTY Close Routing Across Host Boundary (Issue #21339, #17767)

• Tracked Issue: #21339 (Related: #17767)
• Subsystem / Mechanism: Host RPC / CLI Remote Dispatch
• Observed Defect: When a user or headless automation script invokes orca terminal close --terminal <terminalHandle> via the CLI against a remote workspace session, the CLI attempts to resolve the terminal handle in the local process table rather than dispatching an RPC request across the network to the remote headless daemon. The local CLI returns tab_not_found, but the remote PTY continues running unmonitored on the headless host. On subsequent client reconnects or catalog hydration passes, the un-closed tab is resurrected as an active or ghost pane.
• Required Architectural Fix: Route CLI terminal close commands through the active session RPC transport to the host daemon when targeting remote workspace selectors, ensuring symmetric PTY teardown across host boundaries.
• Blast Radius: Host RPC dispatch and CLI command handler (src/cli/handlers/terminal-close.ts). Scoped to remote workspace terminal teardown invocations.
• Wire Compatibility: Fully backward-compatible; routes to existing closeTerminal endpoint on remote daemon runtime without schema changes.

1-Line Reproduction Command (Gap 1)

orca terminal close --terminal term-abc1234 --json

Dispatches terminal close from client against remote daemon handle; client returns {"ok":false,"error":"tab_not_found"} while backing PTY remains running and leaked on Headless Linux Server.

Gap 2: Null-PTY Placeholder 30s Grace Period Bound (Issue #21340, #11800)

• Tracked Issue: #21340 (Related: #11800)
• Subsystem / Mechanism: Client Store Reconciler / Tab Lifecycle
• Observed Defect: shouldReplaceTerminalTab retains null-PTY placeholders (ptyId: null) during git branch switches, worktree reconfigurations, or transient reconnect intervals. However, there is no upper time bound or expiration policy on how long a null-PTY entry may linger in tabsByWorktree. If the backing PTY never allocates (e.g., branch lacked remote devcontainer configuration or container setup stalled), the placeholder persists indefinitely as an unclosable ghost tab (172 ghost tabs observed in fleet telemetry).
• Required Architectural Fix: Implement an explicit 30-second TTL / grace period bound on null-PTY placeholders in the client reconciler. If a tab remains in ptyId: null beyond 30 seconds without an active attachment handshake, the reconciler automatically transitions the placeholder to tombstoned/pruned.
• Blast Radius: Client store reconciler (shouldReplaceTerminalTab in src/renderer/store/terminal-tabs.ts). Eliminates 172 unclosable ghost tabs across developer workstations.
• Wire Compatibility: 100% wire-compatible; purely client-side store pruning policy with zero wire protocol changes.

1-Line Reproduction Command (Gap 2)

node -e 'const s=require("./src/main/store"); s.tabsByWorktree.set("wt-orphan", [{ id: "tab-ghost", ptyId: null, createdAt: Date.now() - 3600000 }]); s.reconcile(); console.log("Ghost retained:", s.tabsByWorktree.get("wt-orphan").length)'

Injects a 1-hour-old ptyId: null placeholder into tabsByWorktree; reconciler retains the unbacked tab indefinitely rather than pruning it after 30 seconds.

Gap 3: Offline Paired Client Reconciliation Pass & Routing (Issue #21341, #21236, #21295)

• Tracked Issue: #21341 (Related: #21236, #21235, #21295)
• Associated Draft PR: PR #21332 (fix(routing): treat unhydrated runtime catalog as pending to prevent tab drop (#21295))
• Subsystem / Mechanism: Multi-Client Session Sync / Routing / Catalog Hydration
• Observed Defect: While PR fix(session): scope agent resume to the host that captured the session #21288 fences active session resumes by executionHostId, when a secondary paired client (e.g., a developer laptop) has been offline or asleep while tasks completed on the primary headless server, waking the paired client does not execute an offline catch-up reconciliation pass. The waking client's local store still holds outdated terminal and tab references, causing it to display stale dead tabs or attempt reconnection to terminated sessions.
• Required Architectural Fix: Introduce a formal post-wake / reconnect reconciliation handshake that invalidates cached multi-host tab rosters against the server's authoritative active session manifest before hydrating client view state, and treat unhydrated runtime catalogs as pending (PR fix(routing): treat unhydrated runtime catalog as pending to prevent tab drop (#21295) #21332).
• Blast Radius: Client catalog hydration and tab routing resolver during cold reconnect and sleep-wake cycles. Prevents premature tab deletion and resurrection loops.
• Wire Compatibility: Fully backward-compatible; defers catalog pruning and marks catalog pending during connection warmup; no RPC schema changes.

1-Line Reproduction Command (Gap 3)

orca client simulate-wake --stale-cache '{"tabs":["tab-reaped-1","tab-reaped-2"]}' --remote 10.x.x.x

Simulates client waking with cached tabs that were reaped while sleeping; waking client writes stale tabs back to host orca-data.json, resurrecting dead ghost tabs across active clients.

Gap 4: TerminalErrorToast Dismiss Alert vs Pane Close (Issue #21342, #9194)

• Tracked Issue: #21342 (Related: #9194)
• Subsystem / Mechanism: UI Terminal Surface / Toast Event Boundary
• Observed Defect: When a remote terminal disconnects or closes, Orca surfaces a bilingual error banner/toast ("Remote terminal was closed / 远程终端已关闭"). In the existing UI implementation, clicking the toast dismiss button dismisses only the notification overlay alert but does not dispatch a pane close or teardown event to the parent split container. The broken, unclosable pane remains rendered in the split layout with wedged keyboard focus.
• Required Architectural Fix: Decouple notification dismissal from container lifecycle, and provide an explicit "Close Pane" action on TerminalErrorToast that safely collapses the split node, disposes the dead xterm instance, and triggers durable tombstone registration.
• Blast Radius: UI TerminalErrorToast component and split container event handlers. Decouples notification alert dismiss from pane teardown.
• Wire Compatibility: 100% wire-compatible; UI event decoupling with zero wire or RPC modifications.

1-Line Reproduction Command (Gap 4)

document.querySelector(".terminal-error-toast")?.querySelector("button[aria-label='Dismiss']")?.click(); console.assert(document.querySelectorAll(".xterm-screen").length > 0, "Pane still rendered in split layout");

Dismissing the bilingual error toast removes only the notification overlay; the broken, unclosable terminal pane remains frozen in the split tree with wedged input focus.

Gap 5: Host Daemon Restart Layout Migration Pass (Issue #21343, #9585)

• Tracked Issue: #21343 (Related: #9585)
• Subsystem / Mechanism: Host Daemon Lifecycle / Split Tree Persistence
• Observed Defect: When the headless orca serve daemon restarts (e.g., service upgrade or host reboot), reconnecting clients receive fresh runtime state. However, existing serialized tab layouts and split trees that referenced previous daemon session PTY handles are not migrated or re-bound to the new daemon instance, leaving orphaned layout trees with dead pane pointers and unhandled terminal_not_found errors.
• Required Architectural Fix: Add a layout migration reconciliation pass during daemon re-attach that walks the client's persisted split tree, maps recovered session identifiers via process incarnation tuples (hostId, pid, startTime), and gracefully prunes unrecoverable layout nodes rather than retaining ghost panes.
• Blast Radius: Client layout reconciler upon reconnect to new daemon generation. Rebinds live PTYs and prunes dead layout nodes.
• Wire Compatibility: Fully wire-compatible; leverages existing process incarnation tuples (hostId, pid, startTime) without changing wire payloads.

1-Line Reproduction Command (Gap 5)

systemctl --user restart orca-serve && orca client reattach --verify-layout-pointers

Restarting headless orca serve causes reconnecting clients to receive fresh daemon state while persisted split trees reference dead PTY handles, freezing split panes with terminal_not_found.

Gap 6: attach() Latching in Connecting on terminal_not_found (Issue #21344, #20923)

• Tracked Issue: #21344 (Related: #20923)
• Subsystem / Mechanism: PTY Transport Protocol / Attachment State Machine
• Observed Defect: While PR fix(remote-runtime): stop a shared inventory snapshot from retiring a sibling split leaf #20922 addresses inventory coalescing, when a client calls transport.attach(terminalId) and the remote daemon returns terminal_not_found (e.g., during a race condition where a split leaf was reaped), the client transport state machine remains latched in the "connecting" state indefinitely. It never transitions to "closed" or "error", blocking subsequent reconnect attempts and preventing the UI from rendering failure state or recovering.
• Required Architectural Fix: Enforce strict state machine transitions in remote-runtime-pty-transport.ts: an explicit terminal_not_found error response on attach() must immediately fail the attachment promise, transition state from "connecting" to "terminated", and allow the caller to unmount or remount cleanly.
• Blast Radius: RemoteRuntimePtyTransport attachment state machine. Transitions to 'terminated' when host returns terminal_not_found.
• Wire Compatibility: Fully backward-compatible; gracefully handles existing error code without altering protocol schema.

1-Line Reproduction Command (Gap 6)

node -e 'const { RemoteRuntimePtyTransport } = require("./src/main/runtime/remote-runtime-pty-transport"); const t = new RemoteRuntimePtyTransport("term-nonexistent"); t.attach().catch(() => console.log("Latched state:", t.connecting, t.disposed))'

Calling attach() on a reaped or missing terminal leaves the transport latched in connecting: true, blocking all future reconnect attempts and preventing clean pane disposal.

Gap 7: View Transitions API Collision Debouncing (Issue #21345)

• Tracked Issue: #21345 (Canary Finding: 2,707 InvalidStateErrors / React Minified Error 185: Maximum update depth exceeded)
• Subsystem / Mechanism: Client UI Renderer / Animation Engine
• Observed Defect: Rapid successive tab switching, reconciler remounts, or split resizing trigger repeated invocations of document.startViewTransition(). Under the W3C View Transitions API specification, starting a transition while another transition is pending snapshot capture or execution immediately aborts the previous transition and throws an InvalidStateError. Unhandled rejections flood the microtask queue, stall the Chromium event loop, and cause severe render thrashing.
• Required Architectural Fix: Implement a debounced view transition coordinator that queues or skips transition animations when a transition is already active, wraps document.startViewTransition() in a safe try/catch wrapper that gracefully falls back to synchronous DOM updates on abort, and isolates tab switches from unhandled promise rejections.
• Blast Radius: Client UI view transition coordinator and error boundary. Catches InvalidStateError and throttles rapid tab remount transitions.
• Wire Compatibility: 100% wire-compatible; client-side animation guard with zero wire or server impact.

1-Line Reproduction Command (Gap 7)

for (let i = 0; i < 10; i++) document.startViewTransition(() => { document.body.dataset.tab = i; });

Fires concurrent view transitions without debouncing or abort guards; triggers 2,707 unhandled InvalidStateError: Transition was aborted because of invalid state and React Minified Error 185: Maximum update depth exceeded crashes.

📈 3. Fleet Telemetry & Debug Logs

Real-World Production Telemetry Summary

Across 12 projects and 2 enterprise bare-metal Linux servers running 24/7 headless execution paired with macOS and Windows desktop clients, a continuous forensic telemetry audit recorded severe stability failures:

• Headless Linux Server (10.x.x.x): 70 dead PTY subshells (90.9% leak rate) and 216 dangling processes pinning 44.9 GB / 46.0 GB RAM (97.6%) in systemd cgroups, swap thrashing at 145 blocks/sec, causing subshell spawns to fail with EAGAIN: resource temporarily unavailable.
• macOS Client Workstation (10.x.x.x): 98 ghost tabs (ptyId: null) out of 128 total tabs (76.6% ghost tab rate), 4x fatal V8 heap Out-Of-Memory (OOM) crashes at 3,970 MB private memory, 502 unhandled InvalidStateError promise rejections.
• Windows Client Workstation (10.x.x.x): 74 ghost tabs driving continuous 15-second remount storms (85 spawn-left-pane-unbound events), 2,707 unhandled InvalidStateError: Transition was aborted exceptions from colliding View Transitions, 3x React Minified Error 185 boundary crashes, 116 uncollected .xterm DOM nodes pinning terminal buffer memory.

Operational Context & Distributed Topology

Modern engineering workflows in Orca have shifted decisively toward headless, distributed topologies:

• 24/7 Remote Servers & Cloud Devboxes: Teams run persistent orca serve daemons on Linux hosts (systemd scopes, SSH remotes, WSL2 instances, and bare-metal build servers) to host long-running builds, test suites, and persistent developer environments.
• Autonomous Multi-Agent Pipelines: Agents (such as Claude, Codex, and automated chore workers) dynamically spawn git worktrees, open transient terminal splits, execute multi-step scripts, and deprovision worktrees upon task completion.
• Paired Desktop & Mobile Runtimes: Developers connect and reconnect from macOS and Windows desktop apps or companion mobile devices across laptops, workstations, and remote sessions.

In this distributed paradigm, the boundary between the Electron/React desktop client and the remote headless daemon must behave as a robust distributed system. However, Orca's terminal and tab subsystem was originally conceived for a monolithic, local process model. When scaled across remote boundaries under heavy multi-agent workloads, the decoupling between client UI state and server OS processes breaks down: daemon restarts, asynchronous worktree deletions, and application instability under resource limits expose missing invariants across the entire tab lifecycle. Crucially, the server hardware did not hit exhaustion limits; rather, the application hit its operational limits under heavy multi-agent workflows.

✅ 4. Verification & Acceptance Criteria

Verification Gates & Acceptance Criteria

• On a released build, within the scope above:
  • Closing a terminal or workspace tab whose backing worktree folder has been deleted (rm -rf) succeeds immediately without error toasts, retains client tombstones, and leaves zero ghost tabs in tabsByWorktree (verifying #21277);
  • No infinite tab flicker loops, zero aborted startViewTransition() InvalidStateErrors, zero React Minified Error 185: Maximum update depth exceeded exceptions, and zero renderer V8 Heap Out-Of-Memory crashes under repeated rapid tab dismissal;
  • Releasing 50 concurrent worker agents or tearing down remote workspaces leaves zero zombie PTYs and zero un-reaped processes on Headless Linux Server, and frees memory back to baseline (verifying #18790 and #19404);
  • Headless server restarts preserve active terminal sessions under dedicated systemd scopes (verifying #19430);
  • Querying inventory on split panes never retires or dismisses sibling panes (verifying #20922);
  • Terminal parking preserves offline scrollback buffers without data loss across client reconnects (verifying #21285);
  • Paired desktop clients never auto-resume Claude or agent sessions initiated on another host (verifying #21288);
  • Offscreen browser tabs are capped at 4 concurrent instances with hung renderers killed within 5 seconds (verifying #19628);
  • orca serve stats --json outputs live PTY allocation, memory usage, and client connection counts (verifying #10612).
• Every decision above (D0–D7) is resolved or explicitly deferred.
• Remaining limitations are documented on their own issues.

Zero PII Enforcement Policy

All hostnames, usernames, IP addresses, and customer data in this engineering dossier are sanitized to standardized placeholders (Headless Linux Server, macOS Client Workstation, Windows Client Workstation, 10.x.x.x) to comply with strict zero PII governance.

[LesleyMurfin]

Addendum — 21 Sep 2026

95 of 96 cores were idle. Every Runtime RPC we had open was waiting on one single-threaded JavaScript event loop.

This is empirical follow-up data for the 6-step tab lifecycle outlined in this issue (Launch → Synchronize → Reconnect → Close → Reconcile → Reap). We captured a live stall cycle this morning on our headless headless production server:

• Runtime: orca-ide serve --port 6768, PID 2135722, 96 CPUs assigned (Cpus_allowed 0-95)
• Transport: WireGuard <client-vpn-ip> → <server-vpn-ip>:6768
• Environment: Official unmodified AppImage release (1.4.197-feat-session-recovery)
• Telemetry: Socket queues (ss -tni), thread states (ps -eLo), serve-fg logs, and desktop CDP console
• All timestamps below are synchronized to UTC

---

1. The Operational Context: Workspace & Terminal Teardown Gaps

To give honest context on what was running on the host during this capture: we currently have ~74 background omp sessions accumulating inside the daemon slice.

This accumulation directly stems from the lifecycle gaps we're actively working through in this tracking issue:

• We recently had a fix merged for worktree deletion, which has resolved workspace cleanup.
• However, we are still debugging why terminal sessions do not reliably teardown upon exit or tab close, leaving worker processes and subshells lingering in the background (Steps 4 & 6).
• Concurrently, a workspace security scan (trivy) was running. With 95 cores completely unutilized, background tasks should never compete with or starve the primary control plane—yet because the event loop is coupled to storage latency, filesystem activity from background workspaces directly cascaded into control-plane unresponsiveness.

---

2. The Mechanics: Single-Thread Control Plane on a 96-Core System

orca-ide serve runs with 38 threads across all 96 assigned CPU cores. However, JavaScript execution and Runtime RPC dispatch are strictly pinned to a single main thread (TID == PID 2135722). Auxiliary worker threads (Chrome_IOThread at 0%, NetworkService at 0.1%, libuv-worker pool at 0.3%) do not dispatch JavaScript callbacks.

While host CPU utilization was only ~11% across the 96 cores, the Linux 1-minute load average spiked from 21 to 62. On Linux, load average counts both runnable threads (R) and threads waiting in uninterruptible disk sleep (D). During this stall, the main thread was repeatedly trapped in D-state waiting on synchronous filesystem I/O:

xychart-beta
    title "CPU Utilization During Stall Window (96-Core Host)"
    x-axis ["Headless Main Thread (TID==PID)", "Auxiliary IO Threads", "Aggregate Host CPU (96 Cores)"]
    y-axis "CPU %" 0 --> 15
    bar [10, 0, 11]

Loading

Because worktree directory scanning and PTY history serialization run on the primary JavaScript event loop rather than an isolated worker pool, storage latency from background workspaces directly couples host I/O stalls to network control-plane availability:

flowchart LR
  C["Desktop Client (Runtime RPC)"] --> RQ["Kernel TCP Recv-Q (:6768)\n[Accumulating Unread Bytes]"]
  RQ --> MAIN["Main Thread (TID == PID)\n[Trapped in Sync Full Scans & Checkpoints]"]
  REST["95 Idle Cores\nAuxiliary IO Threads (0% CPU)\nparcel-watcher (Child Process)"]
  REST -.->|"Cannot Service RPC Dispatch"| RQ

Loading

---

3. The Internal Log Signature: Stacking Scans on the Main Loop

serve-fg confirms that when an admin fingerprint probe times out, the server runtime initiates a full recursive directory crawl on that same main thread. In a single 80-line log tail, we recorded 41 full-scan triggers and 23 checkpoint operations:

[worktree-scan] admin fingerprint probe expired; running a full scan { timeoutMs: 3500 }
[history] checkpoint failed … DaemonRequestTimeoutError: takePendingOutput timed out after 30000ms

While the main thread is blocked executing recursive worktree traversal and awaiting PTY scrollback drain (takePendingOutput), it cannot turn the libuv loop to consume incoming WebSocket frames from kernel buffers.

---

4. Network Telemetry: Kernel Recv-Q Accumulation vs. Desktop Timeouts

The operating system kernel accepted all TCP connections normally (LISTEN Recv-Q stayed 0–2). Socket buffers had abundant headroom (tcp_rmem max is 16 MiB).

However, because the server runtime stopped calling read() / recv() on the established socket descriptors, unread bytes backed up in kernel socket memory (ESTAB Recv-Q climbed from 0 to 11,107 bytes in ~40 seconds). The moment this queue crested, the desktop client's keepalive threshold was breached:

xychart-beta
    title "ESTAB Recv-Q Max (Unread Bytes) vs 1m Load Average (UTC)"
    x-axis ["05:54:18","05:55:03","05:55:23","05:55:34","05:55:45","05:57:36","05:59:49","06:00:36"]
    y-axis "Recv-Q Bytes" 0 --> 12000
    line [0, 1566, 10866, 11107, 0, 742, 5706, 5707]

Loading

xychart-beta
    title "Linux 1-Minute Load Average Over Same Stall Window (UTC)"
    x-axis ["05:54:18","05:55:03","05:55:23","05:55:34","05:55:45","05:57:36","05:59:49","06:00:36"]
    y-axis "Load Average" 0 --> 70
    line [36.8, 33.9, 45.8, 61.3, 58.1, 40.1, 39.9, 61.0]

Loading

Correlating server socket telemetry with the desktop CDP console down to millisecond precision (converted to UTC):

UTC Timestamp	Kernel Recv-Q	Loadavg	Main Thread State	Desktop Client Event (CDP Console)
05:54:52	0 B	35.7	R (Runnable)	Normal operation.
05:55:03	1,566 B	33.9	D (Disk Sleep)	Main thread blocks on filesystem I/O. Socket buffer begins accumulating.
05:55:34.90	11,107 B	61.3	R ↔ D Thrash	Socket queue peaks; daemon event loop is unresponsive.
05:55:39.309	11,107 B	~60	D (Disk Sleep)	Desktop drops connection: RuntimeRpcCallError: Remote Orca runtime stopped responding; resetting the control connection (at 23:55:39.309 MDT).
05:55:54.175	CLOSE-WAIT	58.1	R (Runnable)	Sockets abandon into CLOSE-WAIT. Host session recovery fails on client with Tailscale timeout string (23:55:54.175 MDT).
05:56:57.352	0 B	47.0	S (Sleeping)	Socket drains. Catch-up output floods client, producing xterm code: 127 parse events (23:56:57.352 MDT).
05:59:49	5,707 B	61.0	R ↔ D Thrash	Client reset storm repeats; extra sockets (59094, 59098, 59102) abandoned.

sequenceDiagram
  participant C as Desktop Client
  participant K as Kernel TCP Buffer (:6768)
  participant M as Server Main Thread (TID==PID)
  participant D as Disk & Filesystem
  C->>K: Runtime RPC / WebSocket Heartbeat
  Note over K: Kernel buffers bytes (Recv-Q: 0 to 11,107 bytes)
  M->>D: Synchronous full scan & checkpoint
  D-->>M: Uninterruptible I/O wait (D-state)
  Note over M: 95 Cores Idle — Loop Blocked
  K--xM: recv() blocked, socket queue fills
  C->>C: Timeout — "Resetting control connection"
  C->>K: RST / Reconnect Storm (CLOSE-WAIT pileup)

Loading

---

5. Alignment with the 6-Step Distributed Tab Lifecycle

Step	Lifecycle Role	Observed Point of Contention
Step 1: Launch & Bind	Host PTY & worktree allocation	Fingerprint probe timeouts (3500ms) kick off recursive full scans directly on the primary event loop.
Step 2: Synchronize	Remote snapshot propagation & catalog hydration	Synchronous checkpoints block the event loop, triggering DaemonRequestTimeoutError: takePendingOutput timed out after 30000ms.
Step 3: Reconnect & Attach	Reconnection after transient drop	Head-of-line blocking: Incoming connect handshakes sit in TCP Recv-Q waiting for Step 2 scans to yield the thread, causing client RPC resets.
Step 4: Close & Teardown	Idempotent tab teardown	Rapid client reconnect attempts abandon half-open sockets into CLOSE-WAIT faster than the daemon can prune them.
Step 5: Reconcile & Prune	Placeholder cleanup & ViewTransition handling	Unhandled connection drops cause the client to revive dead PTY handles as null placeholders.
Step 6: Reap & Clean	OS process reaping & cgroup cleanup	Long-running background processes and unreaped process trees pin cgroup memory, increasing kernel page reclaim latency during Step 2 scans.

---

6. Seeking Maintainer Perspective & Collaboration

Rather than making prescriptive assertions, we wanted to share this empirical telemetry to see how it aligns with your internal performance benchmarks:

1. Event Loop Isolation: In multi-tenant and high-density enterprise environments, have you considered offloading recursive worktree tree-scans ([worktree-scan]) or PTY scrollback history serialization (takePendingOutput) to an isolated worker thread (worker_threads) or helper process, so that transport heartbeats and Runtime RPCs remain fully responsive even during heavy background workspace I/O?
2. Terminal Lifecycle Telemetry: We recently validated that workspace deletion cleanup landed cleanly in our environment (thank you!), but we are still tracing why terminal sessions occasionally linger and fail to cleanly terminate their child process trees upon tab close. Any guidance or pointers on where best to hook or instrument the terminal teardown flow on headless serve would be immensely valuable.
3. Transport-Layer Decoupling: Would it be beneficial to have the transport layer acknowledge ping/pong heartbeats and basic health probes independently from deep catalog or storage serialization, preventing desktop clients from entering reconnect cascades during transient storage spikes?

We run dedicated bare-metal infrastructure simulating heavy, sustained multi-agent enterprise workloads 24/7, instrumented with continuous kernel, socket, and cgroup-level telemetry. Because these distributed lifecycle edges only reveal themselves under high-density concurrency that standard edge environments rarely capture, we're very glad to act as an extended testing ground—running candidate builds through our soak harness, stress-testing patches touching the transport/PTY boundaries, or sharing any deeper kernel traces that would help the core team.

We're deeply invested in Orca's headless and multi-agent future, and would welcome collaborating closely with Neil, Jinjing, or the runtime team on hardening these lifecycle paths as Orca scales into larger enterprise and cloud deployments.