fix: self-heal sessions evicted from the backend (Session is not active -32004)

CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.11.3] - 2026-08-22
+
+### Fixed
+
+- Sessions evicted from the backend no longer break prompting or load empty:
+  the zcode backend evicts idle resident runtimes (~10min idle timeout plus
+  an LRU cap), after which every session-scoped RPC fails with
+  `Session is not active` (-32004). The bridge now self-heals on every entry
+  point — `session/prompt` reloads via `session/resume` and retries the
+  subscribe once (re-baselining the differ so turn completion doesn't replay
+  history), `session/load`·`resume` stop trusting a stale loaded-verification
+  flag (5-minute TTL, `BACKEND_RESIDENT_TTL_MS`), and `ensureRealSession`
+  reloads stale mappings for config/slash/extension calls (skipped while a
+  turn is in flight, fail-safe on reload failure).
+- Tests for the eviction recovery paths in `tests/session-eviction.test.ts`.
+- Troubleshooting guide: dedicated `Session is not active` (-32004) entry,
+  corrected the subscribe timeout retry numbers.
+
 ## [0.11.2] - 2026-08-20
 
 ### Fixed

docs/TROUBLESHOOTING.md

@@ -68,12 +68,35 @@ a hardcoded version string — read the message text to identify the root cause.
 **Common causes:**
 
 | Message fragment                 | Cause                                                                                                                                                                                                                                                      |
-| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `reader exited (backend dead)`   | The zcode subprocess crashed/exited. Restart the editor session.                                                                                                                                                                                           |
-| `timeout`                        | The per-attempt 10s subscribe deadline elapsed. The bridge retries transient timeouts up to 3× (the backend can be briefly busy finalising a cancelled turn after a preempt / `session/stop`); if all retries fail, the backend was unresponsive for ~30s. |
+| `timeout`                        | The per-attempt 5s subscribe deadline elapsed. The bridge retries transient timeouts once (2 attempts total, ~10.5s worst case); if both fail, the backend was unresponsive for that window.                                                               |
 | `pipe broken`                    | The stdin pipe to the zcode subprocess broke (process died mid-write).                                                                                                                                                                                     |
 | `method not found (code -32601)` | The CLI genuinely is too old (< 0.14.8). Upgrade.                                                                                                                                                                                                          |
-| session-level business error     | The target session no longer exists or was evicted.                                                                                                                                                                                                        |
+| `Session is not active` (-32004) | The backend evicted the session's resident runtime (idle ~10min, or its LRU cap). The bridge self-heals via `session/resume` (see below).                                                                                                                   |
+
+**`Session is not active` (code -32004) in detail:**
+
+The zcode backend keeps session runtimes ("residents") in memory and evicts
+them after ~10 minutes idle (log event `session.resident_deactivated`,
+`reason: "idle_timeout"`) or under its resident LRU cap. An evicted session
+fails every session-scoped RPC with `-32004` while the session file stays
+intact — the editor still shows its local copy of the conversation, but
+sending a message errors and remote clients replay an empty session.
+
+The bridge self-heals on every entry point:
+
+- `session/prompt` reloads the session via `session/resume` and retries the
+  subscribe once when it sees this error;
+- the "loaded in backend" verification carries a 5-minute TTL
+  (`BACKEND_RESIDENT_TTL_MS`), so `session/load` / `session/resume` re-issue
+  the backend resume RPC instead of trusting a stale in-memory flag;
+- `ensureRealSession` (config/slash/extension entry points) reloads a stale
+  mapping before use, unless a turn is in flight.
+
+If the error still surfaces, the resume itself is failing — check the backend
+log (`~/.zcode/cli/log/zcode-YYYY-MM-DD.jsonl`) for the underlying cause
+(corrupt session file, lock contention from another zcode process).
 
 **Troubleshooting steps:**
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "zcode-acp-server",
-  "version": "0.11.2",
+  "version": "0.11.3",
   "description": "Agent Client Protocol (ACP) server bridging headless ZCode to editors like Zed and JetBrains.",
   "type": "module",
   "license": "Apache-2.0",

src/handlers/session.ts

@@ -130,7 +130,29 @@ export async function newSession(
  */
 export async function ensureRealSession(server: ZcodeAcpServer, acpSid: string): Promise<string> {
   const existing = server.resolveSid(acpSid);
-  if (existing) return existing;
+  if (existing) {
+    // The mapping exists, but the backend may have evicted the resident
+    // runtime since it was loaded (~10min idle timeout + LRU cap): every
+    // session-scoped RPC would then fail with "Session is not active"
+    // (-32004). Reload via session/resume when the verification went stale
+    // and no turn is in flight (a running turn proves the resident is live).
+    // Fail-safe: a failed reload just returns the mapping — the subsequent
+    // RPC surfaces the backend's real error, same as before this guard.
+    if (server.isBackendSessionLive(acpSid)) return existing;
+    const turnInFlight = [...server.pendingTurns.values()].some((t) => t.zcodeSid === existing);
+    if (!turnInFlight) {
+      try {
+        log(`ensureRealSession: ${acpSid} possibly evicted from backend — reloading`);
+        await reloadBackendSession(server, acpSid, existing);
+      } catch (e) {
+        log(
+          `ensureRealSession: reload failed, continuing with existing mapping ` +
+            `(${e instanceof Error ? e.message : String(e)})`,
+        );
+      }
+    }
+    return existing;
+  }
   let pending = server.pendingSessions.get(acpSid);
   if (!pending) {
     // Placeholder from a previous bridge lifetime: recover it from the durable
@@ -188,7 +210,7 @@ export async function ensureRealSession(server: ZcodeAcpServer, acpSid: string):
     server.pendingSessions.delete(acpSid);
     server.registerSession(acpSid, sid);
     // session/create loads the session into this backend process.
-    server.backendLoadedSessions.add(acpSid);
+    server.markBackendLoaded(acpSid);
     // Keep the durable alias in sync so a later bridge restart can still
     // resume this session via the placeholder id.
     recordMaterializedSession(acpSid, sid, pending.cwd);
@@ -281,10 +303,11 @@ async function adoptStoredTitle(
  * the editor may resume it anyway (panel reopen, bridge restart) — resolving it
  * here prevents an otherwise unavoidable "Session not found". Resolution order:
  *   1. in-memory mapping → live only if verified loaded in this backend
- *      subprocess (`backendLoadedSessions`); a bare mapping may have been
- *      re-registered from the durable store without a resume, and the backend
- *      only serves messages for sessions it has loaded — those must fall
- *      through to the resume RPC or the replay comes back empty;
+ *      subprocess RECENTLY (`isBackendSessionLive`); a bare mapping may have
+ *      been re-registered from the durable store without a resume, and the
+ *      backend also evicts idle resident runtimes (~10min) — either way it
+ *      only serves messages for sessions with a live resident, so those must
+ *      fall through to the resume RPC or the replay comes back empty;
  *   2. pending placeholder → materialize it (an empty session, matching the
  *      pre-lazy behavior where a never-used session/new always resumed);
  *   3. durable store → a placeholder from a previous bridge lifetime: with a
@@ -300,7 +323,7 @@ async function resolveResumeTarget(
 ): Promise<{ zcodeSid: string; alreadyLive: boolean }> {
   const mapped = server.resolveSid(acpSid);
   if (mapped) {
-    return { zcodeSid: mapped, alreadyLive: server.backendLoadedSessions.has(acpSid) };
+    return { zcodeSid: mapped, alreadyLive: server.isBackendSessionLive(acpSid) };
   }
   if (server.pendingSessions.has(acpSid)) {
     return { zcodeSid: await ensureRealSession(server, acpSid), alreadyLive: true };
@@ -356,7 +379,7 @@ export async function resumeSession(
     await syncProviderRegistry(server, cwd);
     await resumeBackendSession(server, zcParams);
     // The resume RPC succeeded — the session is now loaded in this backend.
-    server.backendLoadedSessions.add(acpSid);
+    server.markBackendLoaded(acpSid);
   }
 
   server.registerSession(acpSid, zcodeSid);
@@ -407,7 +430,7 @@ export async function loadSession(
     await syncProviderRegistry(server, cwd);
     await resumeBackendSession(server, zcParams);
     // The resume RPC succeeded — the session is now loaded in this backend.
-    server.backendLoadedSessions.add(acpSid);
+    server.markBackendLoaded(acpSid);
   }
   server.registerSession(acpSid, zcodeSid);
   // Same as resumeSession: record the cwd as the session root for file access.
@@ -560,7 +583,28 @@ export async function prompt(
   // — this call site is outside the try/finally below.
   let snapshot: ZcodeSnapshot;
   try {
-    snapshot = await listener.subscribe(() => server.nextId());
+    try {
+      snapshot = await listener.subscribe(() => server.nextId());
+    } catch (e) {
+      // The backend evicts idle resident runtimes (~10min) and can drop them
+      // under its LRU cap even sooner — an evicted session fails every
+      // session-scoped RPC with code -32004 "Session is not active" although
+      // the session file is intact. Recover by reloading it (session/resume
+      // is idempotent) and retrying the subscribe once; any other error, or
+      // a second failure, propagates to the editor.
+      const msg = e instanceof Error ? e.message : String(e);
+      if (!/session is not active/i.test(msg)) throw e;
+      log(`prompt: session ${zcodeSid} no longer active in backend — reloading via session/resume`);
+      await reloadBackendSession(server, params.sessionId, zcodeSid);
+      // The pre-subscribe fetchMessages ran against the evicted session and
+      // came back empty — re-baseline the differ so turn completion doesn't
+      // diff-replay the whole history as new output.
+      differ.markSeen(await fetchMessages(server, zcodeSid));
+      snapshot = await listener.subscribe(() => server.nextId());
+    }
+    // A successful subscribe proves the resident runtime is live — refresh
+    // the verification so concurrent/later entry points skip a reload.
+    server.markBackendLoaded(params.sessionId);
   } catch (e) {
     server.pendingTurns.delete(requestId);
     await emitTurnState(false);
@@ -768,8 +812,10 @@ export async function prompt(
     server.pendingTurns.delete(requestId);
     // Turn end = session activity — refresh the discovery summary and mark the
     // session discoverable regardless of outcome (end_turn, cancelled, retries
-    // exhausted).
+    // exhausted). Also refresh the backend-loaded verification: the resident
+    // runtime was demonstrably live through this turn.
     server.markSessionActive(params.sessionId);
+    server.markBackendLoaded(params.sessionId);
     // Report "running" only while no other turn for the session took over
     // (preempt): the preempting turn's own running:true must survive.
     const stillBusy = [...server.pendingTurns.values()].some((t) => t.zcodeSid === zcodeSid);
@@ -1147,6 +1193,28 @@ async function resumeBackendSession(
   }
 }
 
+/**
+ * Reload a session into the backend subprocess via `session/resume` — the
+ * recovery path after the backend evicted the resident runtime (idle timeout
+ * / LRU). Same param shape as session/load·resume (workspace from the
+ * recorded session cwd, runtimeModel overlay for stale history models).
+ * Marks the session backend-loaded on success.
+ */
+async function reloadBackendSession(
+  server: ZcodeAcpServer,
+  acpSid: string,
+  zcodeSid: string,
+): Promise<void> {
+  const zcParams: Record<string, unknown> = {
+    sessionId: zcodeSid,
+    workspace: workspaceFor(server.sessionCwds.get(acpSid) ?? process.cwd()),
+  };
+  const runtimeModel = buildResumeRuntimeModel();
+  if (runtimeModel !== null) zcParams.runtimeModel = runtimeModel;
+  await resumeBackendSession(server, zcParams);
+  server.markBackendLoaded(acpSid);
+}
+
 /** Get or create the session-level ProjectionDiffer (persists across turns). */
 function getOrCreateDiffer(server: ZcodeAcpServer, zcodeSid: string): ProjectionDiffer {
   let d = server.differs.get(zcodeSid);

src/server.ts

@@ -44,6 +44,17 @@ export interface PendingTurn {
   stallRecovered?: boolean;
 }
 
+/**
+ * How long a "loaded in backend" verification stays trusted. The backend
+ * evicts resident runtimes after ~10min idle (observed
+ * `session.resident_deactivated`, idleTimeoutMs 600000) and also keeps a
+ * small LRU cap, after which every session-scoped RPC fails with
+ * "Session is not active" (-32004). Trusting a verification for half the
+ * eviction window makes callers redo the resume RPC well before eviction
+ * can bite.
+ */
+export const BACKEND_RESIDENT_TTL_MS = 5 * 60_000;
+
 export class ZcodeAcpServer {
   /** The ZCode subprocess client (lazy — spawned on first use). */
   backend: ZcodeBackend | null = null;
@@ -128,16 +139,16 @@ export class ZcodeAcpServer {
   /** Session titles already set, to enforce set-once (acp_sid → title). */
   readonly sessionTitles = new Map<string, string>();
   /**
-   * Sessions verified as loaded in the CURRENT backend subprocess — populated
-   * only after a successful session/create or session/resume RPC. A bare
-   * `registerSession` mapping does NOT qualify: the backend answers
-   * `session/messages` only for sessions it has loaded, so `session/load`
-   * must not skip the resume RPC for a mapping that was never loaded (e.g.
-   * re-registered from the durable store by an early ensureRealSession
-   * caller, or left behind by a failed resume) — the replay would silently
-   * come back empty.
+   * Sessions verified as loaded in the CURRENT backend subprocess, with the
+   * verification timestamp — populated only after a successful
+   * session/create or session/resume RPC and refreshed when a turn runs. A
+   * bare `registerSession` mapping does NOT qualify, and neither does an old
+   * timestamp: the backend answers `session/messages` only for sessions with
+   * a live resident runtime, so `session/load` must not skip the resume RPC
+   * for those (the replay would silently come back empty). Use
+   * `markBackendLoaded`/`isBackendSessionLive` instead of touching the map.
    */
-  readonly backendLoadedSessions = new Set<string>();
+  private readonly backendLoadedSessions = new Map<string, number>();
   /**
    * Sessions eligible for auto-title on first end_turn. Only `session/new`
    * populates this — resumed/loaded sessions already carry a title, so their
@@ -211,6 +222,21 @@ export class ZcodeAcpServer {
     this.touchSessionSummary(acpSid);
   }
 
+  /** Record that a session is loaded in the current backend subprocess (now). */
+  markBackendLoaded(acpSid: string): void {
+    this.backendLoadedSessions.set(acpSid, Date.now());
+  }
+
+  /**
+   * True when the session was verified backend-loaded recently enough that the
+   * backend's resident idle eviction (~10min) can't have dropped it. Stale or
+   * unknown entries count as NOT live so callers redo the session/resume RPC.
+   */
+  isBackendSessionLive(acpSid: string): boolean {
+    const at = this.backendLoadedSessions.get(acpSid);
+    return at !== undefined && Date.now() - at < BACKEND_RESIDENT_TTL_MS;
+  }
+
   /** Update a session's discovery summary (title sticky once set). */
   touchSessionSummary(acpSid: string, title?: string): void {
     const existing = this.sessionSummaries.get(acpSid);