Fence CLI and Bot stale lease recovery

.memory/aiden-cli-standalone.md

@@ -1,7 +1,11 @@
 # Aiden CLI standalone (PR #121, target 0.50.0)
 
-- `packages/cli` is its own npm package. It is not included in the desktop
-  electron-builder `files` (`build/**` only), so desktop packaging is unaffected.
+## Lease ownership (issues #405 and #392, 2026-10-15)
+
+CLI `acquireLease` holds an OS `flock` on a permanent `<resource>.lease.lock` file until release. A short native helper locks a duplicate of the parent's descriptor, then exits; the parent keeps the lock through async work, and process death releases it. A raw read of the guard cannot cancel the lock. The former `<resource>.lock/owner.json` is diagnostic metadata, published and removed only under the guard; malformed values including JSON `null` are stale. Serve recovery, including foreground startup, and stop cleanup acquire that same guard, recheck the owner, and remove stale metadata and socket only while holding it. New metadata carries `protocol: "flock-v1"`, so a recycled live PID cannot wedge recovery after the guard is free; legacy live metadata remains protected unless its PID is verifiably foreign. The daemon child removes a stale socket after it acquires the serve lease. Never unlink, rename, or copy over guard files. Stop older CLI processes and daemons before upgrading a shared agent directory; mixed-version writers do not share this protocol.
+
+- `packages/cli` is its own npm package. The shared lease helper is copied into
+  `dist/app/native/` for CLI bundles and `Helpers/` by desktop electron-builder.
 - Several desktop services are split into portable `*-core.ts` modules
   (schedule-store, subagent-supervisor, vision-analysis-tool, local-speech-downloads,
   aiden-remote-speech, subagent-mcp-mutation-host). When main changes one of

.memory/bot-durable-runtime.md

@@ -4,7 +4,7 @@ Plan: `docs/superpowers/plans/2026-10-07-bots-rework.md`. Spec: `docs/superpower
 
 ## Layout
 - `main/services/bot-runtime/`
-  - `profile-lock.ts`: `<profile>/bots/runtime.lock` (`{pid, startedAt, token}`), published with `writeFileAtomic({exclusive})`. Live pid (or this process's own token) refuses; a dead pid or torn file is reclaimed by moving it aside first.
+  - `profile-lock.ts`: `<profile>/bots/runtime.lease.lock` is a permanent OS `flock` guard held through a parent-owned descriptor for the host lifetime. The `runtime.lock` JSON record is diagnostic; stale new-protocol metadata is replaced only while the guard is held. A crashed process releases the OS lock without path reclamation, and a raw guard read cannot cancel it. Host opens serialize behind closes, and failed teardown entries cannot be reused. Shutdown closes mutation admission, drains in-flight operations and harnesses, then releases the guard; an old host cannot mutate a successor's data. Never delete or replace the guard. Stop old Bot hosts before upgrading a shared profile because old binaries do not honor it.
   - `harness-host.ts`: one Pi Durable harness per Bot at `<profile>/bots/<dir>/session.sqlite`. `<dir>` is `botDirectoryName(botId)`: characters outside `[A-Za-z0-9_-]` become `~xx` UTF-8 bytes (`bot:1` → `bot~3a1`), because Bot ids contain `:`. Lazy open, 10-minute idle close (only with no live task or submission), corrupt DB (SQLite errcode 26/11) moved to `session.sqlite.corrupt-<ts>` and a fresh root opened, `destroy` = abort (bounded 10 s) → close → `rm -rf`, deleted ids refuse reopen.
   - `tool-adapter.ts`: pi-agent-core `AgentTool` → Pi Durable tool (TypeBox args validated by the harness, chord signal → `AbortSignal`, `onUpdate` snapshots → appended `api.output`, details → `api.details`).
   - `bot-extension.ts`: `aiden-bot` extension. Sections `aiden-base`, `aiden-persona`, `aiden-authority` (untagged, in that order). `refresh({requestId})` reloads Bot/sections/tools and reinstalls the extension; call it before every submit/resume. `beforeTool` checks policy; approvals persist `{waitId, toolCallId, summary}` in the tool task memo `aiden.approval`.

.memory/lease-reclaim-fencing.md

@@ -0,0 +1,5 @@
+# Lease reclaim fencing
+
+PR #499 protects CLI leases and Bot profiles with permanent native `flock` guards held through inherited descriptors. Owner JSON is diagnostic; a free guard permits stale-owner recovery, including recycled PIDs. Bot shutdown closes mutation admission, drains admitted work, and releases the profile guard last. Keep the root test-chain union and Aiden Remote revision aligned when merging main.
+
+The bounded vendor-command descendant cleanup test is adjacent CI coverage, not part of the lease implementation. Its leader deliberately delays spawning a descendant that retains stdout and ignores SIGTERM. The descendant writes its PID only after installing that handler. The test holds the parent command timer with Node's test clock until this readiness file exists, then advances the original 250 ms timeout and verifies prompt rejection and descendant exit. This separates fixture startup from timeout cleanup without relaxing the production timeout, the test deadline, or `--fail-on-flaky-tests`. The earlier CI `ENOENT` and its run link remain in the PR description as a real flaky symptom.

main/services/bot-runtime/harness-host.test.ts

@@ -89,6 +89,65 @@ test("(a2) an idle harness closes by itself and reopens mid-conversation", async
   }
 });
 
+test("open waits for an overlapping close and returns a usable new session", async () => {
+  const profile = tempProfile();
+  const host = await hostFor(profile, createFauxModels());
+  const first = await host.open("bot-overlap");
+  const originalClose = first.harness.close.bind(first.harness);
+  let closeStarted!: () => void;
+  let finishClose!: () => void;
+  const entered = new Promise<void>((resolve) => { closeStarted = resolve; });
+  const gate = new Promise<void>((resolve) => { finishClose = resolve; });
+  first.harness.close = async (context) => {
+    closeStarted();
+    await gate;
+    return originalClose(context);
+  };
+  const closing = host.close("bot-overlap");
+  try {
+    await entered;
+    let reopened = false;
+    const reopening = host.open("bot-overlap").then((result) => {
+      reopened = true;
+      return result;
+    });
+    await Promise.resolve();
+    assert.equal(reopened, false, "open must wait for the old session to close");
+    finishClose();
+    await closing;
+    const second = await reopening;
+    assert.notEqual(second.harness, first.harness);
+    assert.equal(second.conversation.id, first.conversation.id);
+    await second.conversation.context(ctx);
+  } finally {
+    finishClose();
+    await closing.catch(() => undefined);
+    await host.shutdown();
+  }
+});
+
+test("failed close retains the profile guard and open retries teardown", async () => {
+  const profile = tempProfile();
+  const host = await hostFor(profile, createFauxModels());
+  const first = await host.open("bot-close-retry");
+  const originalClose = first.harness.close.bind(first.harness);
+  let attempts = 0;
+  first.harness.close = async (context) => {
+    if (++attempts === 1) throw new Error("teardown failed");
+    return originalClose(context);
+  };
+  try {
+    await assert.rejects(host.close("bot-close-retry"), /teardown failed/);
+    const contender = spawnHarnessChild("lock", [path.join(profile, "bots")]);
+    try {
+      assert.equal(JSON.parse(await contender.waitFor("LOCK")).ok, false);
+    } finally { await contender.kill(); }
+    const reopened = await host.open("bot-close-retry");
+    assert.notEqual(reopened.harness, first.harness);
+    await reopened.conversation.context(ctx);
+  } finally { await host.shutdown(); }
+});
+
 test("(b) after SIGKILL mid-stream the scan reports the Bot without calling the provider", async () => {
   const profile = tempProfile();
   const child = spawnHarnessChild("stream", [profile, "bot:killed", "req-1"]);
@@ -162,7 +221,7 @@ test("(d) sweepOrphans removes directories with no known Bot", async () => {
     writeFileSync(path.join(profile, "bots", "bot-orphan", BOT_SESSION_FILE), "");
     const removed = await host.sweepOrphans(new Set(["bot:known"]));
     assert.deepEqual(removed, ["bot-orphan"]);
-    assert.deepEqual(readdirSync(path.join(profile, "bots")).sort(), ["bot~3aknown", "runtime.lock"]);
+    assert.deepEqual(readdirSync(path.join(profile, "bots")).sort(), ["bot~3aknown", "runtime.lease.lock", "runtime.lock"]);
   } finally {
     await host.shutdown();
   }
@@ -195,6 +254,54 @@ test("(e2) the profile is available again after shutdown", async () => {
   }
 });
 
+test("a shut down host cannot delete a replacement host's Bot", async () => {
+  const profile = tempProfile();
+  const oldHost = await hostFor(profile, createFauxModels());
+  await oldHost.shutdown();
+  const replacement = await hostFor(profile, createFauxModels());
+  try {
+    await replacement.open("bot-replacement");
+    const session = path.join(profile, "bots", "bot-replacement", BOT_SESSION_FILE);
+    assert.equal(existsSync(session), true);
+    await assert.rejects(oldHost.destroy("bot-replacement"), /shut down/);
+    await assert.rejects(oldHost.sweepOrphans(new Set()), /shut down/);
+    await assert.rejects(oldHost.close("bot-replacement"), /shut down/);
+    assert.throws(() => oldHost.touch("bot-replacement"), /shut down/);
+    assert.equal(existsSync(session), true);
+    assert.equal(replacement.isOpen("bot-replacement"), true);
+  } finally { await replacement.shutdown(); }
+});
+
+test("shutdown holds the profile guard until an admitted inspection finishes", async () => {
+  const profile = tempProfile();
+  const host = await hostFor(profile, createFauxModels());
+  const entry = await host.open("bot-inspected");
+  const inspect = entry.harness.inspect.bind(entry.harness);
+  let started!: () => void;
+  let resume!: () => void;
+  const entered = new Promise<void>((resolve) => { started = resolve; });
+  const gate = new Promise<void>((resolve) => { resume = resolve; });
+  entry.harness.inspect = async (context) => {
+    started();
+    await gate;
+    return inspect(context);
+  };
+  const pending = host.inspect("bot-inspected");
+  await entered;
+  const shutdown = host.shutdown();
+  const contender = spawnHarnessChild("lock", [path.join(profile, "bots")]);
+  try {
+    assert.equal(JSON.parse(await contender.waitFor("LOCK")).ok, false);
+  } finally {
+    resume();
+    await contender.kill();
+  }
+  await pending;
+  await shutdown;
+  const replacement = await hostFor(profile, createFauxModels());
+  await replacement.shutdown();
+});
+
 test("(f) a corrupt session file is moved aside and a fresh conversation opens", async () => {
   const profile = tempProfile();
   const dir = path.join(profile, "bots", "bot-corrupt");

main/services/bot-runtime/harness-host.ts

@@ -24,7 +24,7 @@ import {
   type Registry,
 } from "@earendil-works/pi-durable";
 import { openNodeSqliteStorage } from "@earendil-works/pi-durable/storage/sqlite/node";
-import { acquireBotProfileLock, BOT_PROFILE_LOCK_FILE } from "./profile-lock.js";
+import { acquireBotProfileLock, BOT_PROFILE_LOCK_FILE, BOT_PROFILE_GUARD_FILE } from "./profile-lock.js";
 
 export const BOT_SESSION_FILE = "session.sqlite";
 export const DEFAULT_BOT_IDLE_CLOSE_MS = 600_000;
@@ -184,12 +184,23 @@ export async function createBotHarnessHost(
   const ctx: Context = BACKGROUND_CONTEXT;
   const idleCloseMs = opts.idleCloseMs ?? DEFAULT_BOT_IDLE_CLOSE_MS;
   const now = opts.now ?? Date.now;
-  const opening = new Map<string, Promise<OpenEntry>>();
   const open = new Map<string, OpenEntry>();
+  /** Keep failed teardowns out of the reusable-entry path. */
+  const closing = new Set<string>();
   const deleted = new Set<string>();
   /** Per-Bot serialization of open/close/destroy. */
   const lifecycle = new Map<string, Promise<unknown>>();
+  const admittedOperations = new Set<Promise<unknown>>();
   let shutDown = false;
+  let shutdownPromise: Promise<void> | null = null;
+
+  function admit<T>(action: () => Promise<T>): Promise<T> {
+    if (shutDown) return Promise.reject(new BotHarnessHostClosedError());
+    const operation = action();
+    admittedOperations.add(operation);
+    void operation.finally(() => admittedOperations.delete(operation)).catch(() => undefined);
+    return operation;
+  }
 
   const botDir = (botId: string) => path.join(botsDir, botDirectoryName(botId));
   const sessionFile = (botId: string) => path.join(botDir(botId), BOT_SESSION_FILE);
@@ -248,12 +259,13 @@ export async function createBotHarnessHost(
     if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
     entry.idleTimer = setTimeout(() => {
       entry.idleTimer = null;
-      void closeIfIdle(botId, entry);
+      void admit(() => closeIfIdle(botId, entry)).catch(() => undefined);
     }, idleCloseMs);
     entry.idleTimer.unref?.();
   }
 
   async function closeIfIdle(botId: string, entry: OpenEntry): Promise<void> {
+    if (shutDown) return;
     if (open.get(botId) !== entry) return;
     try {
       const inspection = await entry.harness.inspect(ctx);
@@ -264,25 +276,26 @@ export async function createBotHarnessHost(
     } catch {
       // A harness that cannot be inspected is closed below.
     }
-    await host.close(botId).catch(() => undefined);
+    if (!shutDown) {
+      await serialize(botId, () => open.get(botId) === entry ? closeEntry(botId) : Promise.resolve()).catch(() => undefined);
+    }
   }
 
   async function ensureOpen(botId: string): Promise<OpenEntry> {
     assertBotId(botId);
     if (shutDown) throw new BotHarnessHostClosedError();
     if (deleted.has(botId)) throw new BotDeletedError(botId);
-    const existing = open.get(botId);
-    if (existing !== undefined) {
-      scheduleIdle(botId, existing);
-      return existing;
-    }
-    const pending = opening.get(botId);
-    if (pending !== undefined) return pending;
-    const promise = serialize(botId, async () => {
+    return serialize(botId, async () => {
       if (shutDown) throw new BotHarnessHostClosedError();
       if (deleted.has(botId)) throw new BotDeletedError(botId);
       const already = open.get(botId);
-      if (already !== undefined) return already;
+      if (already !== undefined) {
+        if (closing.has(botId)) await closeEntry(botId);
+        else {
+          scheduleIdle(botId, already);
+          return already;
+        }
+      }
       await fs.mkdir(botDir(botId), { recursive: true, mode: 0o700 });
       let opened: { harness: Harness; conversation: Conversation };
       let notice: string | undefined;
@@ -295,6 +308,7 @@ export async function createBotHarnessHost(
       }
       const entry: OpenEntry = { ...opened, idleTimer: null, ...(notice === undefined ? {} : { pendingNotice: notice }) };
       open.set(botId, entry);
+      closing.delete(botId);
       scheduleIdle(botId, entry);
       // pi-durable 1.2 closes a Session for good when any Storage call fails;
       // drop it so the next open reopens and recovers from what was committed.
@@ -306,20 +320,19 @@ export async function createBotHarnessHost(
       });
       return entry;
     });
-    opening.set(botId, promise);
-    try {
-      return await promise;
-    } finally {
-      if (opening.get(botId) === promise) opening.delete(botId);
-    }
   }
 
   async function closeEntry(botId: string): Promise<void> {
     const entry = open.get(botId);
-    if (entry === undefined) return;
-    open.delete(botId);
+    if (entry === undefined) {
+      closing.delete(botId);
+      return;
+    }
+    closing.add(botId);
     if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
     await entry.harness.close(ctx);
+    if (open.get(botId) === entry) open.delete(botId);
+    closing.delete(botId);
   }
 
   async function listBotDirectories(): Promise<string[]> {
@@ -332,15 +345,15 @@ export async function createBotHarnessHost(
     }
     const result: string[] = [];
     for (const name of names) {
-      if (name === BOT_PROFILE_LOCK_FILE || name.startsWith(".")) continue;
+      if (name === BOT_PROFILE_LOCK_FILE || name === BOT_PROFILE_GUARD_FILE || name.startsWith(".")) continue;
       const stat = await fs.lstat(path.join(botsDir, name)).catch(() => null);
       if (stat?.isDirectory()) result.push(name);
     }
     return result;
   }
 
   const host: BotHarnessHost = {
-    async open(botId) {
+    open(botId) { return admit(async () => {
       const entry = await ensureOpen(botId);
       const notice = entry.pendingNotice;
       delete entry.pendingNotice;
@@ -349,19 +362,20 @@ export async function createBotHarnessHost(
         conversation: entry.conversation,
         ...(notice === undefined ? {} : { recoveredCorruptFile: notice }),
       };
-    },
+    }); },
 
     touch(botId) {
+      if (shutDown) throw new BotHarnessHostClosedError();
       const entry = open.get(botId);
       if (entry !== undefined) scheduleIdle(botId, entry);
     },
 
-    async inspect(botId) {
+    inspect(botId) { return admit(async () => {
       const entry = await ensureOpen(botId);
       return entry.harness.inspect(ctx);
-    },
+    }); },
 
-    async interruptedBots() {
+    interruptedBots() { return admit(async () => {
       const result: BotInterruption[] = [];
       for (const name of await listBotDirectories()) {
         const botId = botIdFromDirectoryName(name);
@@ -378,21 +392,19 @@ export async function createBotHarnessHost(
           result.push({ botId, submissionId: String(unfinished.id) });
         } else if (inspection.tasks.length === 0 && inspection.submissions.length === 0) {
           // Nothing to recover: do not keep this Bot's database open.
-          await host.close(botId);
+          await serialize(botId, () => closeEntry(botId));
         }
       }
       return result;
-    },
+    }); },
 
     close(botId) {
-      return serialize(botId, () => closeEntry(botId));
+      return admit(() => serialize(botId, () => closeEntry(botId)));
     },
 
-    async destroy(botId) {
+    destroy(botId) { return admit(async () => {
       assertBotId(botId);
       deleted.add(botId);
-      const pending = opening.get(botId);
-      if (pending !== undefined) await pending.catch(() => undefined);
       await serialize(botId, async () => {
         const entry = open.get(botId);
         if (entry !== undefined) {
@@ -411,9 +423,9 @@ export async function createBotHarnessHost(
         }
         await fs.rm(botDir(botId), { recursive: true, force: true });
       });
-    },
+    }); },
 
-    async sweepOrphans(knownBotIds) {
+    sweepOrphans(knownBotIds) { return admit(async () => {
       const removed: string[] = [];
       for (const name of await listBotDirectories()) {
         const botId = botIdFromDirectoryName(name);
@@ -425,18 +437,25 @@ export async function createBotHarnessHost(
         removed.push(name);
       }
       return removed;
-    },
+    }); },
 
     isOpen(botId) {
       return open.has(botId);
     },
 
     async shutdown() {
-      if (shutDown) return;
-      shutDown = true;
-      await Promise.allSettled([...opening.values()]);
-      await Promise.allSettled([...open.keys()].map((botId) => serialize(botId, () => closeEntry(botId))));
-      await lock.release();
+      if (shutdownPromise === null) {
+        shutDown = true;
+        shutdownPromise = (async () => {
+          await Promise.allSettled([...admittedOperations]);
+          const closes = await Promise.allSettled([...open.keys()].map((botId) => serialize(botId, () => closeEntry(botId))));
+          const failed = closes.find((result): result is PromiseRejectedResult => result.status === "rejected");
+          if (failed) throw failed.reason;
+          await Promise.all([...lifecycle.values()]);
+          await lock.release();
+        })();
+      }
+      return shutdownPromise;
     },
   };
   return host;