fix(eve): close custom worlds during graceful shutdown

.changeset/clean-workflow-world-shutdown.md

@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Self-hosted eve servers now await Workflow world and sandbox cleanup during SIGINT or SIGTERM, and local development asks worker-owned Workflow worlds to close before retiring a worker during reload or shutdown. Configured Worlds that release queue workers from `close()` can therefore resume durable sessions after restart without waiting for stale locks to expire.

docs/guides/deployment/self-hosting.md

@@ -18,6 +18,8 @@ The build writes the Nitro server under `.output/`. `eve start` serves that outp
 
 Run this process under the same process manager or container platform you use for other Node web services. Configure Transport Layer Security (TLS), scaling, restarts, and log collection in that platform.
 
+On `SIGINT` or `SIGTERM`, eve runs Nitro's coordinated close lifecycle before exiting. That lifecycle awaits the configured Workflow world's `close()` implementation and tracked sandbox cleanup, so a replacement process can immediately reclaim queue work and resume durable sessions. Give the process enough termination grace to finish those adapters' shutdown paths.
+
 ## Configure model access and route auth
 
 Set `AI_GATEWAY_API_KEY` to use a string model ID through the Vercel AI Gateway from a non-Vercel host. To call a provider directly, install its [AI SDK provider package](https://ai-sdk.dev/docs/foundations/providers-and-models). Then pass its model object in `agent.ts` and set its API key. See [Agent configuration](../../agent-config#set-the-model) for examples.

docs/reference/cli.md

@@ -220,6 +220,8 @@ Local callback-based connection authorization requires a persistent server. Run
 
 Local dev records the last ready URL per resolved app root in `.eve/dev-server-state.v1.json`. A second interactive `eve dev` reconnects only when that URL is loopback and healthy; each terminal UI creates a fresh client session while sharing the server process. A stale or malformed record is replaced when eve starts a new server. Passing `--host`, `--port`, or a `PORT` environment value skips reconnection and reports a healthy recorded server instead.
 
+When a rebuild retires a development worker, or when `eve dev` shuts down, eve gives the worker up to 300 ms to finish its Nitro close lifecycle before terminating it. That bounded grace lets a configured Workflow world's `close()` release queue work and connections without allowing an unresponsive worker to exceed the CLI's shutdown budget.
+
 Local dev keeps immutable runtime source snapshots under `.eve/dev-runtime/snapshots/` so in-flight turns hold a consistent code revision while new turns pick up rebuilds. The terminal REPL keeps its logical session across successful rebuilds, so the next turn continues the conversation on the latest generation; `/new` terminally retires that session before clearing the transcript, and the next prompt starts a fresh session with a new session-scoped sandbox on first sandbox use. After a generation is superseded, `eve dev` retains it for at least 30 minutes and also retains the five most recently superseded generations, regardless of the configured Workflow World. The active generation is never pruned. Old runtime snapshots and local sandbox templates are pruned in the background. For manual cleanup, stop `eve dev` before deleting `.eve/dev-runtime/snapshots/` or `.eve/sandbox-cache/local/templates/`. A turn that remains unfinished beyond the automatic retention window can no longer resume after its generation is pruned.
 
 When no authored `agent/instrumentation.ts` exists, local dev also records traces under `.eve/traces/`, and bounds that store by age, size, and a keep-newest floor. Configure it with `EVE_TRACES*` in `.env.local`; see [local trace retention](../guides/instrumentation#local-trace-retention) for the rules and defaults.

packages/eve/src/internal/application/compiled-artifacts.test.ts

@@ -26,6 +26,9 @@ describe("createWorkflowWorldPluginSource", () => {
     expect(source).toContain("setWorld(workflowWorld);");
     expect(source).toContain("await getWorld();");
     expect(source).toContain("await workflowWorld.start?.();");
+    expect(source).toContain("workflowWorldClosePromise ??= Promise.resolve()");
+    expect(source).toContain("await workflowWorld.close?.();");
+    expect(source).toContain('nitroApp.hooks.hook("close", closeWorkflowWorld);');
   });
 
   it("configures the vendored local World with eve's app-local data resolver", () => {

packages/eve/src/internal/application/compiled-artifacts.ts

@@ -336,7 +336,22 @@ export function createWorkflowWorldPluginSource(input: {
     "await getWorld();",
     "await workflowWorld.start?.();",
     "",
-    "export default function installWorkflowWorldPlugin() {}",
+    "let workflowWorldClosePromise;",
+    "",
+    "async function closeWorkflowWorld() {",
+    "  workflowWorldClosePromise ??= Promise.resolve()",
+    "    .then(async () => {",
+    "      await workflowWorld.close?.();",
+    "    })",
+    "    .catch((error) => {",
+    '      console.error("[eve] Failed to close Workflow world:", error);',
+    "    });",
+    "  await workflowWorldClosePromise;",
+    "}",
+    "",
+    "export default function installWorkflowWorldPlugin(nitroApp) {",
+    '  nitroApp.hooks.hook("close", closeWorkflowWorld);',
+    "}",
     "",
   ].join("\n");
 }

packages/eve/src/internal/nitro/host/create-application-nitro.scenario.test.ts

@@ -627,7 +627,7 @@ describe("application Nitro creation", () => {
     );
   });
 
-  it("includes the sandbox shutdown plugin only for production builds", async () => {
+  it("includes coordinated shutdown plugins only for production builds", async () => {
     const productionNitroStub = createNitroStub();
     const devNitroStub = createNitroStub();
     createNitroMock.mockResolvedValueOnce(productionNitroStub.nitro);
@@ -644,10 +644,16 @@ describe("application Nitro creation", () => {
     const devPlugins = createNitroMock.mock.calls[1]?.[0].plugins as string[];
 
     expect(productionPlugins).toEqual(
-      expect.arrayContaining([expect.stringContaining("sandbox-shutdown-plugin.ts")]),
+      expect.arrayContaining([
+        expect.stringContaining("sandbox-shutdown-plugin.ts"),
+        expect.stringContaining("server-shutdown-plugin.ts"),
+      ]),
     );
     expect(devPlugins).not.toEqual(
-      expect.arrayContaining([expect.stringContaining("sandbox-shutdown-plugin.ts")]),
+      expect.arrayContaining([
+        expect.stringContaining("sandbox-shutdown-plugin.ts"),
+        expect.stringContaining("server-shutdown-plugin.ts"),
+      ]),
     );
   });
 

packages/eve/src/internal/nitro/host/create-application-nitro.ts

@@ -813,6 +813,7 @@ export async function createProductionApplicationNitro(
   const nitroPlugins = createApplicationNitroPlugins(preparedHost);
   nitroPlugins.push(
     resolvePackageSourceFilePath("src/internal/nitro/host/sandbox-shutdown-plugin.ts"),
+    resolvePackageSourceFilePath("src/internal/nitro/host/server-shutdown-plugin.ts"),
   );
 
   await prepareEveVersionedCacheDirectory(options.buildDir);

packages/eve/src/internal/nitro/host/dev-runner.scenario.test.ts

@@ -0,0 +1,88 @@
+import { readFile, writeFile } from "node:fs/promises";
+import { join } from "node:path";
+
+import { describe, expect, it } from "vitest";
+
+import { createNodeDevelopmentRunner } from "#internal/nitro/host/dev-runner.js";
+import { useTemporaryDirectories } from "#internal/testing/use-temporary-app-roots.js";
+
+const createScratchDirectory = useTemporaryDirectories();
+
+describe("Node development runner shutdown", () => {
+  it("awaits worker-owned close hooks exactly once", async () => {
+    const scratchRoot = await createScratchDirectory("eve-dev-runner-close-");
+    const eventsPath = join(scratchRoot, "events.log");
+    const entryPath = join(scratchRoot, "entry.mjs");
+
+    await writeFile(
+      entryPath,
+      [
+        'import { appendFile } from "node:fs/promises";',
+        `const eventsPath = ${JSON.stringify(eventsPath)};`,
+        "export default {",
+        '  fetch() { return new Response("ok"); },',
+        "  ipc: {",
+        "    async onClose() {",
+        '      await appendFile(eventsPath, "close:start\\n");',
+        "      await new Promise((resolve) => setTimeout(resolve, 25));",
+        '      await appendFile(eventsPath, "close:end\\n");',
+        "    },",
+        "  },",
+        "};",
+        "",
+      ].join("\n"),
+      "utf8",
+    );
+
+    const runner = createNodeDevelopmentRunner({
+      entry: entryPath,
+      name: "close-probe",
+      workerData: {},
+    });
+    await runner.waitForReady(5_000);
+
+    await runner.close();
+    await runner.close();
+
+    await expect(readFile(eventsPath, "utf8")).resolves.toBe("close:start\nclose:end\n");
+  });
+
+  it("force-reaps a worker whose close hook never settles", async () => {
+    const scratchRoot = await createScratchDirectory("eve-dev-runner-stuck-close-");
+    const eventsPath = join(scratchRoot, "events.log");
+    const entryPath = join(scratchRoot, "entry.mjs");
+
+    await writeFile(
+      entryPath,
+      [
+        'import { appendFile } from "node:fs/promises";',
+        `const eventsPath = ${JSON.stringify(eventsPath)};`,
+        "export default {",
+        '  fetch() { return new Response("ok"); },',
+        "  ipc: {",
+        "    async onClose() {",
+        '      await appendFile(eventsPath, "close\\n");',
+        "      await new Promise(() => undefined);",
+        "    },",
+        "  },",
+        "};",
+        "",
+      ].join("\n"),
+      "utf8",
+    );
+
+    const runner = createNodeDevelopmentRunner({
+      entry: entryPath,
+      name: "stuck-close-probe",
+      workerData: {},
+    });
+    await runner.waitForReady(5_000);
+
+    const startedAt = Date.now();
+    await runner.close();
+    await runner.close();
+
+    expect(Date.now() - startedAt).toBeLessThan(1_000);
+    await expect(readFile(eventsPath, "utf8")).resolves.toBe("close\n");
+  });
+});

packages/eve/src/internal/nitro/host/dev-runner.ts

@@ -4,6 +4,11 @@ import { Worker } from "node:worker_threads";
 import { BaseEnvRunner } from "#compiled/env-runner/index.js";
 import { resolvePackageCompiledFilePath } from "#internal/application/package.js";
 
+// The outer dev-server process grants its child 550ms to close over IPC before
+// escalating to process signals. Keep worker cleanup inside that window while
+// still giving Nitro hooks enough time to release worker-owned resources.
+const DEVELOPMENT_WORKER_SHUTDOWN_GRACE_MS = 300;
+
 export interface DevelopmentRunner {
   readonly closed: boolean;
   close(cause?: unknown): Promise<void>;
@@ -98,8 +103,15 @@ class NodeDevelopmentRunner extends BaseEnvRunner implements DevelopmentRunner {
     }
 
     this.#worker = undefined;
+    await requestWorkerShutdown(worker);
     worker.removeAllListeners();
-    await worker.terminate();
+    const ignoreTerminationError = () => undefined;
+    worker.on("error", ignoreTerminationError);
+    try {
+      await worker.terminate();
+    } finally {
+      worker.off("error", ignoreTerminationError);
+    }
   }
 
   protected override _handleMessage(message: unknown): void {
@@ -139,6 +151,56 @@ class NodeDevelopmentRunner extends BaseEnvRunner implements DevelopmentRunner {
 export const createNodeDevelopmentRunner: DevelopmentRunnerFactory = (input) =>
   new NodeDevelopmentRunner(input);
 
+async function requestWorkerShutdown(worker: Worker): Promise<void> {
+  await new Promise<void>((resolve) => {
+    let settled = false;
+    let timeout: NodeJS.Timeout | undefined;
+
+    const settle = () => {
+      if (settled) {
+        return;
+      }
+      settled = true;
+      if (timeout !== undefined) {
+        clearTimeout(timeout);
+      }
+      worker.off("error", settle);
+      worker.off("exit", settle);
+      worker.off("message", onMessage);
+      resolve();
+    };
+    const onMessage = (message: unknown) => {
+      if (isWorkerExitMessage(message)) {
+        settle();
+      }
+    };
+
+    worker.once("error", settle);
+    worker.once("exit", settle);
+    worker.on("message", onMessage);
+    timeout = setTimeout(settle, DEVELOPMENT_WORKER_SHUTDOWN_GRACE_MS);
+    timeout.unref();
+
+    if (worker.threadId === -1) {
+      settle();
+      return;
+    }
+
+    try {
+      worker.postMessage({ event: "shutdown" });
+    } catch {
+      settle();
+    }
+  });
+}
+
+function isWorkerExitMessage(value: unknown): value is { readonly event: "exit" } {
+  if (typeof value !== "object" || value === null) {
+    return false;
+  }
+  return (value as Record<string, unknown>).event === "exit";
+}
+
 function isWorkerInitializationError(
   value: unknown,
 ): value is { readonly error: string; readonly event: "init-error" } {

packages/eve/src/internal/nitro/host/sandbox-shutdown-plugin.test.ts

@@ -5,91 +5,16 @@ import {
   trackActiveSandboxHandle,
 } from "#execution/sandbox/active-handles.js";
 import {
-  installSandboxShutdownHandlers,
+  installSandboxShutdownHook,
   runSandboxShutdown,
-  shouldInstallSandboxShutdown,
 } from "#internal/nitro/host/sandbox-shutdown-plugin.js";
 
-type SignalListener = () => void;
-
-function createFakeProcess(env: Record<string, string | undefined> = {}) {
-  const listeners = new Map<string, SignalListener>();
-  const exit = vi.fn<(code?: number) => void>();
-  return {
-    emit(event: string): void {
-      listeners.get(event)?.();
-    },
-    env,
-    exit,
-    listeners,
-    once(event: "SIGINT" | "SIGTERM", listener: SignalListener): unknown {
-      listeners.set(event, listener);
-      return this;
-    },
-  };
-}
-
 afterEach(() => {
   clearActiveSandboxHandlesForTest();
   vi.unstubAllEnvs();
 });
 
-describe("shouldInstallSandboxShutdown", () => {
-  it("installs on a plain production server", () => {
-    expect(shouldInstallSandboxShutdown({})).toBe(true);
-  });
-
-  it("skips eve dev processes", () => {
-    vi.stubEnv("EVE_DEV", "1");
-    expect(shouldInstallSandboxShutdown({})).toBe(false);
-  });
-
-  it("skips dev sandbox run workers", () => {
-    expect(shouldInstallSandboxShutdown({ EVE_DEVELOPMENT_SANDBOX_RUN_ID: "dev-run" })).toBe(false);
-  });
-
-  it("skips Vercel serverless instances", () => {
-    expect(shouldInstallSandboxShutdown({ VERCEL: "1" })).toBe(false);
-  });
-});
-
-describe("installSandboxShutdownHandlers", () => {
-  it("registers no handlers when shutdown ownership is elsewhere", () => {
-    const fakeProcess = createFakeProcess({ VERCEL: "1" });
-
-    installSandboxShutdownHandlers({ log: () => {}, process: fakeProcess });
-
-    expect(fakeProcess.listeners.size).toBe(0);
-  });
-
-  it("stops tracked sandboxes and exits 143 on SIGTERM", async () => {
-    const handle = { shutdown: vi.fn(async () => {}) };
-    trackActiveSandboxHandle({ backendName: "docker", handle, sessionKey: "session-1" });
-    const fakeProcess = createFakeProcess();
-
-    installSandboxShutdownHandlers({ log: () => {}, process: fakeProcess });
-    fakeProcess.emit("SIGTERM");
-
-    await vi.waitFor(() => {
-      expect(fakeProcess.exit).toHaveBeenCalledWith(143);
-    });
-    expect(handle.shutdown).toHaveBeenCalledTimes(1);
-  });
-
-  it("stops tracked sandboxes and exits 130 on SIGINT", async () => {
-    const handle = { shutdown: vi.fn(async () => {}) };
-    trackActiveSandboxHandle({ backendName: "docker", handle, sessionKey: "session-1" });
-    const fakeProcess = createFakeProcess();
-
-    installSandboxShutdownHandlers({ log: () => {}, process: fakeProcess });
-    fakeProcess.emit("SIGINT");
-
-    await vi.waitFor(() => {
-      expect(fakeProcess.exit).toHaveBeenCalledWith(130);
-    });
-    expect(handle.shutdown).toHaveBeenCalledTimes(1);
-  });
-
+describe("installSandboxShutdownHook", () => {
   it("stops tracked sandboxes through the nitro close hook", async () => {
     const handle = { shutdown: vi.fn(async () => {}) };
     trackActiveSandboxHandle({ backendName: "docker", handle, sessionKey: "session-1" });
@@ -105,15 +30,27 @@ describe("installSandboxShutdownHandlers", () => {
       },
     };
 
-    installSandboxShutdownHandlers({
+    installSandboxShutdownHook({
+      env: {},
       log: () => {},
       nitroApp,
-      process: createFakeProcess(),
     });
     await closeHandler?.();
 
     expect(handle.shutdown).toHaveBeenCalledTimes(1);
   });
+
+  it("registers no close hook when shutdown ownership is elsewhere", () => {
+    const hook = vi.fn();
+
+    installSandboxShutdownHook({
+      env: { VERCEL: "1" },
+      log: () => {},
+      nitroApp: { hooks: { hook } },
+    });
+
+    expect(hook).not.toHaveBeenCalled();
+  });
 });
 
 describe("runSandboxShutdown", () => {