From 20a146903853f09541184e7c774327a602f7fa49 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 10:33:53 +0000 Subject: [PATCH 1/3] Refuse a second opener of the embedded database and report a damaged one clearly PGlite has no lock, so a second process could open the same folder and damage it, and a damaged folder ended the whole process with an uncaught Aborted(). Take a lock file beside the folder, turn the abort into an actionable error, and have the workers print the reason once. Document where the database lives per start path and how to recover. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Lu73uozkZGX8gP8LWNbBqt --- CHANGELOG.md | 2 + docs/contributing/development.md | 13 +++ docs/operations/troubleshooting.md | 3 + package.json | 2 +- src/server/adapters/db/config.ts | 3 + src/server/adapters/db/orm.ts | 20 +++-- .../adapters/db/pglite-guard.db.test.ts | 78 ++++++++++++++++++ src/server/adapters/db/pglite-guard.ts | 79 +++++++++++++++++++ src/server/workers/command-loop.ts | 3 +- src/server/workers/decision-loop.ts | 5 +- src/server/workers/freshness-loop.ts | 3 +- src/server/workers/notification-loop.ts | 5 +- src/server/workers/push-loop.ts | 5 +- src/server/workers/reconciliation-loop.ts | 3 +- src/server/workers/report.ts | 15 ++++ src/server/workers/subscription-loop.ts | 3 +- 16 files changed, 223 insertions(+), 19 deletions(-) create mode 100644 src/server/adapters/db/pglite-guard.db.test.ts create mode 100644 src/server/adapters/db/pglite-guard.ts create mode 100644 src/server/workers/report.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ee39692..8175d7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,8 @@ the project uses [semantic versioning](https://semver.org/) once it reaches 1.0; - **Dev server serves `127.0.0.1`.** `next dev` now allows the `127.0.0.1` origin, which the launcher and docs print; before, the page loaded there but never became interactive. - **Browsing an agent's tasks is a page, not a dialog.** **Find tasks** (`/discover`) works across agents and is linked from the sidebar, the **Tasks** list and each agent's **Task discovery** card. +- **Clear errors when the embedded database cannot be opened.** PGlite has no lock of its own, so a second process (for example `npm run db:migrate` while `npm run dev` runs) could open the same folder silently and damage it. Taskbay now takes a `.lock` file and refuses with "already open in another process"; a stale lock from a dead process is taken over. A damaged folder, which made PGlite end the whole process with `RuntimeError: Aborted()`, now fails the open with an error that says what to do. Background workers print the reason once instead of a bare "will retry" in a loop. Nothing is deleted or moved automatically. + ### Added - **Browse and track tasks started elsewhere.** **Browse tasks** on an agent page reads live `ListTasks` pages without storing them; **Track** and **Track by task ID** re-read and store one task to answer, steer, cancel or claim. Requires an administrator or whole-agent `operate` grant. Stored tasks carry **Discovered** labels, an origin filter and tracking details; tracking never claims a task or notifies people about its first stored state. The **Task discovery** card defaults to **Browse only**, with **Off** and a deployment-wide `A2A_TASK_DISCOVERY` cap. Schema version 2. diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 5507a78..fa966af 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -19,6 +19,19 @@ npm run dev The console is at `http://localhost:3002`. `npm run dev` uses embedded PGlite under `.data/`, applies migrations on start (`A2A_AUTO_MIGRATE`), and signs you in as a labelled development administrator. Copy `.env.example` to `.env.local` to change settings; see the [configuration reference](../reference/configuration.md). +### Where the database is + +| How you start it | Database | Migrations | Sign-in | +| --- | --- | --- | --- | +| `npx agent-taskbay` (or `node bin/agent-taskbay.mjs` after `npm run build`) | PGlite in `~/.agent-taskbay/pglite` (`--data-dir` or `A2A_DATA_DIR` to change), with generated secrets and artifacts beside it | Applied on start, after a backup of an existing database | Local development administrator, loopback only | +| `npm run dev` | PGlite in `.data/pglite` under the repository (`A2A_PGLITE_DATA_DIR` to change) | Applied on start | Local development administrator | +| `npm run db:migrate`, `npm run auth:provision` and the other scripts | The same folder `npm run dev` uses, unless you set `A2A_*` variables | `db:migrate` applies them | Not applicable | +| PostgreSQL (`A2A_DATABASE_PROFILE=postgresql`, `A2A_DATABASE_URL`) | Your server | Explicit `npm run db:migrate`, unless `A2A_AUTO_MIGRATE=true` | OIDC | + +PGlite allows one owner. Taskbay now takes a lock file beside the folder (`.lock`), so a second process, such as `npm run db:migrate` while `npm run dev` is running, stops with "already open in another process" instead of sharing the folder. Stop the dev server before running a script against its database, or point the script at a different folder. + +If the folder is damaged (typically by a process killed while writing, or by two processes opening it before the lock existed), opening it fails with "could not be opened (PGlite aborted)". Taskbay does not delete or move it for you, because it holds your data. Move the folder aside (`mv .data/pglite .data/pglite.damaged`), start again to get a fresh one, and keep the old one if you need to recover anything. Stop the dev server with Ctrl+C rather than killing it. + To give it agents, start the sample ones in a second terminal and register a printed Agent Card URL in the console: ```bash diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.md index cbc3040..57e1171 100644 --- a/docs/operations/troubleshooting.md +++ b/docs/operations/troubleshooting.md @@ -21,6 +21,9 @@ API errors have the shape `{"error":{"message":"..."}}`, sometimes with a `name` | `agent-taskbay: Local mode signs everyone in as an administrator, so it cannot listen on . Use 127.0.0.1, or configure OIDC (A2A_AUTH_MODE=oidc and the A2A_OIDC_* settings).` | Launcher | `--host` is not loopback and OIDC is not configured. | Bind loopback or configure [OIDC](../guides/sign-in-and-roles.md). | | `agent-taskbay: /secrets.json is not valid. Fix or remove it. Removing it makes stored agent credentials unreadable.` | Launcher | Corrupt generated key file. The launcher never overwrites it. | Restore it from backup. Remove it only if you accept re-entering every credential. | | `agent-taskbay: --port needs a value.`, `"x" is not a valid port.`, `Unknown option --x.`, `Unexpected argument x.` | Launcher | Bad flags. | See `agent-taskbay --help`. | +| `The embedded database in is already open in another process (pid N).` | Web, launcher, scripts | PGlite allows one owner and a live process holds the lock file `.lock`. | Stop that process (the dev server, the launcher, a migration), then retry. A lock left by a process that died is taken over automatically. | +| `The embedded database in could not be opened (PGlite aborted).` | Web, launcher, scripts | The PGlite folder is damaged, usually by a process killed mid-write or two processes opening it. | Stop everything that uses it, move the folder aside, start again for a fresh one, and keep the old folder if you need to recover data. See [Development setup](../contributing/development.md#where-the-database-is). | +| `… could not access durable state; will retry. ` | Web, workers | A background worker cannot reach the database. The reason is printed once, and again only if it changes. | Fix the reason shown (for example the two rows above). | | `External command workers require PostgreSQL.` | Web, at start | `A2A_COMMAND_WORKER_MODE=external` with PGlite. | Use PostgreSQL, or leave the mode `embedded`. | | `Invalid A2A_COMMAND_WORKER_MODE.` | Web, at start | Value is neither `embedded` nor `external`. | Fix it. | | `The separate worker requires PostgreSQL; PGlite uses embedded dispatch.` | Worker | Worker started with `A2A_DATABASE_PROFILE` not `postgresql`. | Set the profile and URL for the worker's environment. | diff --git a/package.json b/package.json index e47b0c5..683b697 100644 --- a/package.json +++ b/package.json @@ -59,7 +59,7 @@ "start": "next start -p 3002", "lint": "eslint", "test": "vitest run --exclude 'src/server/adapters/db/**/*.db.test.ts'", - "test:db": "vitest run src/server/adapters/db/database.db.test.ts src/server/adapters/db/migration-startup.db.test.ts src/server/adapters/db/repositories.db.test.ts src/server/adapters/db/agent-registry.db.test.ts src/server/adapters/db/task-observation.db.test.ts src/server/adapters/db/commands.db.test.ts src/server/adapters/db/subscriptions.db.test.ts src/server/adapters/db/push.db.test.ts src/server/adapters/db/push-cleanup.db.test.ts src/server/adapters/db/reconciliation.db.test.ts src/server/adapters/db/projection-rebuild.db.test.ts src/server/adapters/db/freshness.db.test.ts src/server/adapters/db/identity.db.test.ts src/server/adapters/db/scoped-security.db.test.ts src/server/adapters/db/skill-less-tasks.db.test.ts src/server/adapters/db/auto-discovery.db.test.ts src/server/adapters/db/agent-settings.db.test.ts src/server/adapters/db/discovered-notifications.db.test.ts src/server/adapters/db/discovery.db.test.ts src/server/adapters/db/required-extensions.db.test.ts src/server/adapters/db/terminal-guard.db.test.ts src/server/adapters/db/discovery-model.db.test.ts src/server/adapters/db/decisions.db.test.ts src/server/adapters/db/workflow.db.test.ts src/server/adapters/db/audit.db.test.ts src/server/adapters/db/notifications.db.test.ts src/server/adapters/db/inbox.db.test.ts src/server/adapters/db/phase6-exit.db.test.ts src/server/adapters/db/task-history.db.test.ts src/server/adapters/db/task-list.db.test.ts src/server/adapters/db/steering.db.test.ts src/server/adapters/db/extended-card.db.test.ts src/server/adapters/db/task-links.db.test.ts src/server/adapters/db/people.db.test.ts --maxWorkers=1 --testTimeout=30000", + "test:db": "vitest run src/server/adapters/db/database.db.test.ts src/server/adapters/db/migration-startup.db.test.ts src/server/adapters/db/repositories.db.test.ts src/server/adapters/db/agent-registry.db.test.ts src/server/adapters/db/task-observation.db.test.ts src/server/adapters/db/commands.db.test.ts src/server/adapters/db/subscriptions.db.test.ts src/server/adapters/db/push.db.test.ts src/server/adapters/db/push-cleanup.db.test.ts src/server/adapters/db/reconciliation.db.test.ts src/server/adapters/db/projection-rebuild.db.test.ts src/server/adapters/db/freshness.db.test.ts src/server/adapters/db/identity.db.test.ts src/server/adapters/db/scoped-security.db.test.ts src/server/adapters/db/skill-less-tasks.db.test.ts src/server/adapters/db/auto-discovery.db.test.ts src/server/adapters/db/agent-settings.db.test.ts src/server/adapters/db/discovered-notifications.db.test.ts src/server/adapters/db/discovery.db.test.ts src/server/adapters/db/required-extensions.db.test.ts src/server/adapters/db/terminal-guard.db.test.ts src/server/adapters/db/discovery-model.db.test.ts src/server/adapters/db/decisions.db.test.ts src/server/adapters/db/workflow.db.test.ts src/server/adapters/db/audit.db.test.ts src/server/adapters/db/notifications.db.test.ts src/server/adapters/db/inbox.db.test.ts src/server/adapters/db/phase6-exit.db.test.ts src/server/adapters/db/task-history.db.test.ts src/server/adapters/db/task-list.db.test.ts src/server/adapters/db/steering.db.test.ts src/server/adapters/db/extended-card.db.test.ts src/server/adapters/db/task-links.db.test.ts src/server/adapters/db/people.db.test.ts src/server/adapters/db/pglite-guard.db.test.ts --maxWorkers=1 --testTimeout=30000", "test:http": "node scripts/verify-service-security.mjs && node scripts/verify-task-http.mjs && node --import tsx scripts/verify-identity-http.mjs && node scripts/verify-decisions-http.mjs && node --import tsx scripts/verify-workflow-http.mjs && node --import tsx scripts/verify-notifications-http.mjs && node scripts/verify-forms-http.mjs && node scripts/verify-agui-http.mjs && node scripts/verify-a2ui-http.mjs && node scripts/verify-agent-approvals-http.mjs && node scripts/verify-phase6-exit-http.mjs && node scripts/verify-discovery-http.mjs && node scripts/verify-history-http.mjs && node --import tsx scripts/verify-steering-http.mjs && node scripts/verify-extended-card-http.mjs", "test:coverage": "vitest run --coverage --exclude 'src/server/adapters/db/**/*.db.test.ts'", "db:migrate": "mikro-orm migration:up", diff --git a/src/server/adapters/db/config.ts b/src/server/adapters/db/config.ts index 5d86cd8..99fa09c 100644 --- a/src/server/adapters/db/config.ts +++ b/src/server/adapters/db/config.ts @@ -8,6 +8,7 @@ import { PostgreSqlDriver } from "@mikro-orm/postgresql"; import { z } from "zod"; import { persistenceEntities } from "./entities"; +import { claimPgliteDirectory } from "./pglite-guard"; import { nextMigrationName } from "./schema-version"; const databaseEnvironmentSchema = z @@ -122,6 +123,8 @@ export function createMikroOrmOptions( // PGlite creates its directory without `recursive`, so a clean checkout (no `.data`) would fail with ENOENT. mkdirSync(path.dirname(config.dataDir), { recursive: true }); + // PGlite has no lock of its own; this also covers the MikroORM CLI (`npm run db:migrate`), which builds its options here. + claimPgliteDirectory(config.dataDir); return { ...commonOptions, diff --git a/src/server/adapters/db/orm.ts b/src/server/adapters/db/orm.ts index b55dc58..a6c660c 100644 --- a/src/server/adapters/db/orm.ts +++ b/src/server/adapters/db/orm.ts @@ -11,6 +11,7 @@ import { type DatabaseConfig, } from "./config"; import { migrateOnStartup } from "./migration-startup"; +import { openGuarded } from "./pglite-guard"; import { createPersistenceRepositories } from "./repositories"; type DatabaseGlobal = typeof globalThis & { @@ -22,14 +23,17 @@ const databaseGlobal = globalThis as DatabaseGlobal; export async function createDatabaseOrm( config: DatabaseConfig = loadDatabaseConfig(), ): Promise { - const orm = await MikroORM.init(createMikroOrmOptions(config)); - try { - await orm.connect(); - return orm; - } catch (error) { - await orm.close(true).catch(() => undefined); - throw error; - } + const open = async () => { + const orm = await MikroORM.init(createMikroOrmOptions(config)); + try { + await orm.connect(); + return orm; + } catch (error) { + await orm.close(true).catch(() => undefined); + throw error; + } + }; + return config.profile === "pglite" ? openGuarded(config.dataDir, open) : open(); } export async function bootstrapRuntimeDatabase(orm: MikroORM) { diff --git a/src/server/adapters/db/pglite-guard.db.test.ts b/src/server/adapters/db/pglite-guard.db.test.ts new file mode 100644 index 0000000..6b62f7b --- /dev/null +++ b/src/server/adapters/db/pglite-guard.db.test.ts @@ -0,0 +1,78 @@ +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { claimPgliteDirectory, DatabaseOpenError, lockPathFor } from "./pglite-guard"; + +function scratch() { + const dir = mkdtempSync(join(tmpdir(), "a2a-guard-")); + return { dir, dataDir: join(dir, "db"), done: () => rmSync(dir, { recursive: true, force: true }) }; +} + +/** Runs one script in a separate process; the open that aborts would end this test process. */ +function run(code: string, dataDir: string) { + return spawnSync(process.execPath, ["--import", "tsx", "--input-type=module", "-e", code], { + cwd: process.cwd(), env: { ...process.env, A2A_PGLITE_DATA_DIR: dataDir, A2A_DATABASE_PROFILE: "pglite" }, encoding: "utf8", timeout: 90_000, + }); +} + +describe("PGlite directory lock", () => { + it("refuses a live owner, takes over a dead one's lock and ignores an unreadable one", () => { + const { dataDir, dir, done } = scratch(); + try { + mkdirSync(dir, { recursive: true }); + writeFileSync(lockPathFor(dataDir), JSON.stringify({ pid: process.ppid })); + expect(() => claimPgliteDirectory(dataDir)).toThrow(DatabaseOpenError); + expect(() => claimPgliteDirectory(dataDir)).toThrow(/already open in another process/); + writeFileSync(lockPathFor(dataDir), JSON.stringify({ pid: 2 ** 22 + 7 })); + claimPgliteDirectory(dataDir); + claimPgliteDirectory(dataDir); // the same process may claim again + const other = scratch(); + writeFileSync(lockPathFor(other.dataDir), "not json"); + claimPgliteDirectory(other.dataDir); + other.done(); + } finally { done(); } + }); +}); + +describe("PGlite open failures", () => { + it("a second process is refused with a clear message instead of sharing the folder", () => { + const { dataDir, done } = scratch(); + try { + const script = ` + import { createDatabaseOrm } from "./src/server/adapters/db/orm"; + const a = await createDatabaseOrm(); + const { spawnSync } = await import("node:child_process"); + const child = spawnSync(process.execPath, ["--import", "tsx", "--input-type=module", "-e", + 'import { createDatabaseOrm } from "./src/server/adapters/db/orm"; try { await createDatabaseOrm(); console.log("OPENED"); } catch (e) { console.log(e.name + ": " + e.message); } process.exit(0);'], + { encoding: "utf8", env: process.env }); + console.log(child.stdout); + await a.close(true); process.exit(0);`; + const result = run(script, dataDir); + expect(result.stdout).toContain("DatabaseOpenError: The embedded database"); + expect(result.stdout).toContain("already open in another process"); + expect(result.stdout).not.toContain("OPENED"); + } finally { done(); } + }, 120_000); + + it("an Aborted() thrown from inside PGlite becomes a clear error, not a crash of the whole process", () => { + const { dataDir, done } = scratch(); + try { + // PGlite's WebAssembly throws this from a callback, so it is uncaught rather than a rejection of the open. + const result = run(` + import { openGuarded } from "./src/server/adapters/db/pglite-guard"; + const dir = process.env.A2A_PGLITE_DATA_DIR; + try { await openGuarded(dir, () => new Promise(() => setTimeout(() => { throw new Error("Aborted(). Build with -sASSERTIONS for more info."); }, 20))); console.log("OPENED"); } + catch (e) { console.log(e.name + ": " + e.message); } + try { await openGuarded(dir, async () => { setTimeout(() => { throw new Error("something else"); }, 20); await new Promise((r) => setTimeout(r, 200)); }); console.log("SWALLOWED"); } + catch (e) { console.log("unexpected " + e.message); }`, dataDir); + expect(result.stdout).toContain("DatabaseOpenError: The embedded database"); + expect(result.stdout).toContain("could not be opened"); + expect(result.stdout).toContain("move the folder aside"); + // An unrelated uncaught error must still end the process; the guard only claims the abort. + expect(result.stdout).not.toContain("SWALLOWED"); + expect(result.status).not.toBe(0); + } finally { done(); } + }, 120_000); +}); diff --git a/src/server/adapters/db/pglite-guard.ts b/src/server/adapters/db/pglite-guard.ts new file mode 100644 index 0000000..b79ecc6 --- /dev/null +++ b/src/server/adapters/db/pglite-guard.ts @@ -0,0 +1,79 @@ +import { readFileSync, rmSync, writeFileSync } from "node:fs"; + +/** + * A refusal to open the embedded database, with text a person can act on. It is the only database error the workers + * and the console repeat; anything else is reported by its name alone, because driver messages can carry connection details. + */ +export class DatabaseOpenError extends Error { + constructor(message: string) { + super(message); + this.name = "DatabaseOpenError"; + } +} + +const held = new Set(); + +function alive(pid: number): boolean { + try { process.kill(pid, 0); return true; } catch (error) { return (error as NodeJS.ErrnoException).code === "EPERM"; } +} + +export const lockPathFor = (dataDir: string) => `${dataDir}.lock`; + +/** + * PGlite is a single-owner database with no lock of its own: a second process that opens the same folder succeeds, and + * both then write to it, which damages it. This takes a lock file beside the folder (never inside it) and refuses while + * another live process holds it. A lock left by a process that died is taken over. Held until this process exits. + */ +export function claimPgliteDirectory(dataDir: string): void { + if (held.has(dataDir)) return; + const file = lockPathFor(dataDir); + for (let attempt = 0; attempt < 2; attempt++) { + try { + writeFileSync(file, `${JSON.stringify({ pid: process.pid, startedAt: new Date().toISOString() })}\n`, { flag: "wx" }); + held.add(dataDir); + process.once("exit", () => { try { rmSync(file); } catch { /* already gone */ } }); + return; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") return; // an unwritable location is reported by the database itself + let owner: number | undefined; + try { owner = (JSON.parse(readFileSync(file, "utf8")) as { pid?: number }).pid; } catch { /* unreadable: treat as stale */ } + if (owner === process.pid) { held.add(dataDir); return; } + if (owner !== undefined && alive(owner)) { + throw new DatabaseOpenError( + `The embedded database in ${dataDir} is already open in another process (pid ${owner}). ` + + "Stop that process first (for example the dev server, the launcher, or a migration), then try again. " + + "Opening it twice damages it."); + } + rmSync(file, { force: true }); // stale: its owner is gone + } + } +} + +/** + * PGlite reports an unreadable folder by throwing `RuntimeError: Aborted()` from inside its WebAssembly, which escapes as an + * uncaught exception and ends the whole process instead of rejecting the open. Catch that for the duration of the open and + * turn it into an error that says what happened and what to do. + */ +export async function openGuarded(dataDir: string, open: () => Promise): Promise { + let fail!: (error: unknown) => void; + const aborted = new Promise((_, reject) => { fail = reject; }); + const onUncaught = (error: unknown) => { + if (error instanceof Error && /Aborted\(\)/.test(error.message)) { + fail(new DatabaseOpenError( + `The embedded database in ${dataDir} could not be opened (PGlite aborted). It is usually damaged by a process killed ` + + "while writing, or opened by two processes at once. Stop everything that uses it. If it still fails, move the folder " + + "aside (it holds your data) and start again to get a fresh one.")); + } else { + process.removeListener("uncaughtException", onUncaught); + throw error; + } + }; + process.on("uncaughtException", onUncaught); + try { + const opening = open(); + opening.catch(() => undefined); // if the abort wins the race, a later rejection is not an unhandled one + return await Promise.race([opening, aborted]); + } finally { + process.removeListener("uncaughtException", onUncaught); + } +} diff --git a/src/server/workers/command-loop.ts b/src/server/workers/command-loop.ts index 8bbea97..c432369 100644 --- a/src/server/workers/command-loop.ts +++ b/src/server/workers/command-loop.ts @@ -1,4 +1,5 @@ import { createCommandDispatcher } from "../runtime/commands"; +import { reportRetry } from "./report"; /** Loop is shared by embedded local dispatch and the PostgreSQL worker. */ export function startCommandLoop() { @@ -9,7 +10,7 @@ export function startCommandLoop() { const tick = async () => { try { for (let count = 0; count < 10 && !stopped; count++) if (!await dispatcher.runOne()) break; - } catch { console.error("Command dispatcher could not access durable state; will retry."); } + } catch (error) { reportRetry("Command dispatcher", error); } if (!stopped) { timer = setTimeout(() => { running = tick(); }, 250); timer.unref?.(); } }; running = tick(); diff --git a/src/server/workers/decision-loop.ts b/src/server/workers/decision-loop.ts index d5aed24..82b56e7 100644 --- a/src/server/workers/decision-loop.ts +++ b/src/server/workers/decision-loop.ts @@ -1,5 +1,6 @@ import { createDecisionService } from "../runtime/decisions"; import { createWorkflowService } from "../runtime/workflow"; +import { reportRetry } from "./report"; const interval = () => { const configured = Number(process.env.A2A_DECISION_SWEEP_MS); @@ -17,8 +18,8 @@ export function startDecisionLoop() { let timer: ReturnType | undefined; let running: Promise = Promise.resolve(); const tick = async () => { - try { await service.sweep(); } catch { console.error("Decision sweeper could not access durable state; will retry."); } - try { await workflow.sweepEscalations(); } catch { console.error("Escalation sweeper could not access durable state; will retry."); } + try { await service.sweep(); } catch (error) { reportRetry("Decision sweeper", error); } + try { await workflow.sweepEscalations(); } catch (error) { reportRetry("Escalation sweeper", error); } if (!stopped) { timer = setTimeout(() => { running = tick(); }, interval()); timer.unref?.(); } }; running = tick(); diff --git a/src/server/workers/freshness-loop.ts b/src/server/workers/freshness-loop.ts index 93850b6..97ec727 100644 --- a/src/server/workers/freshness-loop.ts +++ b/src/server/workers/freshness-loop.ts @@ -1,4 +1,5 @@ import { createFreshnessDispatcher } from "../runtime/freshness"; +import { reportRetry } from "./report"; export function startFreshnessLoop() { const dispatcher = createFreshnessDispatcher(); @@ -8,7 +9,7 @@ export function startFreshnessLoop() { const tick = async () => { try { for (let count = 0; count < 100 && !stopped; count++) if (!await dispatcher.runOne()) break; - } catch { console.error("Freshness dispatcher could not access durable state; will retry."); } + } catch (error) { reportRetry("Freshness dispatcher", error); } if (!stopped) { timer = setTimeout(() => { running = tick(); }, 250); timer.unref?.(); } }; running = tick(); diff --git a/src/server/workers/notification-loop.ts b/src/server/workers/notification-loop.ts index 1e16c17..d9135ff 100644 --- a/src/server/workers/notification-loop.ts +++ b/src/server/workers/notification-loop.ts @@ -1,4 +1,5 @@ import { createNotificationDelivery, createNotificationFanout } from "../runtime/notifications"; +import { reportRetry } from "./report"; /** Turns queued events into inbox rows, then delivers each through the external channel when one is configured. */ export function startNotificationLoop() { @@ -10,9 +11,9 @@ export function startNotificationLoop() { let running: Promise = Promise.resolve(); const tick = async () => { try { for (let count = 0; count < 100 && !shutdown.signal.aborted; count++) if (!await fanout.runOne()) break; } - catch { console.error("Notification fan-out could not access durable state; will retry."); } + catch (error) { reportRetry("Notification fan-out", error); } try { for (let count = 0; count < 20 && delivery && !shutdown.signal.aborted; count++) if (!await delivery.runOne(shutdown.signal)) break; } - catch { console.error("Notification delivery could not access durable state; will retry."); } + catch (error) { reportRetry("Notification delivery", error); } if (!shutdown.signal.aborted) { timer = setTimeout(() => { running = tick(); }, 250); timer.unref?.(); } }; running = tick(); diff --git a/src/server/workers/push-loop.ts b/src/server/workers/push-loop.ts index 0c904a0..29dedea 100644 --- a/src/server/workers/push-loop.ts +++ b/src/server/workers/push-loop.ts @@ -1,6 +1,7 @@ import { randomUUID } from "node:crypto"; import { createPushWorker } from "../runtime/push"; import { loadPushCredentials } from "../runtime/push-config"; +import { reportRetry } from "./report"; export function startPushLoop() { const credentials = loadPushCredentials(); @@ -15,13 +16,13 @@ export function startPushLoop() { try { if (!adopted) { await worker.adopt(); adopted = true; } worked = await worker.runOne(randomUUID(), shutdown.signal); - } catch { console.error("Push worker could not access durable state; will retry."); } + } catch (error) { reportRetry("Push worker", error); } if (!worked && Date.now() >= nextCleanup) { nextCleanup = Date.now() + 60_000; try { const cleaned = await worker.cleanupOne(shutdown.signal); if (cleaned) console.log(`Push cleanup removed ${cleaned} leftover configuration${cleaned === 1 ? "" : "s"}.`); - } catch { console.error("Push cleanup could not access durable state; will retry."); } + } catch (error) { reportRetry("Push cleanup", error); } } if (!worked) await new Promise((resolve) => { const done = () => { clearTimeout(timer); shutdown.signal.removeEventListener("abort", done); resolve(); }; diff --git a/src/server/workers/reconciliation-loop.ts b/src/server/workers/reconciliation-loop.ts index 0114009..bcdc25c 100644 --- a/src/server/workers/reconciliation-loop.ts +++ b/src/server/workers/reconciliation-loop.ts @@ -1,6 +1,7 @@ import { randomUUID } from "node:crypto"; import { createReconciliationWorker } from "../runtime/reconciliation"; import { ensureAutomaticDiscoveryCursors } from "../runtime/discovery-cursors"; +import { reportRetry } from "./report"; export function startReconciliationLoop(concurrency = 2) { const worker = createReconciliationWorker(); @@ -11,7 +12,7 @@ export function startReconciliationLoop(concurrency = 2) { while (!shutdown.signal.aborted) { let worked = false; try { worked = await worker.runOne(randomUUID(), shutdown.signal); } - catch { console.error("Reconciliation worker could not access durable state; will retry."); } + catch (error) { reportRetry("Reconciliation worker", error); } if (!worked) await new Promise((resolve) => { const done = () => { clearTimeout(timer); shutdown.signal.removeEventListener("abort", done); resolve(); }; const timer = setTimeout(done, 250); timer.unref?.(); diff --git a/src/server/workers/report.ts b/src/server/workers/report.ts new file mode 100644 index 0000000..5144150 --- /dev/null +++ b/src/server/workers/report.ts @@ -0,0 +1,15 @@ +import { DatabaseOpenError } from "../adapters/db/pglite-guard"; + +const lastCause = new Map(); + +/** + * A worker could not reach the database and will try again. Say why, but only when the reason changes, so a database that + * stays down is one explanation, not one line per tick. Only our own refusals are quoted; other errors are named, never + * quoted, because a driver message can carry connection details. + */ +export function reportRetry(label: string, error: unknown): void { + const cause = error instanceof DatabaseOpenError ? error.message : error instanceof Error ? error.name : "Error"; + if (lastCause.get(label) === cause) return; + lastCause.set(label, cause); + console.error(`${label} could not access durable state; will retry. ${cause}`); +} diff --git a/src/server/workers/subscription-loop.ts b/src/server/workers/subscription-loop.ts index 2c11721..2445f10 100644 --- a/src/server/workers/subscription-loop.ts +++ b/src/server/workers/subscription-loop.ts @@ -1,5 +1,6 @@ import { randomUUID } from "node:crypto"; import { createSubscriptionWorker } from "../runtime/subscriptions"; +import { reportRetry } from "./report"; /** A bounded pool prevents a quiet long-lived stream from blocking other tasks. */ export function startSubscriptionLoop(concurrency = 8) { @@ -9,7 +10,7 @@ export function startSubscriptionLoop(concurrency = 8) { while (!shutdown.signal.aborted) { let worked = false; try { worked = await worker.runOne(randomUUID(), shutdown.signal); } - catch { console.error("Subscription worker could not access durable state; will retry."); } + catch (error) { reportRetry("Subscription worker", error); } if (!worked) await new Promise((resolve) => { const done = () => { clearTimeout(timer); shutdown.signal.removeEventListener("abort", done); resolve(); }; const timer = setTimeout(done, 250); timer.unref?.(); From 528a0545ab395a1ee7a6ebbf997639d6ca4503a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 10:35:28 +0000 Subject: [PATCH 2/3] Tell people how to clear a lock whose process number was reused Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Lu73uozkZGX8gP8LWNbBqt --- docs/operations/troubleshooting.md | 2 +- src/server/adapters/db/pglite-guard.ts | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.md index 57e1171..91984fa 100644 --- a/docs/operations/troubleshooting.md +++ b/docs/operations/troubleshooting.md @@ -21,7 +21,7 @@ API errors have the shape `{"error":{"message":"..."}}`, sometimes with a `name` | `agent-taskbay: Local mode signs everyone in as an administrator, so it cannot listen on . Use 127.0.0.1, or configure OIDC (A2A_AUTH_MODE=oidc and the A2A_OIDC_* settings).` | Launcher | `--host` is not loopback and OIDC is not configured. | Bind loopback or configure [OIDC](../guides/sign-in-and-roles.md). | | `agent-taskbay: /secrets.json is not valid. Fix or remove it. Removing it makes stored agent credentials unreadable.` | Launcher | Corrupt generated key file. The launcher never overwrites it. | Restore it from backup. Remove it only if you accept re-entering every credential. | | `agent-taskbay: --port needs a value.`, `"x" is not a valid port.`, `Unknown option --x.`, `Unexpected argument x.` | Launcher | Bad flags. | See `agent-taskbay --help`. | -| `The embedded database in is already open in another process (pid N).` | Web, launcher, scripts | PGlite allows one owner and a live process holds the lock file `.lock`. | Stop that process (the dev server, the launcher, a migration), then retry. A lock left by a process that died is taken over automatically. | +| `The embedded database in is already open in another process (pid N).` | Web, launcher, scripts | PGlite allows one owner and a live process holds the lock file `.lock`. | Stop that process (the dev server, the launcher, a migration), then retry. A lock left by a process that died is taken over automatically. If nothing of yours is running, the number was reused by another process (more likely on Windows and in containers): delete the `.lock` file the message names. | | `The embedded database in could not be opened (PGlite aborted).` | Web, launcher, scripts | The PGlite folder is damaged, usually by a process killed mid-write or two processes opening it. | Stop everything that uses it, move the folder aside, start again for a fresh one, and keep the old folder if you need to recover data. See [Development setup](../contributing/development.md#where-the-database-is). | | `… could not access durable state; will retry. ` | Web, workers | A background worker cannot reach the database. The reason is printed once, and again only if it changes. | Fix the reason shown (for example the two rows above). | | `External command workers require PostgreSQL.` | Web, at start | `A2A_COMMAND_WORKER_MODE=external` with PGlite. | Use PostgreSQL, or leave the mode `embedded`. | diff --git a/src/server/adapters/db/pglite-guard.ts b/src/server/adapters/db/pglite-guard.ts index b79ecc6..1f9b588 100644 --- a/src/server/adapters/db/pglite-guard.ts +++ b/src/server/adapters/db/pglite-guard.ts @@ -42,7 +42,8 @@ export function claimPgliteDirectory(dataDir: string): void { throw new DatabaseOpenError( `The embedded database in ${dataDir} is already open in another process (pid ${owner}). ` + "Stop that process first (for example the dev server, the launcher, or a migration), then try again. " + - "Opening it twice damages it."); + "Opening it twice damages it. If nothing of yours is running (an operating system can give a dead process's number " + + `to an unrelated one), delete ${file} and try again.`); } rmSync(file, { force: true }); // stale: its owner is gone } From b3484a5de1fb8dbf75dc241f8b1993720da1de3e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 10:36:34 +0000 Subject: [PATCH 3/3] Add an informational macOS and Windows CI job Runs lint, unit tests, the embedded-database lock and people tests, and the build. continue-on-error so a platform-specific failure is visible without blocking merges. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Lu73uozkZGX8gP8LWNbBqt --- .github/workflows/ci.yml | 27 +++++++++++++++++++++++++++ docs/contributing/development.md | 2 ++ 2 files changed, 29 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 95f1f26..e77f5a3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -52,3 +52,30 @@ jobs: cache: npm - run: npm ci - run: npm run verify:package + + platforms: + # Informational: the console runs locally on macOS and Windows through `npx agent-taskbay`, and CI otherwise only covers Linux. + # `continue-on-error` keeps a platform-specific failure visible without blocking a merge; remove it once this job has + # been green for a while. It runs what needs no PostgreSQL: lint, unit tests, the embedded-database lock and people + # tests, and the production build. The PostgreSQL, HTTP and package checks stay on Linux in `verify` and `package`. + name: platforms (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + continue-on-error: true + strategy: + fail-fast: false + matrix: + os: [macos-latest, windows-latest] + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-node@v7 + with: + node-version: 22.19.0 + cache: npm + - run: npm ci + - run: npm run lint + - run: npx vitest run --exclude "src/server/adapters/db/**/*.db.test.ts" + - run: npx vitest run src/server/adapters/db/pglite-guard.db.test.ts src/server/adapters/db/people.db.test.ts --maxWorkers=1 --testTimeout=60000 + - run: npm run build diff --git a/docs/contributing/development.md b/docs/contributing/development.md index fa966af..41a95eb 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -79,6 +79,8 @@ export A2A_TEST_POSTGRES_URL=postgresql://postgres:postgres@127.0.0.1:55432/post The test account must be able to create and drop temporary databases. Use a disposable server, never a real one. +The `verify` and `package` jobs run on Linux. An informational `platforms` job also runs lint, the unit tests, the embedded-database lock and people tests, and the production build on macOS and Windows; it is allowed to fail without blocking a merge until it has proved stable. + ### Test and verification variables These are read by the repository's tests and verification scripts, not by the application, so they are not in the [configuration reference](../reference/configuration.md).