feat: storage abstraction, PostgreSQL backend, and memo mirror contract (#69)

.changeset/storage-abstraction-postgres.md

@@ -0,0 +1,57 @@
+---
+"@agentrail/core": minor
+"@agentrail/capabilities": minor
+"@agentrail/app": minor
+"@agentrail/storage-postgres": minor
+---
+
+**Storage Abstraction + Memo FS + PostgreSQL reference implementation (#69)**
+
+This release completes the Pre-GA storage contract consolidation and ships the `@agentrail/storage-postgres` reference package.
+
+### `@agentrail/core` — breaking shape changes + new optional methods
+
+- `MemoryIndex` no longer contains `sessionDir` or `userDir` host-absolute paths. The `entries[].path` field now stores the canonical `/workspace/memo/…` path directly.
+- `AgentrailSessionStore` gains five new **optional** methods for memo-document and tool-result-artifact persistence:
+  - `readMemoryDocument(tenantId, ownerId, scope, name)`
+  - `writeMemoryDocument(tenantId, ownerId, scope, name, content)`
+  - `appendMemoryDocument(tenantId, ownerId, scope, name, content)`
+  - `readToolResultArtifact(sessionRef, toolCallId)`
+  - `writeToolResultArtifact(sessionRef, toolCallId, content)`
+- New types exported: `MemoDocumentScope`, `MemoDocumentName`
+
+### `@agentrail/capabilities` — new exports + breaking signature
+
+- `OrchestrationPersistence` gains two new required methods: `loadAgentHistory(agentId)` and `writeAgentHistory(agentId, history)`.
+- `DefaultCapabilityContextOptions.compactMessages` signature updated: `sessionDir` replaced by `writeToolResultArtifact` callback injection.
+- New public exports: `OrchestrationMailboxEvent`, `OrchestrationSnapshot`, `recoverOrchestrationState`.
+- `WorkerInitMessage` gains an optional `storageConfig` field for sub-agent worker initialization.
+
+### `@agentrail/app` — breaking API surface changes
+
+- `CreateAgentAppOptions` gains: `traceStoreFactory`, `createOrchestrationPersistence`, `createWorkerStorageConfig`, and `inspector` (now accepts `true | InspectorDataSource`).
+- `createInspectorRoute` now accepts `InspectorDataSource | string` (string is deprecated backward-compat path; direct `dataDir` strings will be removed in a future release).
+- `UserMemoryConsolidationService` constructor signature changed from `(sessionManager, dataDir, config)` to `(store: AgentrailSessionStore, sessionLister: UserSessionLister, dataDir, config)`.
+- New `UserSessionLister` interface exported.
+- New `InspectorDataSource`, `InspectorSessionItem`, `SessionTraceStore` types exported.
+- New `createFilesystemInspectorDataSource(dataDir)` function exported.
+- `compactToolResults` now prefers `writeToolResultArtifact` from the session store when available.
+- Pure compaction helpers exported: `computeCompactionSplit`, `buildCompactionMessage`, `buildCompactionNotesEntry`.
+
+### `@agentrail/storage-postgres` — new package
+
+PostgreSQL reference implementation for all Agentrail storage contracts:
+
+- `PostgresSessionStore` — full `AgentrailSessionStore` including memo documents and tool-result artifacts.
+- `PostgresSessionTraceStore` + `createPostgresSessionTraceStore` factory.
+- `PostgresOrchestrationPersistence` + `createPostgresOrchestrationPersistence` factory (includes agent history).
+- `PostgresInspectorDataSource` — reads session list, messages, trace envelopes, and orchestration state from PostgreSQL without any filesystem dependency.
+- `buildSchemaDDL(schema?)` — returns idempotent DDL to create all required tables.
+- `createSqlClient(options)` — creates a `postgres` (postgres.js) connection pool.
+
+**Upgrade notes:**
+
+1. If you implement `AgentrailSessionStore`, the new optional methods do not need to be implemented immediately; the runtime falls back gracefully. However, implementing them is required to use the postgres backend.
+2. If you implement `OrchestrationPersistence`, you must add `loadAgentHistory` and `writeAgentHistory`.
+3. Replace `createInspectorRoute(dataDir)` with `createInspectorRoute(createFilesystemInspectorDataSource(dataDir))` when providing a custom `InspectorDataSource`.
+4. `UserMemoryConsolidationService` callers must provide a `UserSessionLister` (the default `SessionManager` now implements this interface).

docs/.vitepress/config.ts

@@ -99,6 +99,7 @@ export default defineConfig({
           { text: "Tool Permissions", link: "/guides/tool-permissions" },
           { text: "Add Context", link: "/guides/add-context" },
           { text: "Configure Sessions", link: "/guides/configure-sessions" },
+          { text: "Build a Storage Backend", link: "/guides/build-a-storage-backend" },
           { text: "Consume Stream (SSE)", link: "/guides/consume-stream" },
           { text: "Write a Plugin", link: "/guides/write-a-plugin" },
           {

docs/concepts/sessions.md

@@ -32,18 +32,19 @@ getOrCreate → existing session resumed
 
 The host layer depends on the `AgentrailSessionStore` contract rather than any concrete storage implementation. This means you can swap out the storage backend without changing your host code.
 
-The contract includes:
+The required contract includes:
 
 | Method                   | Called when                                                                 |
 | ------------------------ | --------------------------------------------------------------------------- |
 | `getOrCreate`            | Every request — creates or resumes the session                              |
-| `getSessionDir`          | Before agent construction — returns the session data path                   |
 | `loadMessagesWithBudget` | Every request — loads history trimmed to the token budget                   |
 | `loadAllMessages`        | During compaction — loads full history to assess whether to compact         |
 | `appendMessages`         | After agent completes — persists the new turn's messages                    |
 | `recordTurn`             | After agent completes — persists token usage                                |
 | `compactIfNeeded`        | During each request — summarizes old history if token threshold is exceeded |
 
+Additional optional methods (`readMemoryDocument`, `writeMemoryDocument`, `readToolResultArtifact`, `writeToolResultArtifact`, `listToolResultArtifactIds`) unlock memo tools and full `/workspace/memo/**` sandbox access. See [Session Store Reference](../reference/session-store.md) for the complete interface.
+
 ## Default Implementation: `SessionManager`
 
 The default session store is `SessionManager` from `@agentrail/app`. It uses the local filesystem to persist sessions, with one directory per session:

docs/examples/playground-server.md

@@ -31,11 +31,28 @@ The two host entry points are mounted in `routes/chat.ts` and `routes/stream.ts`
 // examples/playground-server/src/routes/stream.ts (simplified)
 import { createStreamRoute } from "@agentrail/app/advanced";
 import { createFileSystemSessionTraceStore } from "@agentrail/app";
+import type { WorkflowTraceEventEnvelope } from "@agentrail/app";
+import type { SessionRef } from "@agentrail/core";
 import { resolvePlaygroundProfile } from "../profiles/default-profile.js";
 import { sessionManager, sandboxManager, orchestrationRegistry } from "../context/index.js";
 import { plugins } from "../plugins/index.js";
 import { summarize } from "../agents/summarizer.js";
 
+// Cache one trace store per session so the underlying file is opened once per
+// process lifetime, not once per event.
+const traceStoreCache = new Map<
+  SessionRef,
+  ReturnType<typeof createFileSystemSessionTraceStore<WorkflowTraceEventEnvelope>>
+>();
+function getTraceStore(sessionRef: SessionRef) {
+  let store = traceStoreCache.get(sessionRef);
+  if (!store) {
+    store = createFileSystemSessionTraceStore<WorkflowTraceEventEnvelope>(dataDir, sessionRef);
+    traceStoreCache.set(sessionRef, store);
+  }
+  return store;
+}
+
 export const streamRoute = createStreamRoute({
   dataDir,
   defaultAgentId: "default",
@@ -58,14 +75,34 @@ export const streamRoute = createStreamRoute({
   getOrchestrationManager: ({ tenantId, userId, sessionId, sessionRef }) =>
     orchestrationRegistry.getManager({ tenantId, userId, sessionId, sessionRef }),
   onTraceEvent: (ctx, envelope) => {
-    const traceStore = createFileSystemSessionTraceStore(dataDir, ctx.sessionRef);
-    void traceStore.appendEnvelope(envelope);
+    void getTraceStore(ctx.sessionRef).appendEnvelope(envelope);
   },
 });
 ```
 
 Both chat and stream routes share the same session store, profile resolver, and plugin list — that reuse is the main design goal.
 
+### Inspector Route
+
+The Inspector API is mounted as a sub-app. Pass an `InspectorDataSource` to `createInspectorRoute` — for filesystem-backed setups use `createFilesystemInspectorDataSource`; for database backends, pass a `PostgresInspectorDataSource` or your own implementation:
+
+```ts
+// examples/playground-server/src/main.ts (simplified)
+import { createInspectorRoute, createFilesystemInspectorDataSource } from "@agentrail/app/advanced";
+
+// Filesystem-backed (default playground setup)
+app.route(
+  "/__inspector",
+  createInspectorRoute(createFilesystemInspectorDataSource(config.dataDir)),
+);
+
+// PostgreSQL backend — swap the data source, everything else stays the same
+// import { PostgresInspectorDataSource } from "@agentrail/storage-postgres";
+// app.route("/__inspector", createInspectorRoute(new PostgresInspectorDataSource(sql)));
+```
+
+`createInspectorRoute` is backend-agnostic: it only calls methods on the `InspectorDataSource` interface, so switching from filesystem to PostgreSQL is a one-line change.
+
 ### Profile Definition
 
 The default profile lives in `profiles/default-profile.ts`. It shows the recommended `defineProfile` shape with capability descriptors:
@@ -133,10 +170,21 @@ export const defaultProfile = defineProfile({
         },
         listSkills: () => skillManager.listSkills(),
         listWorkspaceSnapshot: (ctx) => sandboxManager.listWorkspace(ctx.sessionId),
+        // Persists compacted tool results to the session store and refreshes
+        // the live sandbox mirror so the agent can read the file immediately.
+        writeToolResultArtifact: async (ctx, toolCallId, content) => {
+          const sessionRef = `${ctx.tenantId}:${ctx.sessionId}`;
+          await Promise.all([
+            sessionManager.writeToolResultArtifact?.(sessionRef, toolCallId, content),
+            sandboxManager.refreshMemoMirror(
+              ctx.sessionId,
+              `/workspace/memo/session/tool-results/${toolCallId}.txt`,
+              content,
+            ),
+          ]);
+        },
         compactMessages: (msgs, ctx) =>
-          compactToolResults(msgs, {
-            sessionDir: ctx?.sessionDir,
-          }),
+          compactToolResults(msgs, { writeToolResultArtifact: ctx?.writeToolResultArtifact }),
         delegateSkillsToSubAgent: config.skillDelegateToSubAgent,
       },
       { cacheTtlMs: 5_000 },