stabilize storage abstractions, memo FS, and PostgreSQL reference implementation

[yai-dev]

Summary

This issue is no longer about adding a standalone Redis- or memory-based persistence layer.

The real Pre-GA goal is to fully stabilize Agentrail’s storage-related public contracts so that the default hosted experience no longer depends on filesystem-only assumptions. This includes:

1. Session persistence
2. Session trace persistence
3. Orchestration persistence, including worker/history paths
4. Inspector read-side access
5. Session/user memory access, including stable memo paths

This issue will define the stable interfaces, rewire the host SDK and Inspector around them, preserve /workspace/memo/** as the canonical memory contract, and ship @agentrail/storage-postgres as the first reference implementation.

Scope

This issue covers:

• AgentrailSessionStore contract cleanup
• SessionTraceStore export + host wiring
• OrchestrationPersistence stabilization, including agent history
• InspectorDataSource extraction
• /workspace/memo/** as the canonical memory contract
• sandbox memo mirror for non-filesystem backends
• filesystem backend parity and PostgreSQL reference backend
• required host/example/docs/changeset updates

This issue does not cover distributed orchestration queue semantics (#73) or Bash write-back synchronization for memo paths.

Canonical Memory Contract

The following paths are part of the formal hosted-memory contract:

• /workspace/memo/session/NOTES.md
• /workspace/memo/session/TODO.md
• /workspace/memo/user/USER.md
• /workspace/memo/session/tool-results/<toolCallId>.txt

They must remain readable to the agent across both filesystem and PostgreSQL backends.

Write rules are fixed as follows:

Path	Read	Write / Edit	Notes
/workspace/memo/session/NOTES.md	Yes	Yes	Session notes are directly editable by the agent
/workspace/memo/session/TODO.md	Yes	No (TodoWrite only)	TodoWrite remains authoritative
/workspace/memo/user/USER.md	Yes	No	Only the user-memory consolidation flow may write
/workspace/memo/session/tool-results/*.txt	Yes	No	Host-managed artifact output only
Bash on /workspace/memo/**	Read-only	No persistent write-back	Memo mirror is read-only in sandbox

Phase 1 — Stabilize Core Contracts

Session store

Extend AgentrailSessionStore so it can support the hosted runtime without leaking filesystem layout:

• existing session/message/turn methods remain
• add optional memo document methods:
  • readMemoryDocument
  • writeMemoryDocument
  • appendMemoryDocument
• add optional tool-result artifact methods:
  • readToolResultArtifact
  • writeToolResultArtifact

This must cover:

• NOTES.md
• TODO.md
• USER.md
• persisted tool-result artifacts

Memory index cleanup

Current MemoryIndex still leaks filesystem paths through sessionDir and userDir, and compactMessages still depends on { sessionDir }.

This issue will clean that up:

• remove sessionDir / userDir from MemoryIndex
• make MemoryIndex.entries[].path the canonical /workspace/memo/** path directly
• change compactMessages contract to receive store-backed artifact capability rather than filesystem path context

Session trace store

Keep SessionTraceStore as a session-scoped interface and expose host wiring through:

traceStoreFactory?: (sessionRef: SessionRef) => SessionTraceStore;

Orchestration persistence

OrchestrationPersistence must become the single persistence contract for both manager and worker paths.

Add:

• loadAgentHistory(agentId)
• writeAgentHistory(agentId, history)

Worker-side storage must stop directly instantiating filesystem persistence.

Memo FS parser

Introduce a memo-path abstraction layer for /workspace/memo/** that can map:

• session docs
• user docs
• tool-result artifacts

onto either filesystem-backed or database-backed storage.

Phase 2 — Host, Inspector, Sandbox, and Memory Consumer Wiring

Host SDK

createAgentApp should support:

• sessionStore
• traceStoreFactory
• createOrchestrationPersistence
• createWorkerStorageConfig
• inspector?: true | InspectorDataSource

dataDir remains only as the convenience path for default filesystem implementations.

Inspector

Extract InspectorDataSource so Inspector no longer depends on filesystem layout.

createInspectorRoute(dataDir) becomes:

createInspectorRoute(dataSource: InspectorDataSource)

createAgentApp({ inspector: true }) remains valid only for the default filesystem case. Any custom-store setup must explicitly provide InspectorDataSource.

Sandbox memo mirror

For non-filesystem backends, sandbox containers should receive a materialized memo mirror mounted at:

• /workspace/memo/session
• /workspace/memo/user

This preserves the canonical memo path contract without requiring a true filesystem backend.

Memo-aware writes must refresh the mirror after store writes.

Memory consumers

The following consumers must be updated together:

• buildMemoryIndex
• tool-result compaction
• user-memory consolidation
• sandboxed Read / Write / Edit
• TodoWrite

User-memory consolidation

UserMemoryConsolidationService must stop depending directly on SessionManager.

Its constructor should depend on:

• AgentrailSessionStore
• UserSessionLister
• existing consolidation config/state path

Required capabilities:

• loadAllMessages
• readMemoryDocument
• writeMemoryDocument

If UserSessionLister is missing, consolidation should be explicitly disabled with a warning. It must not silently degrade.

Phase 3 — PostgreSQL Reference Backend

Add a new package:

• @agentrail/storage-postgres

It should provide:

• PostgresSessionStore
• PostgresSessionTraceStore
• PostgresOrchestrationPersistence
• PostgresInspectorDataSource

Schema areas

• sessions
• session messages
• session turns
• session compaction archives
• memory documents
• tool-result artifacts
• trace envelopes
• orchestration events / snapshots / mailbox state / mailbox events / agent histories
• todo storage
• skill sub-agent logs

The PostgreSQL backend must preserve the same hosted-memory semantics as the default filesystem backend.

Phase 4 — Tests, Changesets, and Docs

Tests

Required coverage:

• filesystem and PostgreSQL contract tests for:
  • AgentrailSessionStore
  • SessionTraceStore
  • OrchestrationPersistence
  • InspectorDataSource
• memo FS behavior across both backends
• sandbox memo mirror behavior
• user-memory consolidation with PostgreSQL
• playground-server filesystem regression
• optional PostgreSQL smoke path for playground-server if wired in example

Changesets

Expected package changes:

• @agentrail/core
• @agentrail/capabilities
• @agentrail/app
• @agentrail/storage-postgres

Docs

Update at minimum:

• docs/reference/create-agent-app.md
• docs/reference/inspector-route.md
• docs/reference/session-store.md
• docs/reference/host-primitives.md
• memory / sandbox / playground docs describing /workspace/memo/**
• add a new guide:
  • docs/guides/build-a-storage-backend.md

Acceptance Criteria

• Storage-related host contracts are stabilized for Pre-GA
• Inspector no longer requires filesystem layout as its only backend model
• Worker/history orchestration persistence is unified with manager persistence
• /workspace/memo/** remains the canonical memory contract across filesystem and PostgreSQL backends
• NOTES.md, TODO.md, USER.md, and tool-result artifacts are all readable in both backends
• UserMemoryConsolidationService works without filesystem-only assumptions
• @agentrail/storage-postgres ships as the first reference backend
• Official docs explain both the stable contracts and how to implement additional storage backends