storage.md

Storage Reference

hypha uses a storage-profile model. Framework specs reference storage by stable profile ids; concrete connection details live in provider profiles or deployment config.

Local Storage Backbone

The local backbone is designed for a complete local harness:

Component	Provider	Role
Events	SQLiteEventStore	Trace, replay, audit, regression, and runtime projection source.
Structured records	SQLiteStructuredStore	Source of truth for runs, memory records, policies, evaluations, and task state.
Lexical vector recall	LocalVectorIndexProvider	Vector index with metadata filters; the default local embedding uses deterministic lexical feature hashing.
Artifacts	FileArtifactStore	Files, snapshots, large tool outputs, and exports.
Memory	HybridMemoryProvider	Simple composition of structured source of truth plus optional vector/artifact indexes.

Create the full local stack:

import { createLocalStorageBackbone } from '@codesoul-co/hypha-adapters-local';

const storage = createLocalStorageBackbone({
  rootPath: './data/storage',
  sqliteMode: 'auto',
});

sqliteMode: "auto" uses a real SQLite engine (node:sqlite when available, otherwise better-sqlite3) and falls back to JSON sidecar files only when no SQLite engine can be loaded. Use "sqlite" when SQLite is required and "json" for deterministic test fixtures. "node-sqlite" is accepted as a compatibility alias for required SQLite mode.

SQLiteEventStore also supports trace exchange as JSONL:

const count = await storage.eventStore.exportJsonl('./data/runtime/events/run_1.events.jsonl', {
  runId: 'run_1',
});

await storage.eventStore.importJsonl('./data/runtime/events/run_1.events.jsonl');

Use JSONL exports for replay fixtures, audits, regression snapshots, and local debugging. The event log remains the source of truth; exported files are portable snapshots.

The zero-dependency default is LocalHashEmbeddingProvider. It preserves repeatable lexical similarity for local/offline use; it is not a neural semantic embedding model. Inject a concrete EmbeddingProvider through the embeddings option when semantic retrieval quality is required.

Storage Provider Profile

StorageProviderProfile declares:

Field	Description
kind	Store category: relational, document, messaging, cache, vector, object, event, or hybrid.
engine	Concrete engine: sqlite, postgres, mongodb, redis, kafka, local-vector, pgvector, qdrant, milvus, chroma, file-artifact, s3, and others.
deployment	local, self_hosted, managed, or cloud.
role	How the runtime uses the store: source_of_truth, event_log, semantic_index, cache, message_queue, artifact_store, document_store, or hybrid_memory.
connection	URI/env/host/port/database/TLS metadata.
capabilities	Declared features such as structured, transactions, events, cache, queue, pubsub, streams, vector_search, metadata_filter, or artifact_bytes.

Use StorageTopologySpec to group profiles and declare default refs for relational, document, messaging, cache, vector, artifact, event, and memory stores. messagingRef is the primary queue/stream/pub-sub default; cacheRef can point to the same Redis profile when cache behavior is colocated.

Common profile factories are exported from @codesoul-co/hypha-storage:

import {
  createSQLiteStorageProfile,
  createMongoStorageProfile,
  createRedisStorageProfile,
  createKafkaStorageProfile,
  createQdrantStorageProfile,
  createPineconeStorageProfile,
} from '@codesoul-co/hypha-storage';

const eventStore = createSQLiteStorageProfile({ role: 'event_log' });
const mongo = createMongoStorageProfile({ deployment: 'cloud', tls: true });
const redis = createRedisStorageProfile({ deployment: 'cloud', tls: true });
const kafka = createKafkaStorageProfile({ deployment: 'self_hosted' });
const qdrant = createQdrantStorageProfile({ host: 'localhost', port: 6333 });
const pinecone = createPineconeStorageProfile();

Profile factories declare portable contracts; they do not instantiate provider clients. The stock Server currently composes MongoDB, Redis, local SQLite/JSON storage, the local vector index, and the filesystem artifact store. It fails configuration validation if Kafka, Postgres, Qdrant, Chroma, Pinecone, or S3 is marked enabled without a registered concrete Server composition.

Storage Recovery Contract

classifyStorageFailure(error, context) normalizes relational, document, event, object, vector, lease, and transaction failures into RecoveryFailure. The context declares the operation, provider, role, engine, resource key, expected/observed revision, idempotency key, and any known side-effect state. The classifier recognizes authentication/authorization/policy failures, timeouts, rate limits, capacity limits, revision/CAS/ETag/deadlock/lease conflicts, unavailable dependencies, and corruption or checksum invariants.

Storage mutations default to sideEffectState: "unknown" unless a durable receipt or the caller proves not_started or committed. adviseStorageRecovery() therefore requires receipt, transaction, lease, revision, or idempotency reconciliation before replaying ambiguous writes. Known committed mutations use a declared compensation or human review; integrity failures are quarantined and derived caches are invalidated. Read-only operations may retry within the shared FSM circuit budget and may use only a replica with compatible consistency and durability.

Configuration Taxonomy

Runtime configuration is grouped by function before provider:

Config Path	Function	Examples
storage.document	Document records	MongoDB local or Atlas.
storage.messaging	Cache, streams, queues, pub/sub	Redis adapter; Kafka profile contract.
storage.relational	Event logs and structured source of truth	SQLite adapter; Postgres profile contract.
storage.vector	Semantic indexes	Local JSON adapter; remote profile contracts.
storage.artifacts	File/blob payloads	Filesystem adapter; S3 profile contract.

Each store declares a deployment mode: local, self_hosted, managed, or cloud. Use .env for deployment-specific URLs, credentials, and local paths. Use config.yaml for typed structure and safe defaults.

Execution provides a concrete S3ExecutionArtifactStore and an explicit S3ExecutionArtifactStoreFactory. The Factory can be added to ArtifactStoreProviderRegistry, but it is not registered or selected by default and is not yet wired into Server composition. Its capability contract reports versioning, range reads, signed access, server-side copy, multipart upload, and content addressing; encryption remains false. Deployment requires a bucket with object versioning enabled, explicit endpoint policy, and current real MinIO/S3 acceptance evidence.

Execution also exports PostgresExecutionStoreFactory with the stable execution-store.postgres identity. The Factory validates an explicit Postgres connection string, TLS mode, pool limits, and operation timeouts; initializes the versioned schema under an advisory lock before returning the Store; and closes its pool when initialization fails. It can be added to ExecutionStoreRegistry, but it is not registered or selected by default and is not wired into Server composition. Current adapter evidence covers runtime-schema-validated records, restart persistence, idempotency, CAS, lease renewal and fencing, concurrent migration, lock timeout, database outage and crash recovery, unsupported schema, read-only migration failure, corrupt-record quarantine, health, and close. These adapter guarantees do not by themselves make a deployment release-ready.

Local defaults are organized under data/: events in data/runtime/events/, structured records in data/runtime/structured/, vector indexes in data/storage/vector/, artifacts in data/storage/artifacts/, and system logs in data/logs/system.log.

Document Storage

MongoDB is currently used by the API server for permanent conversation memory. Local deployment uses host/port settings:

MONGODB_HOST=localhost
MONGODB_PORT=27017
MONGODB_DATABASE=hypha
MONGODB_DEPLOYMENT=local

Cloud deployment should use a provider URI, such as MongoDB Atlas:

MONGODB_URI=mongodb+srv://user:password@cluster.example.mongodb.net/hypha
MONGODB_DEPLOYMENT=cloud
MONGODB_TLS=true

MONGODB_URI takes precedence over host/port config. Optional fields include MONGODB_AUTH_SOURCE, MONGODB_REPLICA_SET, and MONGODB_DIRECT_CONNECTION.

Messaging Storage

Redis is currently used for temporary chat memory, session-scoped streams, and API runtime cache behavior. Local deployment:

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0
REDIS_KEY_PREFIX=hypha:
REDIS_DEPLOYMENT=local

Cloud deployment should use a provider URL:

REDIS_URL=rediss://default:password@redis.example.com:6380/0
REDIS_DEPLOYMENT=cloud
REDIS_TLS=true

REDIS_URL takes precedence over host/port config. KV_URL and RENDER_REDIS_URL are also recognized as compatibility fallbacks.

Kafka configuration is available under storage.messaging.kafka as an extension contract. The stock Server does not compose a Kafka client and rejects KAFKA_ENABLED=true; a deployment must first provide an adapter, lifecycle hooks, readiness probe, and real acceptance tests.

Relational and Vector Extension Points

StructuredStoreProvider is the relational/source-of-truth interface. SQLiteStructuredStore is the local implementation. Other providers such as Postgres or MySQL must implement the same methods: get, insert, update, query, and transaction, then be registered in Server composition before their configuration can be enabled.

VectorIndexProvider is the vector retrieval interface. LocalVectorIndexProvider is the local implementation. Provider profiles cover pgvector, Qdrant, Milvus, Chroma, Pinecone, and Weaviate; each concrete adapter must implement upsert, search, and delete and register lifecycle and readiness behavior before Server enablement.

Vector stores are retrieval indexes, not the full source of truth. Persist factual memory records in structured storage, then index selected semantic or episodic values in a vector provider.

Memory Composition

MemorySpec can reference:

structuredStoreRef
vectorIndexRef
artifactStoreRef
embeddingProviderRef
retrievalPolicy
writePolicyConfig

The current HybridMemoryProvider writes scoped records to structured storage, embeds indexable memory types, writes vectors when a vector provider is configured, and reads back full records from structured storage after vector search. Search merges vector candidates with text matches from structured storage, deduplicates by record id, and returns the highest-ranked bounded result set.

Use MemoryManager above a provider when agent code reads or writes memory. The manager applies MemoryWritePolicy and records memory events when a trace recorder is provided:

const storage = createLocalStorageBackbone({ rootPath: './data/storage' });
const memory = new MemoryManager(storage.memory, { trace: storage.eventStore });

MemoryContextBuilder is the kernel-level bridge from memory to model context. It searches memory by text or vector, enforces configured memory type filters, applies budget limits, records provenance for each selected item, and injects memory as tagged context data rather than executable instructions.