agent-memory-runtime

A local-first, provider-agnostic persistent memory runtime for AI agents.

Project Thesis

This project treats agent memory as a user-owned, durable store that is independent of any single LLM provider. The goal is to make memory reusable across OpenAI, Anthropic, Ollama, and other models without baking memory logic into provider integrations.

V1 Scope

This milestone includes:

• Core Memory data model for semantic, preference, episodic, and procedural memories.
• SQLite-backed persistent storage.
• Memory metadata covering identity, type, namespace, content, status, timestamps, access tracking, and supersession.
• A MemoryStore interface and concrete SQLiteMemoryStore implementation.
• Basic CRUD operations plus superseding and namespace/type listing.
• Example usage script and unit tests.

Getting Started

pip install -e .
pytest

Current Structure

• src/agent_memory/models.py — memory data model
• src/agent_memory/storage/sqlite.py — SQLite memory store implementation
• src/agent_memory/storage/schema.sql — SQLite schema for persistent memory + FTS5 index
• src/agent_memory/retrieval/ — retrieval abstractions, lexical (FTS5), semantic, and hybrid retrievers
• src/agent_memory/policy/ — memory admission, consolidation, supersession, and utility scoring
• src/agent_memory/context/ — context compilation (relevance + utility + token budget -> LLM-ready context)
• src/agent_memory/extraction/ — conversation -> structured candidate memories (LLM-agnostic)
• src/agent_memory/providers/ — provider-independent LLMProvider interface + OpenAI/Anthropic/Ollama adapters
• src/agent_memory/runtime/ — AgentRuntime: thin orchestration of provider + retrieval + compiler + extraction + policy
• tests/test_sqlite_memory_store.py — tests for store behavior
• tests/test_lexical_retrieval.py — tests for lexical retrieval
• tests/test_context_compiler.py — tests for context compilation
• tests/test_memory_extraction.py — tests for memory extraction
• tests/test_providers.py — tests for provider adapters (no network calls)
• tests/test_agent_runtime.py — tests for the agent runtime
• examples/01_basic_memory.py — example of creating and retrieving memories
• examples/02_lexical_retrieval.py — example of full-text lexical retrieval
• examples/06_context_compiler.py — example of compiling retrieved memories into an LLM-ready context
• examples/07_memory_extraction.py — example of extracting candidate memories from a conversation
• examples/08_provider_runtime.py — example of a full provider + memory chat turn (offline, fake provider)
• examples/09_cross_provider_handoff.py — example of switching providers while memory stays put

Retrieval

Memory storage answers: "What memories exist?" Retrieval answers: "Which memories are relevant to this query?"

                       Query
                         |
              +----------+----------+
              |                     |
              v                     v
        LexicalRetriever      SemanticRetriever
          SQLite FTS5          Local Embeddings
              |                     |
              v                     v
             BM25              Cosine Similarity

store = SQLiteMemoryStore("memories.db")

lexical = LexicalRetriever(store)
lexical.search("coding language")

embedder = SentenceTransformerEmbeddingProvider()
index = SQLiteEmbeddingIndex(store)
semantic = SemanticRetriever(store=store, embedder=embedder, index=index)
semantic.sync_index()
semantic.search("What language should I use for interviews?")

• Lexical retrieval (LexicalRetriever) is good for exact terms, identifiers, names, and keywords. It uses SQLite FTS5 with BM25 ranking.
• Semantic retrieval (SemanticRetriever) is good for matching meaning even when wording differs. All embedding generation happens locally (e.g. sentence-transformers); embeddings are persisted as blobs in SQLite and compared with cosine similarity — no external vector database.

Both return the same RetrievalResult type and, by default, only return active memories (superseded/archived/ deleted memories are excluded). Hybrid retrieval combining both signals is planned for the next milestone.

Local embeddings are an optional extra so the core SQLite + lexical runtime stays lightweight:

pip install -e ".[local-embeddings]"

Hybrid Retrieval

             Query
               |
      +--------+--------+
      |                 |
   FTS5/BM25        Embeddings
   lexical          semantic
      |                 |
      +--------+--------+
               |
         Reciprocal
         Rank Fusion
               |
               v
         Top-K Memories

lexical = LexicalRetriever(store)
semantic = SemanticRetriever(store=store, embedder=embedder, index=index)
semantic.sync_index()  # explicit; hybrid never re-embeds automatically

hybrid = HybridRetriever(lexical_retriever=lexical, semantic_retriever=semantic)
hybrid.search("preferred Python language for technical interviews")

HybridRetriever composes the existing retrievers rather than duplicating their logic. It over-fetches candidates from each (limit * candidate_multiplier), merges duplicate memories by ID, and ranks them with Reciprocal Rank Fusion (score = sum(1 / (rrf_k + rank)) per contributing retriever). We use rank fusion instead of averaging BM25 and cosine scores because the two scores have different, uncalibrated distributions — combining ranks avoids that problem. Each result's metadata exposes lexical_rank, semantic_rank, and the component scores for debugging/evaluation.

• Lexical retrieval is strong for exact terminology, names, identifiers, and keywords.
• Semantic retrieval is strong for paraphrases, conceptual similarity, and wording variation.
• Hybrid retrieval combines both, so a memory needs to be found by only one signal to be considered.

Milestone 5 (below) adds memory policy signals (importance, confidence, recency, frequency, consolidation). These are intentionally not combined with retrieval scores yet.

Memory Policy and Lifecycle

              Candidate Memory
                     |
                     v
              Memory Policy
                     |
        +------------+------------+
        |            |            |
        v            v            v
      STORE         MERGE        REJECT
        |
        +-------> SUPERSEDE
                     |
                     v
              Persistent Memory

engine = MemoryPolicyEngine(store=store)  # optional: semantic_retriever, admission_config, duplicate_config

decision, memory = engine.process(MemoryCandidate(
    content="User prefers Python for coding interviews",
    type=MemoryType.PREFERENCE,
    namespace="user",
    confidence=0.9,
    importance=0.9,
))

MemoryPolicyEngine.evaluate() inspects a MemoryCandidate and returns a MemoryPolicyDecision (STORE/REJECT/MERGE/SUPERSEDE) without mutating storage; apply() performs the actual mutation (process() does both). This keeps decisions inspectable/testable separately from their effects.

• Admission: score = confidence_weight*confidence + importance_weight*importance + novelty_weight*novelty (defaults 0.35/0.40/0.25, threshold 0.50) — configurable heuristics, not learned parameters.
• Duplicates: exact normalized-content matches (same namespace/type) always MERGE; an optional, disabled-by-default semantic near-duplicate check can reuse SemanticRetriever for paraphrased duplicates.
• Supersession: explicit only (MemoryCandidate.supersedes_id) — no automatic natural-language contradiction detection yet.
• observation_count (times a fact was observed, incremented on MERGE) is distinct from access_count (times a memory was retrieved/used).

Retrieval never records access automatically (to avoid double-counting across HybridRetriever's two underlying retrievers). Call engine.record_access(memory_ids) explicitly once memories are actually used.

Memory Utility

Memory Utility
    =
Importance
  + Confidence
  + Recency
  + Frequency

MemoryUtilityScorer answers "how valuable is this stored memory right now?" — independent of any query. Recency uses half-life decay (0.5 ** (age_days / half_life_days), default 30 days) measured from last_observed_at (falling back to updated_at, then created_at). Frequency uses a saturating transform (1 - exp(-access_count / frequency_scale)) so raw access counts don't grow unbounded.

Retrieval relevance ("which memories match this query?") and memory utility ("which memories are valuable over time?") are intentionally kept separate. The Context Compiler (below) decides which retrieved memories actually enter an LLM's context window, using both signals together.

Context Compilation

                   User Query
                       |
                       v
                Hybrid Retrieval
                       |
                candidate memories
                       |
             +---------+---------+
             |                   |
             v                   v
       Retrieval Rank      Memory Utility
             |                   |
             +---------+---------+
                       |
                       v
                Context Compiler
                       |
             +---------+---------+
             |                   |
           Dedupe            Token Budget
             |                   |
             +---------+---------+
                       |
                       v
                 LLM Context

Retrieval and utility answer different questions than context compilation:

• Retrieval relevance: how well does this memory match the current query?
• Memory utility: how valuable is this memory over time?
• Context compilation: is this memory valuable and relevant enough to consume limited context-window tokens, right now?

ContextCompiler consumes RetrievalResults from any Retriever (lexical, semantic, or hybrid) — it does not perform retrieval itself, and never mutates persistent memory.

from agent_memory import ContextCompiler

compiler = ContextCompiler()  # optional: utility_scorer, token_counter, relevance_weight/utility_weight
results = hybrid.search("preferred Python language for technical interviews")
compiled = compiler.compile(results, token_budget=500, max_memories=10)

print(compiled.rendered_text)
compiler.record_usage(compiled, policy_engine=engine)  # explicit; compile() never mutates access_count

• Relevance normalization: raw BM25/cosine/RRF scores are not comparable, so retrieval position is converted into a normalized signal: relevance(rank) = 1 / log2(rank + 1) (1-based rank). This works uniformly across LexicalRetriever, SemanticRetriever, and HybridRetriever without score calibration.
• Selection score: relevance_weight * relevance + utility_weight * utility, defaulting to 0.70/0.30 so relevance normally dominates — a highly important memory about Kubernetes shouldn't be injected into an unrelated interview-language query just because its utility score is high. Utility mainly breaks ties among already-relevant memories.
• Deduplication: always by memory ID; optionally (default on) by exact normalized content too. Semantic near-duplicate detection stays in the policy layer, not here.
• Status: superseded/archived/inactive memories are defensively excluded, even though retrievers already filter them.
• Token budget: uses a pluggable TokenCounter (default SimpleTokenCounter, a deterministic character-based estimate — not exact GPT/Claude/Llama tokenization). Candidates are added greedily in selection-score order; an oversized candidate is skipped (not fatal) so smaller, lower-ranked candidates can still fit. tokens_used is guaranteed <= token_budget.
• Explainability: every selected memory keeps its retrieval_rank, retrieval_score, relevance_score, utility_score, selection_score, and estimated_tokens; every excluded candidate records an ExclusionReason (inactive, duplicate_id, duplicate_content, token_budget, max_memories).
• Explicit usage recording: compile() is pure (no mutation, mirroring retrieval). Call compiler.record_usage(compiled_context, policy_engine) separately to increment access_count for only the memories actually sent to a model.

No LLM provider is required for context compilation — the output is plain, provider-neutral text that any provider adapter can consume later.

Memory Extraction

               Conversation
                    |
                    v
              LLM Extractor
                    |
             MemoryCandidates
                    |
                    v
              Memory Policy
                    |
          +---------+---------+
          |         |         |
        STORE     MERGE     REJECT
          |
       SUPERSEDE
       when explicitly resolved

The LLM does not write memory. LLMMemoryExtractor only proposes structured MemoryCandidates from a conversation; the deterministic MemoryPolicyEngine (Milestone 5) still decides what becomes persistent. The extractor never touches MemoryStore, never calls MemoryPolicyEngine, never increments access_count, and never alters embedding indexes.

The full pipeline so far:

Conversation -> Extraction -> Policy/Lifecycle -> Persistent Memory -> Hybrid Retrieval -> Context Compiler -> LLM

from agent_memory import Conversation, ConversationMessage, LLMMemoryExtractor, MessageRole

extractor = LLMMemoryExtractor(generator)  # generator: StructuredGenerator (fake in tests/examples)
conversation = Conversation([
    ConversationMessage(MessageRole.USER, "For coding interviews I prefer Python."),
])
result = extractor.extract(conversation, conversation_id="demo-1")

for extracted in result.memories:
    decision, memory = policy.process(extracted.candidate)  # STORE / MERGE / REJECT / SUPERSEDE

• StructuredGenerator is the smallest possible abstraction (generate(prompt) -> str) so the extractor depends on an interface, not a concrete provider. Concrete OpenAI/Anthropic/Ollama adapters are planned for Milestone 8; tests and examples use a deterministic fake generator — no network access or API keys required.
• build_extraction_prompt lives in its own module and documents the four supported memory types (preference, semantic, episodic, procedural), confidence vs. importance, namespace conventions, and instructs the model to be conservative and to distinguish user-stated facts from assistant suggestions or hypotheticals.
• Parsing/validation never trusts raw model output: malformed JSON at the top level fails the whole response with a warning, but a single malformed memory entry (bad type, out-of-range confidence/importance, invalid source index, missing content, ...) only drops that entry — other valid entries in the same response still come through. Exact normalized-content duplicates within one response are also collapsed.
• ExtractedMemory wraps a MemoryCandidate with extraction provenance — evidence, source_message_indices, and a non-binding update_hint — without duplicating confidence semantics. The model never specifies a supersedes_id: it has no reliable knowledge of persistent-memory IDs, so cross-memory contradiction resolution stays out of scope for this milestone.
• MemoryCandidate.source is set by the extractor from the caller-supplied source/conversation_id, not by the model.
• Extraction ≠ consolidation: the extractor has no knowledge of existing persistent memories. Re-extracting an already-known preference from a later conversation still produces a fresh MemoryCandidate; it's MemoryPolicyEngine's existing duplicate detection that recognizes it and MERGEs instead of STOREs.

Provider Runtime

                     User
                      |
                      v
                 AgentRuntime
                      |
             +--------+--------+
             |                 |
             v                 v
      Hybrid Retrieval      Conversation
             |
             v
      Context Compiler
             |
             +--------+
                      |
                      v
                 LLM Provider
          Ollama / OpenAI / Anthropic
                      |
                      v
                 Assistant
                      |
                      v
             Memory Extraction
                      |
                      v
                Memory Policy
                      |
                      v
               Local Memory DB

The LLM provider is replaceable. Persistent memory remains local and provider-independent. A conversation with a local Ollama model can create memory that OpenAI or Anthropic later retrieves and uses — the same MemoryStore, EmbeddingIndex, Retriever, and ContextCompiler are reused regardless of provider.

AgentRuntime is intentionally thin: it does not reimplement retrieval, scoring, token budgeting, extraction parsing, or admission/consolidation. For one turn it runs, in order:

retrieve (Retriever.search(user_message)) -> compile (ContextCompiler) -> generate (LLMProvider)
    -> record access for selected memories -> extract (LLMMemoryExtractor, current turn only)
    -> policy.evaluate() / policy.apply() for each candidate

from agent_memory import AgentRuntime, ContextCompiler, LLMMemoryExtractor, MemoryPolicyEngine
from agent_memory.providers import OllamaProvider, OpenAIProvider
from agent_memory.retrieval import LexicalRetriever

store = SQLiteMemoryStore("memories.db")
runtime_local = AgentRuntime(
    provider=OllamaProvider(model="llama3"),
    retriever=LexicalRetriever(store),
    context_compiler=ContextCompiler(),
    policy_engine=MemoryPolicyEngine(store=store),
    extractor=LLMMemoryExtractor(...),  # any StructuredGenerator, e.g. ProviderStructuredGenerator(provider)
)
result = runtime_local.chat([], "For coding interviews I prefer Python.")

# Later, a different provider reads the SAME memory:
runtime_openai = AgentRuntime(provider=OpenAIProvider(model="gpt-4o-mini"), retriever=LexicalRetriever(store), ...)
runtime_openai.chat([], "Solve Two Sum using my preferences.")  # retrieves the memory written above

• LLMProvider is a minimal interface (name, generate(messages) -> str); concrete OpenAIProvider, AnthropicProvider, and OllamaProvider adapters never expose their SDK's request/response objects. SDK clients are created lazily on first generate() call — importing a provider module, or even constructing the class, never requires the optional dependency or makes a network call. Anthropic's separate system parameter is handled inside the adapter; ConversationMessage stays provider-neutral either way.
• ProviderStructuredGenerator adapts any LLMProvider into the StructuredGenerator that LLMMemoryExtractor (Milestone 7) expects, so a provider can drive extraction too — without LLMMemoryExtractor ever importing a concrete provider.
• Memory context injection reuses ContextCompiler's existing rendering; the runtime adds one small, provider-neutral system instruction: use stored memory when relevant, treat it as possibly stale, and let the current user message take precedence over any contradictory remembered preference.
• Extraction input is deliberately narrow: only the just-completed turn (new user message + assistant response), not unbounded conversation history — avoiding both prompt bloat and self-retrieval within the same turn (new memories are written after this turn's retrieval already happened).
• Failure boundaries: retrieval and provider-generation failures propagate to the caller unchanged (no response is fabricated, and a failed generation never triggers extraction). An extraction failure after a valid response does not lose that response — it's returned with a warning on extraction_result instead.
• Semantic index sync stays explicit: runtime.sync_memory_index() mirrors SemanticRetriever.sync_index() — chat() never hides a reindex inside a normal turn.
• Optional provider dependencies: pip install -e ".[openai]", ".[anthropic]", ".[ollama]", or ".[providers]" for all three. Core pip install -e . never requires any of them. Credentials come only from environment variables (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY) or explicit constructor arguments — never hard-coded, never logged.

See examples/08_provider_runtime.py and examples/09_cross_provider_handoff.py for full, offline (no network required) walkthroughs. MCP and a CLI are planned next.