Conversation History: Architecture change

[vanguard-hack]

Feature Ticket: Chatbot Short-Term & Long-Term Memory Architecture (Zoho Catalyst)

Summary

Add memory to the crime analytics assistant so it can (a) hold context within an active conversation, and (b) recall relevant facts and past interactions across sessions — per officer, not shared. Built entirely on native Catalyst services: Cache (short-term), NoSQL (durable structured long-term), and QuickML Knowledge Base (semantic long-term recall). No external memory store, vector DB, or session store needed.

Architecture diagram (Lucid): https://lucid.app/lucidchart/a97dda61-5ffb-4a2c-972f-d2111ddcf68b/edit

This complements the router/agentic-fallback architecture from the earlier ticket — memory context is assembled by the Request Orchestrator before the query reaches the Router, so every pipeline (RAG, Text2ZCQL, Fast Vision, Agentic Fallback) benefits from it automatically.

---

Why this split (short-term vs. long-term, and why two long-term stores)

• Short-term memory needs to be fast and cheap to read/write on every single turn — this is what makes a conversation feel coherent. It doesn't need to survive forever. → Catalyst Cache (Redis-backed, sub-millisecond, TTL-based).
• Long-term structured memory (preferences, known entities, recent case references) needs to persist across sessions and be queried by a known key (badge ID) — a handful of small, precise facts, not a wall of text. → Catalyst NoSQL, same service already used elsewhere in this project, with TTL available per-item for facts that should age out.
• Long-term semantic memory (searchable summaries of past conversations — "what did we discuss about FIR #4029 last week") isn't a key-value lookup, it's a meaning-based search. Catalyst's QuickML Knowledge Base already does exactly this — upload text, it chunks/embeds/retrieves for you — so we reuse it instead of standing up a separate vector database.

Using three different tools for three different access patterns, all native to Catalyst, is simpler to operate than forcing one store to do all three jobs.

---

1. Short-Term Memory — Catalyst Cache

Purpose: Working memory for the active conversation — recent turns, agent scratchpad state, anything the assistant needs "in mind" right now.

Design:

• Cache segment: chat-sessions (create this once from the Catalyst console — segments can't be created via API).
• Key: session_id
• Value: JSON-stringified object (Cache values are String type only):

  {  "badge_id": "KSP-4521",  "turns": [    { "role": "user", "text": "...", "ts": 1755600000 },    { "role": "assistant", "text": "...", "ts": 1755600004 }  ],  "agent_scratchpad": { "last_tool_call": "...", "iteration": 2 }}
  

• TTL: sliding window, 30–60 minutes of inactivity (reset TTL on every turn). Configurable — confirm exact value with the team based on typical officer session length.
• Cap the turns array (e.g., last 15–20 turns) before writing back, to keep the JSON blob small — Cache is meant for fast access, not unbounded history. Full history lives in chat_session_turns (below).

Requirements:

• Request Orchestrator reads this on every turn (fast path, no DB round-trip to a heavier store).
• After the response is generated, Orchestrator writes the updated buffer back and resets the TTL.
• If the Cache key has expired (session genuinely idle-timed-out), treat it as a new session — don't silently resurrect old context from chat_session_turns into the live buffer without the officer knowing.

---

2. Durable Session Log — Catalyst NoSQL (chat_session_turns)

Purpose: A durable, short-lived (TTL'd) copy of every turn — safety net if Cache evicts early, and the actual input to the nightly consolidation job.

Table schema (ready to create in the console — matches the Create Table screen):

Field	Value
Table Name	chat_session_turns
Partition Key	session_id — String
Include Sort Key?	Yes
Sort Key	turn_timestamp — Number (epoch millis, so turns sort chronologically)
Additional Sort Keys	none needed initially
Time To Live (TTL) attribute	expires_at — set to turn_timestamp + 7 days at write time

Other attributes: session_id_range, summary_excerpt, precinct, created_at, retention_expires_at (the policy expiry date, separate from any TTL — used by a manual/scheduled cleanup process, not by Catalyst's auto-TTL, since deleting a QuickML KB document requires an explicit API call, not just letting a NoSQL row expire).

---

Retention & Compliance

This is a police platform, so conversational memory needs the same discipline as case data:

• Every memory write (short-term write-back, long-term fact write, KB document push) is logged to the immutable Audit Layer already defined in the architecture ticket — badge ID, memory type, action, timestamp.
• Memory is per-officer, never shared by default. If a future feature wants precinct-level shared memory (e.g., "the team's common queries"), that's a distinct, explicitly-scoped feature — not a default behavior.
• Support a deletion path: an officer (or an admin, per policy) should be able to request their memory be cleared. Using memory_kb_documents as the index, this means: delete matching rows from officer_long_term_memory, delete the referenced QuickML KB documents by ID, and delete the pointer rows themselves.
• If a case gets sealed or a record is expunged, any memory referencing that case (structured entity references, and any KB documents mentioning it) should be identifiable and removable — this needs a defined process, not just "wait for TTL," since KB documents don't auto-expire on their own.

---

Acceptance Criteria

• Cache segment chat-sessions created; short-term buffer read/written on every turn with sliding TTL
• chat_session_turns NoSQL table created per the schema above; every turn logged with 7-day TTL
• officer_long_term_memory NoSQL table created; structured facts readable at request time, keyed by badge_id
• Memory Consolidation Function runs on schedule, summarizes unconsolidated sessions, writes facts + pushes KB documents
• memory_kb_documents pointer table created and populated by the consolidation job
• Semantic memory recall (QuickML KB) is triggered only on detected recall intent, not on every turn
• Semantic memory recall results pass through the Pre-Retrieval Clearance Filter before reaching the LLM
• Every memory write is logged to the Audit Layer
• A deletion path exists for clearing an officer's memory (structured facts + KB documents + pointers)
• Confirmed current availability/limits of QuickML Knowledge Base/RAG on the target data center before relying on it as a hard dependency

---