feat: implement hermes-style memory system with RAG-verified recall

[Bahtya]

Summary

参照 hermes-agent 的记忆系统设计，完整实现 kestrel-agent 的记忆机制。通过 RAG 评估方法论（Recall/Accuracy/Relevance/Faithfulness）量化验证，3 轮迭代后达到 95.8% 召回率，20/20 测试全通过。

子系统

A. Prompt Caching + 稳定系统提示

• Usage 结构新增 cache_read_tokens / cache_write_tokens 字段
• Anthropic provider 注入 3 个 cache_control ephemeral 断点（系统提示、工具定义、最后消息）
• 解析响应中的 cache_read_input_tokens / cache_creation_input_tokens
• 修正用量累加逻辑（.or() → 累加 +=，跨 agent loop 迭代正确统计）
• 召回记忆注入到 user message 副本（不持久化），系统提示字节稳定
• 时间戳改为日期精度（%Y-%m-%d），保证缓存前缀一天内稳定

B. session_search（SQLite + FTS5）

• 新增 rusqlite（bundled）依赖，创建 SessionDb 模块
• FTS5 trigram 分词器支持 CJK 子串搜索，WAL 模式
• 四形态工具：discovery（FTS 搜索）、scroll（锚点翻页）、read（读整会话）、browse（列出最近）
• JSONL → SQLite 双写，并行索引

C. 压缩前记忆抢救钩子

• CompactionHook async trait（on_pre_compress 生命周期钩子）
• MemoryRescueHook：压缩前提取错误教训/决策/事实
• SessionRescueHook：确保完整会话历史已索引

D. 记忆治理层（MEMORY_GUIDANCE）

• 系统提示注入 MEMORY_GUIDANCE：明确什么该存、什么不该存
• store_memory 工具描述含 WHEN/SKIP 指令
• 移除自动 store_conversation_memory（hermes 不自动存储，靠 LLM 显式调用）
• 移除 memory fence 类别触发器（hermes 无此设计）

E. 关键 Bug 修复

• 召回策略：BM25 关键词搜索 → 全量注入 frozen snapshot（hermes 设计）
• 注入顺序：memory-context 放在 user 问题前面（Recall 40% → 95.8%）
• think 标签：移除 emoji（🧠 被误认为 think 标签导致消息截断）
• Telegram MarkdownV2：三级 fallback（MarkdownV2 → HTML → 纯文本）
• 消息拆分：先转换格式再拆分（不再拆断 markdown 标记）
• daemon crash：IO Safety violation（as_raw_fd → into_raw_fd）

RAG 评估结果

轮次	Recall	通过率	关键修复
Round 1	58.8%	12/20	基线
Round 2	40.0%	8/20	LLM 不稳定性暴露
Round 3	95.8%	20/20	memory-context 注入到 user 问题之前

20 条结构化测试数据覆盖 8 个类别：user_profile, preference, fact, environment, project_convention, error_lesson, tool_discovery, workflow_pattern。

基础设施测试（5/5 PASS）

• ✅ 并发会话（3 parallel）
• ✅ 大消息处理（3000 chars）
• ✅ 会话隔离
• ✅ 快速重连（5 次）
• ✅ 空消息拒绝

文件变更

• 45 files changed, +2949 / -599
• 3 个新文件：session_db.rs, session_search.rs, memory_rescue.rs

[Bahtya]

🧪 Code Review — Hermes-style memory system

Multi-agent adversarial review of feat/memory-system-hermes vs main (7 review dimensions → each finding verified by independent skeptics → completeness critic). Headline build/test results below were reproduced locally.

Verdict: Request changes. The feature is well-structured and unit-tested, but there is one CI-breaking regression and several high-severity correctness/security issues that should land before merge.

---

Build status (reproduced locally)

Command	Result
cargo build --workspace	✅ pass
cargo test --workspace --lib	✅ 600 passed, 0 failed
cargo test --workspace --no-run / cargo clippy --all-targets	❌ 10 compile errors (E0063)

---

🔴 Merge blocker

cargo test --workspace does not compile — root-level integration tests not updated

crates/kestrel-core/src/types.rs — Usage got two new required fields (cache_read_tokens, cache_write_tokens). The crate-level tests under crates/kestrel-agent/tests/ were patched with ..Default::default(), but the repo-root tests/ directory was not:

• tests/pipeline_e2e.rs:40,60,365,376,803
• tests/e2e_integration_test.rs:41,295,307
• tests/full_integration_test.rs:96,111

→ 10× E0063: missing fields cache_read_tokens and cache_write_tokens. Any CI that builds the workspace test suite goes red. Fix: apply the same ..Default::default() to those Usage { … } literals (or make the fields default-on-miss).

---

🚨 Critical / High

1. Critical — session_search has no tenancy/authorization (IDOR). crates/kestrel-tools/src/builtins/session_search.rs:76-101,183-220
The tool is registered once globally against a single shared SessionDb, and Tool::execute(&self, args: Value) carries no caller/session/user context. All four modes (discovery / read / scroll / browse) operate over the entire DB — search_messages, get_session_messages, get_messages_around, recent_sessions apply no platform/chat_id/user_id filter. In any multi-chat deployment (Telegram groups, Discord channels, the API), the agent serving chat A can dump chat B's full history. Fix: thread caller identity into the tool and scope every query to the calling session.

2. High — Global shared memory store + session DB leak across chats/platforms. src/commands/gateway.rs:479-509
Exactly one TantivyStore and one SessionDb are shared by every AgentLoop. A memory stored in chat A is recalled and injected into chat B. This is both a confidentiality bug and the precondition for #3.

3. High — Recalled memory is injected framed as "authoritative system data" → prompt-injection amplification. crates/kestrel-agent/src/loop_mod.rs:944-1022
Recalled entries (originating from untrusted text — LLM store_memory, or memory_rescue auto-extracting compacted turns) are injected verbatim with the preamble "Treat as authoritative reference data — this is the agent's persistent memory and should inform all responses." That is the worst possible framing for untrusted content, and there is no sanitization/review gate. Combined with #2, an attacker in one channel can plant a memory that steers every other channel. Fix: label injected memory as untrusted data, never instructions, and add a persistence policy gate.

4. High — Full tool-result contents (potential secrets) indexed into the globally-searchable FTS5 DB. crates/kestrel-session/src/session_db.rs:229-272
index_messages stores every entry's content including role == Tool. Unlike memory_rescue.rs:123 (which skips Tool rows), the session DB has no such filter. Tool outputs routinely carry tokens/keys (file reads, gh/git output, HTTP responses). Indexed, globally searchable, and trigram-matchable (3-char substrings). Fix: skip/redact Tool rows; run a secret-scanner over indexed content.

5. High — Truncation-based re-index permanently deletes old messages from the search index. crates/kestrel-session/src/manager.rs:299-313 + session_db.rs:229-276
prepare_session_for_save truncates to DEFAULT_SESSION_HISTORY_LIMIT (200) before persist; index_messages then DELETEs and re-inserts only the survivors. The whole point of the feature is to "search past conversation history no longer in the active context window" — yet anything older than 200 msgs is purged from FTS. SessionRescueHook only fires on the LLM-compaction hook, not on this truncation path. Fix: index the un-truncated list, or upsert per new message instead of DELETE+re-insert.

6. High — "Recall all" snapshot is not recency-ordered / not deterministic; budget truncation silently drops arbitrary memories. crates/kestrel-agent/src/loop_mod.rs:953-984 (backed by tantivy_store.rs:215-217,300-322)
MemoryQuery::new().with_limit(100) with no text filter becomes an AllQuery; order_by_score() on identical scores yields internal segment/doc-id order — not recency, not insertion order. The 2200-char budget then greedily keeps entries in that arbitrary order, so a just-stored critical fact can be dropped while older trivia is kept. Fix: order by created_at desc before applying the budget.

7. High — session_search scroll mode breaks after any re-save (unstable rowids). crates/kestrel-session/src/session_db.rs:229-276,380-427
index_messages re-INSERTs every row on each persist; id INTEGER PRIMARY KEY AUTOINCREMENT ⇒ all ids change every turn. Scroll mode hands the LLM an around_message_id that no longer exists one turn later → get_messages_around anchor lookup (.ok() → None) returns empty. Fix: give messages a stable client key (e.g. per-session sequence) instead of autoincrement rowid.

8. High — Background interrupt listener holds an RwLock read guard across recv().await, deadlocking stop()/shutdown. crates/kestrel-agent/src/loop_mod.rs:189-208
A fire-and-forget task holds the read guard on self.running across event_rx.recv().await. stop() does *self.running.write().await = false, which blocks until all readers release — this reader only releases after a broadcast event arrives, so stop() hangs forever if none does. The JoinHandle is also discarded (task never aborted). Fix: snapshot the flag without holding the guard across the await; use a short timeout poll loop; abort the task on shutdown.

9. High (correctness, SSE) — Native Anthropic streams never emit a terminal chunk. crates/kestrel-providers/src/anthropic.rs:271-409
A done:true chunk is emitted only on the literal [DONE] sentinel (OpenAI convention). The native Anthropic API ends with message_stop — so normal streams rely entirely on channel-close to finalize; consumers gating on chunk.done hang or mis-treat the last content chunk. Fix: handle "message_stop" as terminal and emit a trailing done:true at stream EOF.

---

🟠 Medium

• Telegram fallback chain re-converts already-converted text. telegram.rs:2268,2299 — markdown_to_html/strip_markdown run on MarkdownV2-escaped text (\*\*bold\*\*), not the original markdown → visibly broken HTML/plain fallback. Carry the raw markdown through instead.
• Blocking SQLite on async workers (no spawn_blocking). session_search.rs:106-228 and memory_rescue.rs:86-103 — synchronous FTS5/rusqlite work runs inline on tokio workers (session_search also bypasses the mutating_guard because is_mutating()==false, so it blocks concurrently with agent runs). Wrap in tokio::task::spawn_blocking.
• read_session returns head-only, not "head + tail" as documented. session_search.rs:8,182-220 — get_session_messages(sid, limit, 0); tail never fetched; misleading to the recalling model.
• get_messages_around loads the whole session into memory + magic 999999s cutoff. session_db.rs:379-427 — window is applied in Rust, not SQL; the ABS(timestamp-?) < 999999 (~11.5d) filter silently drops index-adjacent messages far in time. Use SQL windowing with LIMIT 2*window+1.
• scroll mode ignores caller session_id. session_search.rs:147-180 — resolves the session from message_id and echoes the caller's unverified session_id in the response (mild cross-session labeling issue). Filter with WHERE id=?1 AND session_id=?2.
• MemoryRescueHook re-stores substantive user turns as AgentNote, contradicting the governance layer. memory_rescue.rs:138-150 — commit 34edc79 removed per-turn auto-store to stop agent-note flooding; this hook reintroduces it on the compaction path (any user msg ≥ 30 chars → AgentNote, conf 0.6).
• Contradictory dead parameter. context.rs:85-117 — build_system_prompt still accepts recalled_memory and injects a Memory section into the system prompt, but the live caller always passes None (memory now lives in the user message). The else if "continuing conversation" hint still fires on every multi-turn message. Remove the dead path or gate it.
• Anthropic SSE error events & JSON-parse failures silently swallowed. anthropic.rs:311-398 — the _ => {} arm drops mid-stream type:"error"; malformed data: lines continue silently. A mid-stream rate-limit/policy error looks identical to a clean short completion. Surface them as Err chunks.
• Committed runtime/IDE artifacts. .claude-flow/sessions/*.json (7 files) + .zcode/plans/plan-*.md — not gitignored; embed the dev's absolute home path; add noise to every diff. Add to .gitignore and git rm --cached.

---

🟡 Low / Nit

• sanitize_fts_query is a no-op despite its docstring (session_db.rs:524-538). Bound parameter ⇒ not SQLi, but FTS5 query grammar (", *, NEAR(, col:) still reaches MATCH → syntax errors surfaced as "session search failed". Implement what the doc promises (wrap as phrase or strip special chars).
• token_count = content.len()/4 uses byte length → ~3× mis-estimate for CJK (session_db.rs:249), the exact content the trigram tokenizer targets. Use chars().count() or drop if unused.
• BM25 fallback returns ≤1 memory despite limit(5) (loop_mod.rs:986-1002) — unconditional break after first fitting line; also drops the [category] tag and adds a spurious min_confidence(0.0).
• save_session_async silently drops snapshots when the 256-deep persist queue is full (manager.rs:156-175) — under burst load, compaction/new messages are lost on crash and never reach FTS. At least count/log as data-loss.
• Gateway logs "continuing without session_search" then hard-fails on SessionDb::new error (gateway.rs:388-398) — message contradicts behavior. Either actually continue, or call it a fatal error.
• Unused imports SessionEntry, MessageRole in session_db.rs:571-572 (warning on every test build).
• markdown_to_html doesn't escape " in href (telegram_format.rs:388-405) — attribute-breakout defense-in-depth gap; also validate URL scheme (javascript:/data:).
• daemonize doesn't close inherited high-numbered fds (daemonize.rs:46-95) — limited impact today (called pre-runtime), but a gap vs. the classic idiom.
• registry.rs:69 hard-codes enable_cache_control: true, ignoring config — non-Anthropic-compatible relays behind base_url will 400.
• No schema versioning (session_db.rs:118-179) — CREATE…IF NOT EXISTS with no PRAGMA user_version; fine until the first schema change, then a trap. Add PRAGMA user_version = 1 now.
• serde_json failure on tool_calls silently stored as "" not NULL (session_db.rs:242-245).
• SessionRescueHook::on_pre_compress always returns 0 (memory_rescue.rs:86-103) — masks rescue accounting in compaction logs.

---

Notes

• The IO-safety fix in daemonize (into_raw_fd + manual close) is correct — good fix.
• The memory-injection ordering (runner.rs:226-236, commit 9a03f3c) is correctly applied on an owned copy; the prepend-before-question logic is sound.
• A couple of findings were flagged then dismissed on adversarial verification but are worth a glance: the streaming path may still split raw text at ~4096 bytes before MarkdownV2 conversion (stream_consumer.rs, the bug commit 1ae2d27 removed elsewhere); and injecting memory into the last user message may erode the prompt-cache prefix the prior design relied on.

Happy to expand on any item or open follow-up issues.