Add keyword/substring search mode alongside semantic search

[imonroe]

Summary

Adds an opt-in keyword/substring search mode alongside the existing semantic search (backlog issue #54). Semantic search ranks by meaning and can miss an exact term — a name, identifier, URL, or rare token — so mode: "keyword" does a case-insensitive substring match over memory text, returning the most recent matches first. The default stays "semantic", so nothing changes unless you ask for it.

Design / why no new Qdrant validation was needed

This reuses the exact vector_store.list() + payload path already validated against the live Qdrant for the dedup feature (#48) — the data field and user_id filtering are confirmed to work. The substring match runs in Python over the user's memories, so it needs no full-text index, no schema change, no migration on the live collection. It's:

• Fail-open — any store error returns {"results": []}.
• Bounded — scans up to a generous limit (5000) of the user's memories per query; ample for a personal store. Documented as a literal-match fallback, not a replacement for semantic retrieval (the tradeoff vs. a Qdrant full-text index is called out in the docs).
• Scoped by user_id — spans the whole user store, consistent with the shared-pool model.

Surface

• REST POST /api/v1/memories/search: new mode: "semantic" | "keyword" (validated; an unknown value → 422). recency_weight applies to semantic mode only.
• MCP search_memories: new mode arg, with a docstring that tells the model to use keyword for exact terms.

Files

• app/memory.py — keyword_search() + _point_to_result() (shapes a Qdrant point into a result, dropping internal text_lemmatized/data plumbing).
• app/rest.py, app/mcp_server.py — route the keyword path.

Tests

• tests/test_memory.py — keyword_search unit cases: case-insensitive substring + user_id scoping, newest-first ordering + limit, result field shaping (keeps agent_id/created_at, drops internals), no-match, and fail-open on store error.
• tests/test_rest.py / tests/test_mcp.py — integration: keyword mode bypasses vector search and uses the listing; invalid mode → 422; MCP exposes the mode arg.
• Full suite: 157 passed, ruff clean.

Docs

• User Guide: "Keyword search (optional)" in the search section, with the scan-limit caveat.
• Developer Guide: memory.py description.

Optional post-deploy check

After deploy: -d '{"query":"Philips Hue","mode":"keyword"}' against /api/v1/memories/search should return memories literally containing that phrase, even ones semantic search ranks low.

Closes #54.

https://claude.ai/code/session_017835DVrvURaYnbQiPQwzue

---

Generated by Claude Code

[Copilot]

Pull request overview

Adds an opt-in keyword/substring search path alongside existing semantic search so callers can find memories by exact/literal terms (names, IDs, URLs) that embeddings can miss, without requiring Qdrant schema/index changes.

Changes:

• Added mode: "semantic" | "keyword" to REST search and routed keyword mode to a new in-process substring scan over vector_store.list().
• Exposed keyword mode via the MCP search_memories tool.
• Added unit + integration tests and documented the new mode and its scan-limit tradeoffs.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
app/memory.py	Implements keyword_search() and point-to-result shaping for keyword mode.
app/rest.py	Adds mode to the REST search request model and routes keyword searches.
app/mcp_server.py	Exposes mode on the MCP search_memories tool and routes keyword searches.
tests/test_memory.py	Unit tests for keyword_search() behavior (matching, ordering, shaping, fail-open).
tests/test_rest.py	REST integration tests for keyword mode + invalid mode validation.
tests/test_mcp.py	MCP integration tests ensuring keyword mode uses listing and exposes mode.
docs/USER_GUIDE.md	Documents keyword mode usage and caveats.
docs/DEVELOPER_GUIDE.md	Updates internal architecture notes to include keyword search.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.