Dead semantic tier is reported but never escalated — silent keyword-only degradation destroys recall (field report for #226 item 3)

[edheltzel]

Summary

A dead semantic tier (Ollama down → embeddings = 0 rows) degrades memory_hybrid_search to keyword-only. The degradation is reported at every layer — and is still invisible to the human, because nothing escalates it. The result is a memory system that confidently answers "No results found" for memories it holds, indefinitely, with no alarm.

This was found in the wild on a live install. It cost real trust in Recall.

Evidence (measured 2026-07-15, live install)

Check	Value
embeddings table	0 rows
vec_embeddings_rowids	0 rows
Ollama binary	installed (/opt/homebrew/bin/ollama)
Ollama service	DOWN (11434 refused)
recall doctor verdict	[WARN] — same severity as two false alarms

User-visible failure:

$ recall search "do you remember when we decided how to handle graphs"
No results found.

$ recall search "CodeGraph"
Found 20 result(s)

The matching record (decision #2892) exists and is important enough to be injected into L1 at every SessionStart. The memory was never lost — the retrieval path to it was dead.

Traced path: memory_search → 0 keyword hits → shouldFallbackToHybrid (src/lib/search-fallback.ts:44) → hybrid → no embeddings → empty → "No results found" (search-fallback.ts:93).

Root cause: this is an escalation failure, not a silence failure

The system tells the truth at every layer:

• CLI: Error: Embedding service not available at http://localhost:11434 / Falling back to keyword-only search...
• MCP: Found 1 results (keyword-only: embeddings unavailable; semantic backend: none)

The signal reaches the agent on every single search. The agent reads it, answers from degraded results, and never tells the human. The information dies in the agent's context. No prose in a guide fixes this — the guides already say "memory first" in four places.

Contributing defects

1. recall doctor has no FAIL tier, so the real signal is buried in noise. Three same-severity WARNs; two are false alarms:

• "CLAUDECODE env var set (will block extraction)" — factually wrong. Every spawn path already clears it: RecallExtract.ts:1034, extract-model.ts:129/:226, RecallInSession.ts:94, RecallClearExtract.ts:190. It flags the parent env, which is irrelevant.
• "messages stale 2.8d" — near-vestigial; structured extraction (decisions/LoA) flows daily.

A dead semantic tier and a bogus env warning cannot share a severity level.

2. The hybrid benchmark is an invalid instrument. The 23.5ms p50 @100k SLO (#217, #146) passes because the vec index is empty — it is fast because it searches nothing. A green benchmark over 0 rows is how this stayed invisible. The benchmark must fail on an empty index.

3. Negative relevance scores on the fallback path. src/lib/search-fallback.ts:63 and src/mcp-server.ts:531 do (r.score * 100).toFixed(1), but FTS5 bm25() returns negative scores (more negative = better). Renders as -1121.0%. This is on the fallback path — i.e. it only shows up while already degraded, compounding the trust loss.

4. Semantic dedup is silently non-functional in this state. dedup.ts:52 (DEFAULT_SEMANTIC_THRESHOLD = 0.95) reads embeddings, which are empty — and has no automatic caller.

Relationship to #226

#226 item 3 predicted this exact failure, verbatim:

recall doctor has no vec checks — extension loaded, index row count vs embeddings count, index format version. A silently-degraded vec tier is currently invisible outside stderr logs.

It sits unstarted in Phase 3 Todo. The failure mode was correctly diagnosed and filed, then deprioritized, then hit in production. This issue is the field report that should pull #226 item 3 forward.

Proposed fix

Detection

• Add vec checks to recall doctor (Harden vec index: zero-vector guard, index-format version stamp, doctor/stats vec observability #226 item 3): extension loaded, embeddings count, index rowid count vs embeddings count, index format version, embedding backend reachable
• Introduce a FAIL severity distinct from WARN. Dead/empty semantic tier = FAIL, non-zero exit
• Remove the CLAUDECODE false positive; downgrade or retire the messages staleness check

Escalation (the actual gap)

• A degraded tier must reach the human, not just the agent's context. Options: prefix the MCP response with an unmissable banner; have SessionStart surface a health line when a tier is down; make memory_hybrid_search return a top-level degraded: true the guide requires escalating
• Add an escalation rule to FOR_CLAUDE.md: if a tier reports degradation, surface it to the user that turn — never answer from degraded results silently

Instrument integrity

• Make the hybrid benchmark fail (not pass) on an empty vec index
• Fix -1121.0% — normalize BM25 or drop the % unit

Recovery UX

• When embeddings are empty/backend down, point at the fix in-product: ollama serve → recall embed backfill → recall embed stats

Acceptance criteria

1. With Ollama stopped, recall doctor exits non-zero with a FAIL naming the dead semantic tier — and no CLAUDECODE warning.
2. With embeddings = 0, the hybrid benchmark fails.
3. With the tier degraded, an agent using memory_hybrid_search receives a signal it cannot answer past without surfacing to the user.
4. No search output renders a negative percentage.

Notes

• Diagnosed via 3-agent RedTeam ParallelAnalysis. Full plan: .agents/atlas/plans/2026-07-15-recall-confidence-diagnosis-and-remediation.md (local, gitignored). Recall learning #720.
• Capture was never at fault — decisions 3004, LoA 910, learnings 717, breadcrumbs 4244, same-day writes. Retrieval was the whole defect.

Refs #226, #217, #146.

[edheltzel]

Root cause found — this is a shipped regression affecting every existing install, not a local misconfiguration

Deeper investigation changes this issue materially. Three compounding causes, in order.

1. The required model was never pulled — and the lifecycle scripts never pull it

src/lib/embeddings.ts:5 requires qwen3-embedding:0.6b (1024-dim, :6).

On-disk manifests (~/.ollama/models/manifests) contain exactly one model:

registry.ollama.ai/library/nomic-embed-text/latest

qwen3-embedding:0.6b is not installed. The default swapped nomic-embed-text/768 → qwen3-embedding:0.6b/1024 in a72fac2, 2026-06-22 (#107/#160) — 23 days ago.

rg "ollama pull|qwen3|embedding" install.sh update.sh lib/install-lib.sh returns nothing. No lifecycle script pulls the model. So on every existing install that updated past #160 and didn't manually ollama pull qwen3-embedding:0.6b, semantic search has been dead since 2026-06-22 — silently. This is not one user's bad setup; it is a migration that shipped without a migration step.

2. The daemon-down check masks the model-missing check

checkEmbeddingService (src/lib/embeddings.ts:127-139) returns at :131 when the daemon is unreachable — before the hasModel check at :137. So "Ollama not reachable" hides "and also the required model isn't installed." Start the daemon and you hit a second, entirely undiagnosed wall. Every error message points at the first cause only.

3. The Ed-locked OPTIONAL invariant makes total failure indistinguishable from opt-out

src/lib/embedding-marker.ts:64-68:

const total = (db.prepare('SELECT COUNT(*) AS c FROM embeddings').get()).c;
if (total === 0) return 'absent';   // "nothing to migrate"

Per the locked invariant at embedding-marker.ts:10-13:

🔒 Invariants (Ed-locked, #107):

• The local embedding model stays OPTIONAL. A DB with no embeddings reports 'absent' (nothing to migrate); search still works on FTS5. The marker never makes embeddings a hard requirement.

This is the deepest root cause. 0 embeddings is classified as 'absent' — a healthy, supported, intentional state meaning "nothing to migrate." The system cannot distinguish:

• opted out — user doesn't want embeddings, FTS5 is fine → absent, correct
• catastrophically broken — model missing, backfill never ran, user expects semantic recall → absent, wrong

Both produce byte-identical state. The invariant is defensible as a policy (embeddings should stay optional), but it was implemented as an inference from row count rather than a declared intent. That inference is what converted a total failure into a silent no-op that persisted 23 days.

The stale path — the one with the loud migration machinery (isRebackfillNeeded, recall embed rebackfill) — is unreachable when the table is empty. All the detection that exists for a partial failure is bypassed by a total one. The worse failure gets the quieter path.

Revised fix

The original checklist stands, but the framing changes: detection alone is insufficient — the system needs to know the user's intent.

• Add explicit intent config (e.g. RECALL_SEMANTIC=required|optional|off, or a schema_meta key set at install). absent + required = FAIL. absent + off = fine. Stop inferring intent from row count.
• Pull the model at install/update time, or fail loudly with the exact ollama pull qwen3-embedding:0.6b command. A model swap must ship with a migration path for existing installs.
• Reorder checkEmbeddingService so model-missing is reported even when the daemon is down (or report both, not the first).
• Backfill on a fresh/empty table should be reachable from the same machinery as rebackfill — a total failure must not get a quieter path than a partial one.
• Retro: add a lifecycle test asserting that the model named in embeddings.ts is pulled or explicitly declared optional. This class of bug (code requires a resource install never provisions) should be caught by CI, not by a user losing confidence for three weeks.

Corrected acceptance criteria

1. On a machine with Ollama down and the model missing, recall doctor reports both causes and exits non-zero.
2. With RECALL_SEMANTIC=required and 0 embeddings, doctor FAILs — it does not report "absent / nothing to migrate."
3. update.sh either pulls the required model or fails with the exact remediation command.
4. A test asserts EMBEDDING_MODEL is provisioned by the lifecycle scripts.

Note on severity

The original report framed this as observability. It is worse: a shipped model swap silently disabled semantic search for every existing install for 23 days, and three independent safety mechanisms (doctor, checkEmbeddingService, embeddingMarkerStatus) each individually failed to surface it — one by burying it among false alarms, one by short-circuiting, one by design invariant. Suggest re-labeling risk:high.