Skip to content

fix: provision embedding model in lifecycle + stop rendering RRF scores as percentages (#240)#242

Merged
edheltzel merged 3 commits into
mainfrom
worktree-fix-embedding-regressions
Jul 15, 2026
Merged

fix: provision embedding model in lifecycle + stop rendering RRF scores as percentages (#240)#242
edheltzel merged 3 commits into
mainfrom
worktree-fix-embedding-regressions

Conversation

@edheltzel

Copy link
Copy Markdown
Owner

Fixes the root cause of #240 and the score-display bug found alongside it.

Found while diagnosing a 23-day silent semantic-search outage on a live install: embeddings = 0 rows, hybrid search degraded to keyword-only, and a natural-language query returned No results found for a memory that was sitting in L1 context at every session start.

1. Model provisioning — the root cause (#240)

The default embedding model swapped nomic-embed-text/768 → qwen3-embedding:0.6b/1024 in a72fac2 (#107/#160, 2026-06-22). No lifecycle script ever pulled it.

$ rg "ollama pull|qwen3|embedding" install.sh update.sh lib/install-lib.sh
(nothing)

Every install that updated past #160 without manually running ollama pull silently lost semantic search. Not a local misconfiguration — a migration that shipped without a migration step.

Fix: recall_provision_embedding_model() in lib/install-lib.sh, called from install.sh (self-check) and update.sh (step_verify) — update.sh being the path that actually regressed.

Two constraints shaped it:

  • The model name is read from src/lib/embeddings.ts, never hardcoded in bash. A duplicated constant is exactly how the last swap drifted; hardcoding here would rebuild this bug for the next swap. Verified: no model name appears in any lifecycle script outside a historical comment.
  • Respects the 🔒 Ed-locked OPTIONAL invariant (src/lib/embedding-marker.ts:10-13). No Ollama → informational, exit 0, never gates the install. Both paths exercised:
# ollama up + model present
✓ Embedding model present: qwen3-embedding:0.6b            exit=0

# ollama absent
→ Ollama not installed — semantic search stays disabled (optional; keyword search is unaffected)
→   To enable later: install Ollama, then 'ollama pull qwen3-embedding:0.6b' and 'recall embed backfill'
                                                            exit=0

2. The score field was polymorphic

score carried raw FTS5 bm25 rank on the FTS-only path (negative, unbounded) and RRF on the fused path (0..~0.033). One formatter rendered both as (score * 100).toFixed(1) + '%'.

Observed output: -1121.0% while degraded, then a meaningless 1.6% against a 3.3% ceiling once semantic recovered.

  • mcp-server.ts — run the FTS-only list through reciprocalRankFusion so score has one unit everywhere. Same ordering, honest unit.
  • search-fallback.ts — render rrf=0.0328, not 3.3%. RRF is a relative ranking signal, not a confidence; the % implied a meaning it never had.
  • mcp-server.ts reused formatHybridResults instead of its hand-copied duplicate — the bug existed in two places and had to be fixed twice. One definition now (DRY law).

Tests: fixture score: 0.830.0328 (0.83 was never a value RRF could produce — the fixture encoded the misconception). New regression test asserts no % is ever rendered and a negative score cannot print -1121.0.

Verification

End-to-end on a live install: model pulled → 14,616 embeddings backfilled → vec index rebuilt → semantic backend: knn. The query that returned No results found now returns the decision it was looking for.

before after
MCP hybrid keyword-only: embeddings unavailable; semantic backend: none hybrid: FTS5 + embeddings; semantic backend: knn
embeddings / vec index 0 / 0 14,616 / 14,616

bun run lint clean. Full suite: 1236 pass.

On the 10 failures: all are recall_configure_opencode_mcp / recall_configure_pi_mcp and are a pre-existing worktree artifact, not from this PR. Proven by stashing every change and re-running on pristine main code inside the same worktree — identical 5 failures per file. Clean main passes 1245/0. Worth its own issue: AGENTS.md mandates workers run in a worktree, so every worker currently sees these 10 spurious failures.

Not included

Refs #240, #241, #226, #234, #107, #160.

…es as percentages (#240)

Two regressions found while diagnosing a 23-day silent semantic-search outage.

1. Model provisioning (the root cause of #240)

The default embedding model swapped nomic-embed-text/768 -> qwen3-embedding:0.6b/1024
in a72fac2 (#107/#160, 2026-06-22), but no lifecycle script ever pulled it. Every
install that updated past #160 silently lost semantic search: hybrid degraded to
keyword-only, embeddings stayed at 0, and no surface reported a failure.

Adds recall_provision_embedding_model() to lib/install-lib.sh, called from
install.sh (self-check) and update.sh (step_verify) -- update.sh being the path
that actually regressed.

The model name is READ FROM src/lib/embeddings.ts, never hardcoded in bash. A
duplicated constant is precisely how the last swap drifted out of sync; hardcoding
it here would rebuild the same bug for the next swap.

Respects the Ed-locked OPTIONAL invariant (src/lib/embedding-marker.ts:10-13):
no Ollama -> informational only, exit 0, never gates the install.

2. Score unit bug

The `score` field was polymorphic: raw FTS5 bm25 rank on the FTS-only path
(negative, unbounded) and RRF on the fused path (0..~0.033). One formatter rendered
both as `(score * 100).toFixed(1) + '%'`, producing an observed "-1121.0%" -- and,
once semantic recovered, a meaningless "1.6%" against a 3.3% ceiling.

- mcp-server.ts: run the FTS-only list through reciprocalRankFusion so `score`
  carries one unit everywhere. Same ordering, honest unit.
- search-fallback.ts: render `rrf=0.0328`, not `3.3%`. RRF is a relative ranking
  signal, not a confidence.
- mcp-server.ts reused formatHybridResults instead of its hand-copied duplicate,
  so the bug existed in two places and had to be fixed twice (DRY law).

Tests: fixture score 0.83 -> 0.0328 (0.83 was never reachable); regression test
asserts no '%' is ever rendered and a negative score cannot print "-1121.0".

Verified end-to-end on a live install: model pulled, 14,616 embeddings backfilled,
vec index rebuilt, semantic backend knn, and the query that returned "No results
found" now returns the decision it was looking for.

Refs #240, #241, #226, #107, #160.
* origin/main:
  claude.md is a symlink to agents.md
…g-regressions

* origin/main:
  fix(ci): align the CLAUDE.md contract with the symlink, unbreaking main (#244)
@edheltzel
edheltzel merged commit c5708e3 into main Jul 15, 2026
2 checks passed
@edheltzel
edheltzel deleted the worktree-fix-embedding-regressions branch July 15, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant