Skip to content

docs: fix drift in Hybrid Search - #268

Draft
jack-arturo wants to merge 1 commit into
mainfrom
docs/audit-core-concepts-hybrid-search-20260728
Draft

docs: fix drift in Hybrid Search#268
jack-arturo wants to merge 1 commit into
mainfrom
docs/audit-core-concepts-hybrid-search-20260728

Conversation

@jack-arturo

Copy link
Copy Markdown
Member

Automated docs-accuracy audit of docs/core-concepts/hybrid-search. This page was flagged for re-verification after PR #202 was closed as superseded; this is that pass. Draft because the fixes rewrite explanatory prose and two mermaid diagrams.

Claim Current state Fix Evidence
context_bonus × SEARCH_WEIGHT_RELEVANCE, and the weights table describes SEARCH_WEIGHT_RELEVANCE as the "Context profile scoring bonus" Two different things were conflated. SEARCH_WEIGHT_RELEVANCE multiplies relevance_score — the consolidation-decay field on the memory. context_bonus is a separate term added with no env weight at all, using fixed sub-weights (tag 0.45, type 0.25, keyword 0.2, anchor 0.9). Split them in the formula, the Component Weights table, the env-var table, and the score-combination diagram automem@8ff266e:automem/utils/scoring.py#L251-L262, #L100-L134, #L215-L217
"Raw combined scores are normalized to the range [0.0, 1.0] during final ranking" (stated 3× in prose, plus a Normalize node in the flowchart) No normalization exists. _compute_metadata_score() returns the raw weighted sum and recall assigns it straight through (result["final_score"] = final_score), then sorts on it. Entity-expanded results even get final_score + 0.15, which can push past 1.0. Removed the claim and the diagram node; documented that scores are a ranking signal, not a probability scoring.py#L251-L262, automem/api/recall.py#L1696-L1699, #L1477
"Missing importance defaults to 0.5", "Missing confidence defaults to 0.7" Both fall back to 0.0 when absent or non-numeric — a missing field contributes nothing, it does not get a mid-range prior Corrected both bullets scoring.py#L163-L167
"Missing last_accessed falls back to timestamp"; recency "based on the time since last access (or creation if never accessed)" _compute_recency_score() is called with memory.get("timestamp") and never reads last_accessed Rewrote the recency section around timestamp scoring.py#L169, #L66-L81
Recency hardcoded as "linear decay over 180 days"; neither knob appears in the config reference The window and curve are both configurable — SEARCH_RECENCY_WINDOW_DAYS (180) and SEARCH_RECENCY_CURVE (linear|exp, where exp treats the window as a half-life) Documented the defaults as defaults and added both rows to the config table automem/config.py#L499-L503, scoring.py#L77-L81

Also refreshed all 9 source-note links from the short SHA 0720da2 to the full permalink 8ff266e6.

Verified against: verygoodplugins/automem@8ff266e

Checked and found correct, left untouched: all ten SEARCH_WEIGHT_* defaults (and the 1.95 sum), every RECALL_* default in the expansion/limit table, and the function names _vector_search(), _graph_trending_results(), fetch_relations(), _build_qdrant_tag_filter(), _compute_metadata_score(), _extract_entities_from_results().

Questions

  • The Component Weights table has always presented the weights as percentages ("35%", "20%"). Given they sum to 195% and aren't renormalized, would you rather show raw weights than percent signs? I left the formatting alone.

Unverified

  • The LongMemEval / LoCoMo benchmark numbers and the "Query Response Times" latency table are not code-verifiable from the checkout and were left as-is.

Follow-ups

Beyond the 5-fix cap:

  1. An eleventh weight exists. SEARCH_WEIGHT_TEMPORAL (default 0.1) is added by the recency re-rank, which is inert while RECALL_RECENCY_BIAS=off (the default) — so "10-component" is defensible at defaults but incomplete. Already tracked in docs: "9-component scoring" stale across multiple pages; new SEARCH_WEIGHT_TEMPORAL weight undocumented everywhere #241; not touched here.
  2. RECALL_RELEVANCE_GATE is undocumented on this page. When enabled it linearly ramps down importance, confidence, recency, tag, and relevance for results with weak topical evidence — a material scoring behavior with no mention in the scoring section.
  3. SEARCH_TAG_SCORE_TOKEN_CAP is undocumented. It caps the denominator of the tag-overlap score so long queries aren't penalized.
  4. Keyword score table (+2/+1/+2/+1) not re-derived in this pass.

Generated by Claude Code

Verified against automem@8ff266e6.

- SEARCH_WEIGHT_RELEVANCE weights consolidation relevance_score, not the
  context bonus; the context bonus is added unweighted
- final_score is never renormalized to [0.0, 1.0]
- missing importance/confidence contribute 0.0, not 0.5/0.7
- recency reads timestamp only, and the window/curve are configurable
- refreshed 9 stale source-note permalinks

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQCAKtAw2NRatRuYVEfDcv
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying automem-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 62f9cc9
Status: ✅  Deploy successful!
Preview URL: https://33fe591c.automem-website.pages.dev
Branch Preview URL: https://docs-audit-core-concepts-hyb-j9mw.automem-website.pages.dev

View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants