Repository navigation
v1.1: chunked retrieval index; cross-encoder rerank rejected by measurement - #1
Merged
Merged
Conversation
added 3 commits
July 2, 2026 17:29
all-MiniLM-L6-v2 truncates input at 256 tokens (~1,000 chars), so v1.0's
whole-narrative embeddings only covered the head of long complaints
(narratives run up to 2,000 chars) — the tail never influenced similarity.
chunk_text() splits narratives into ~800-char (~200-token) windows with
200-char (25%) overlap, breaking at whitespace. ingest_kb.py embeds
chunks with ids '{complaint_id}#{i}' and parent complaint_id in metadata;
the collection is renamed past_tickets_chunks so a stale v1.0 index can't
be silently queried. 1,200 docs -> 2,030 chunks.
…surement retrieve_similar() now vector-searches top-20 chunks, optionally reranks with a local cross-encoder (ms-marco-MiniLM-L-6-v2), and dedupes hits to parent complaints keeping each parent's best chunk — so the reply prompt sees the most relevant passage instead of the narrative head. Measured against the v1.0 baseline (recall@1 0.763, MRR 0.838): - chunked, no rerank: recall@1 0.763, recall@3 0.907, MRR 0.838, 23ms/query - chunked + rerank: recall@1 0.740, recall@3 0.900, MRR 0.825, 83ms/query Chunking holds the proxy metrics at parity while fixing the truncation defect — shipped. The reranker regressed recall@1 by 2.3 points (likely proxy-metric mismatch: it promotes semantically relevant chunks from neighboring categories that category-match relevance can't credit), so rerank_enabled defaults to False; implementation and tests are retained for re-evaluation against human-labeled relevance. Fail-soft contract unchanged: store failure returns [], rerank failure falls back to vector order — both pinned by tests.
Adds the measured v1.0 vs v1.1 retrieval comparison table and the ship/no-ship readout: chunked index shipped at metric parity (truncation defect fixed, +8ms/query), cross-encoder rerank rejected on measured recall@1 regression and kept behind a flag. Roadmap gains human-labeled relevance and hybrid BM25 as the next single-variable experiments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes a real embedding defect and reports an honest experiment readout.
Defect:
all-MiniLM-L6-v2truncates input at 256 tokens (~1,000 chars). v1.0 embedded whole narratives (200–2,000 chars), so the tail of long complaints never influenced similarity.Fix (shipped): chunk narratives into ~200-token windows with 25% overlap (1,200 docs → 2,030 chunks), retrieve chunks, dedupe to parent complaints keeping each parent's best-matching chunk — the reply prompt now sees the most relevant passage instead of the narrative head.
Experiment (rejected): cross-encoder reranking (
ms-marco-MiniLM-L-6-v2) of the top-20 chunk candidates.Measured (300-query retrieval eval, category-match proxy relevance)
Readout: chunking holds proxy metrics at parity while eliminating truncation and improving reply context — ship. The reranker regressed recall@1 by 2.3 points — most likely proxy-metric mismatch (it promotes semantically relevant chunks from neighboring categories, which category-match relevance can't credit) — but we gate ships on the measured number, so it lands off by default behind
rerank_enabled, with implementation + tests retained for re-evaluation against human-labeled relevance.Safety
[], rerank failure → vector order.past_tickets_chunksso a stale v1.0 index can't be silently queried.🤖 Generated with Claude Code