feat: golden eval sets, store benches, batched rescore, greedy HyDE - #1
Merged
Merged
Conversation
- Expectations are written from docs and identifiers, never from search output, so the sets cannot tune the ranker to itself - --runs N prints per-case ranks side by side plus the recall/MRR spread, the noise floor any ranking change must clear - Baselines reset: these sets are not comparable to earlier numbers
- dense, lexical, and fused search were unmeasured; the gate now covers them at 2k and 16k chunks so E1/E2 changes cannot regress silently - refiner_curve sweeps coarse_k against exact KNN on a copy of a live index, excluding each query's own chunk from both lists
- The float rescore ran one point lookup per candidate (400 per query) and was slower than exact search at 22k chunks: 187ms vs 64ms - vec0 accepts a rowid IN set on a KNN query, so the rescore is now a single vector-native query: 54ms at coarse_k 400, recall 0.995 - hydrate_chunks batches the same way and keeps candidate order
- synchronous=NORMAL under WAL trades the last transactions on power loss for fewer fsyncs; the index is rebuildable from the repos - FTS optimize costs 277ms on 22k chunks and VACUUM 1.2s, so they run only at graceful shutdown and after a prune, never per sync - Model/dim guard now runs before migrate, so a mismatched file is refused untouched
- Sampling made every HyDE eval a single draw; at temperature 0 the 2B auto arm repeats bit-for-bit (scry 0.746/0.515 x3, finance-query 0.700/0.556-0.564, soothfast 0.650-0.667/0.395-0.412, p50 1.8-2.4s) - The prompt asks for a short snippet; 120 tokens keeps p95 under the 6s budget on a CPU-bound 2B generator where 220 did not
Verdenroz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The ranking pipeline had no checked-in golden set and no benchmark below the chunker, so nothing measured whether a search change helped. This adds eval sets for three repos, a multi-run eval report that prints the noise floor, and soothfast coverage of dense, lexical, and fused search. Two things fell out of measuring: the coarse rescore was slower than exact search at 22k chunks, and HyDE was sampling, which made every HyDE number a single draw.
Changes
eval/scry.toml,eval/finance-query.toml, andeval/soothfast.toml, written from docs and identifiers rather than search output.--runs Nand--limit Ntoscry eval, with per-case ranks side by side and the recall/MRR spread across runs.bench_dense_search,bench_lexical_search, andbench_search_with_vectorat 2k and 16k chunks, plusexamples/refiner_curve.rsfor coarse-pass recall against exact KNN on a copy of a live index.INset instead of one point lookup per candidate.hydrate_chunksbatches the same way and errors on a missing id.temperatureto 0 on chat completions and capped HyDE at 120 tokens.synchronous=NORMAL,busy_timeout,cache_size, andmmap_sizepragmas, FTS optimize andPRAGMA optimizeat graceful shutdown, andVACUUMafter prune.guard_metanow runs beforemigrate.Testing
cargo test --workspace: 57 passed, 0 failed.cargo clippy --workspace --all-targets -- -D warningsclean.hyde = auto, three runs: scry 0.746 / 0.515 with zero spread, finance-query 0.700 / 0.556-0.564, soothfast 0.650-0.667 / 0.395-0.412, p50 1.8-2.4s.gate.ymlwill report the three new benches as new rather than regressed, since they have no prior baseline.