Skip to content

feat: rerank stage fused as a weighted RRF leg - #4

Merged
Verdenroz merged 4 commits into
mainfrom
feat/rerank
Sep 5, 2026
Merged

Verdenroz merged 4 commits into
mainfrom
feat/rerank

Conversation

@Verdenroz

@Verdenroz Verdenroz commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Description

The fused top candidates already contain most of the misses: on the golden sets, 8 to 12 targets per set sat at ranks 11 to 44. A cross-encoder can see them, but replacing the order with the reranker's lost as many correct top-10 hits as it gained, so the reranker joins the fusion instead as a third reciprocal-rank leg. It is optional config, off when absent, and the search never gets worse when the reranker is slow or broken.

Changes

  • Added an optional [rerank] section: base_url, api_key (optional, env: indirection), model, top_n (default 20), weight (default 1.0).
  • Added scry-core/src/rerank.rs: a client for a Jina-style POST {base_url}/rerank and fuse(), which adds the reranker's ranking as a weighted RRF leg over the pool search_with_vector returns, sharing RRF_K with the other legs. Repeated or out-of-range indices are rejected.
  • The search route asks for max(limit, top_n) candidates, reranks after the store call, and falls back to the fused order on any error or after six seconds, logging at warn. Scores stay the dense cosine.
  • Added rerank to SearchRequest (default true) and --no-rerank to scry search and scry eval.
  • Added eval/simulate_rerank.py, which scores replacement, fusion weights, and rerank pool sizes offline from a dump of one run.
  • Documented the section in docs/search.md and deploy/config.example.toml. The example keeps [rerank] commented out, so nothing changes for a user who does not add the section.

Testing

  • cargo test --workspace: 46 core tests plus the integration suite pass; cargo clippy --workspace --all-targets -- -D warnings clean. New tests: config parsing, fuse() promotion and zero-weight identity, and an integration test with a mock /v1/rerank endpoint checking the fused order and the rerank: false bypass.
  • Offline simulation from one dump (bge-reranker-v2-m3 over the top 50, CPU build, copy of the live index at 22,604 chunks), summed recall@10 / MRR over three sets: no rerank 1.934 / 1.290, replacement 1.954 / 1.234, RRF weight 1 top 20 2.064 / 1.383. A natural-language gate lowered MRR everywhere and was dropped.
  • Confirmation with this branch's binary and the 6s budget, bge served on the Radeon 890M, serial runs on the same copy, --no-rerank pair per set: scry 0.667 / 0.426 to 0.730 / 0.444, finance-query 0.700 / 0.556 to 0.717 / 0.575, soothfast 0.667 / 0.401 to 0.700 / 0.436. Zero fallbacks in 183 reranked queries.
  • The live gain (+0.213 / +0.165 summed) is larger than the simulator's prediction (+0.130 / +0.093). Same direction on every set; the dump's scores came from the CPU build and the confirmation from the Vulkan build, and backend numerics reorder near-tied pairs. The simulator is a direction tool, not ground truth.
  • p50 3.2-4.0s and p95 5.9-7.5s were measured while a CPU dump saturated the machine and include HyDE and embedding, so they are ceilings. On a quiet box the rerank call alone takes 1.85s for 20 documents on the iGPU and 9.5s on CPU, where it would fall back.
  • Documents truncated to 1200 characters keep most of the gain at half the reranker cost; no knob was added since the iGPU fits full-length documents in budget.
  • CI is check-only for this PR: gate.yml has no rerank bench and the route is async.

- Fused top_n candidates go to a Jina-style POST /rerank and come back
  in the reranker's order with its relevance as the score; absent
  config means no call, and --no-rerank skips it per request
- gate stays a config knob (none, natural-language) until measured
- Offline simulation from one dump run (bge-reranker-v2-m3, 20-50
  candidates, index-e6c.db copy): replacing the order lost 8 scry hits
  while gaining 4 on finance-query; fusing the reranker's ranking as a
  third RRF leg at weight 1 over the top 20 gains on every set
- Summed over the three sets, recall@10 1.934 -> 2.064 and MRR
  1.290 -> 1.383; the natural-language gate only lowered MRR, so it is
  gone and top_n defaults to 20, which also halves the CPU cost
- fuse() reads the same k as the dense and lexical legs so the three
  cannot drift; a response naming an index twice is an error instead
  of a double-counted vote
- The simulator drops the rejected natural-language gate and its copy
  of the routing rule
@Verdenroz
Verdenroz merged commit 7785af8 into main Sep 5, 2026
2 checks passed
@Verdenroz
Verdenroz deleted the feat/rerank branch September 5, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant