Skip to content

feat(experiments): bge on the union pool, the local alternative to Jev - #31

Merged
yodablocks merged 2 commits into
masterfrom
exp/local-rerank
Oct 4, 2026
Merged

yodablocks merged 2 commits into
masterfrom
exp/local-rerank

Conversation

@yodablocks

Copy link
Copy Markdown
Owner

What

A privacy follow-up to #28 and #30. Jev is a cloud API, so quarq's local-handling promise does not hold for that arm. The production re-ranker bge-reranker-v2-m3 is already local. This experiment asks whether the union candidate pool from #30 lifts bge the way it lifts Jev, which decides whether a fully local path can get close.

Everything is under experiments/local_rerank/. Nothing in quarq/ changes. The bands were committed before the run.

Result (one run, 3 s pause between questions)

Set Re-ranker + pool Hit@1 Hit@5 MRR Query p50
38 gold bge, embedding pool (#28) 33 36 0.901 2,079 ms
38 gold bge, union pool 33 38 0.926 2,099 ms
38 gold Jev, union pool (#30) 38 38 1.000 862 ms
15 hard bge, union pool 10 12 0.733 2,049 ms
15 hard Jev, union pool 10 (#30 runs) 12 n/a n/a about 850 ms
  • Band: no lift at Hit@1 (33, the lift threshold was 35).
  • The union puts the gold page in bge's pool (Hit@5 36 to 38), but bge does not rank it first. Its five Hit@1 misses on the 38 are the same as before, including b1-020 and b2-011, the two the union added.
  • On the hard set bge misses h-001 and h-002 at rank 1; Jev gets both. h-009, h-011 and h-015 are in no pool at 10, so no re-ranker could fix those.

Caveats

  • One run on small sets that have been iterated on. The hard questions are LLM-drafted (synthetic-draft) and not human-reviewed.
  • Latency compares a local GPU with a network call, so it is only a rough figure. It excludes the 3 s pauses.
  • The Jev hard-set figure combines runs from feat(experiments): bring the recall experiment (#29) onto master #30 (h-001..h-011 and the French rewrites h-012..h-014). h-015 was not run at pool 10 because its gold page is not in the pool, so Jev's 12 of 15 is not a single run.
  • Hypothesis, not tested: the Jev question asks whether the passage states the answer, while bge scores topical relevance, so the wording may matter as much as the model.
  • Not covered: other local re-rankers (needs a model download), and a local LLM as a pointwise judge (needs LM Studio running).

Reproduce

python3 experiments/local_rerank/run.py   # loads bge once, about 4 minutes, paced

Query embeddings and the BM25 index come from the on-disk caches built by experiments/recall/probe.py.

@yodablocks
yodablocks merged commit fb52e11 into master Oct 4, 2026
4 checks passed
@yodablocks
yodablocks deleted the exp/local-rerank branch October 4, 2026 05:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant