[vdavamse]

Summary

Follow-up to #36, which merged while this benchmark was still running.

Benchmarked all six GLiNER-BioMed variants on the target hardware. Two claims in research/linearrag-improvements.md were wrong and are corrected in place, with the measured table added. Results were also posted as a comment on #36.

Documentation only — one file, no source changes.

Method

120 real CTG chunks (mean 1,147 chars), GTX 1650 Ti fp16, batch 8, threshold 0.4, production call path (gliner_batched.py:249). "Agreement" = mean Jaccard of (text, label) entity sets against the production model over 40 docs — a cheap proxy, not F1.

Model	Labels	doc/s	vs prod	VRAM	CTG est	Agreement
uni-large (production)	16	1.13	1.00x	1,616 M	22.8 d	—
uni-large	6	1.22	1.08x	1,616 M	21.1 d	—
bi-large	16	1.43	1.27x	1,618 M	18.0 d	0.46
uni-base	16	3.95	3.50x	744 M	6.5 d	0.56
bi-base	16	3.40	3.02x	1,013 M	7.6 d	0.45
uni-small	16	6.35	5.63x	655 M	4.1 d	0.50
bi-small	16	4.84	4.29x	903 M	5.3 d	0.44

What was wrong

Item 10 recommended switching to the bi-encoder. It helps only at large (1.27x, below the paper's 39-63% band). At base and small it is slower than the uni-encoder (bi-base 0.86x of uni-base, bi-small 0.76x of uni-small) and uses more VRAM, consistent with carrying a second encoder. The paper's gains come from amortising cached label embeddings across many labels; at 16 labels and ~1,147-char chunks, text encoding dominates and the extra encoder is overhead. Their figures were RTX3090 / fp32 / "dataset-specific labels" — a different regime.

Item 11 claimed a superlinear saving from cutting labels. 16 -> 6 gave 1.08x. The quadratic term in (|labels| + |text|) is real but irrelevant at this chunk size, where |text| dominates. The paper's 92-568% figures were at 127 labels. Item 11 now rests on gazetteer precision, not speed, and drops down the priority list.

What replaces them

Model scale is the lever: uni-small 5.63x (22.8 d -> 4.1 d), uni-base 3.50x (-> 6.5 d). That is the 3-6x originally hoped for, from a different knob.

⚠️ But the speedup is not free. Agreement with production is only 0.44-0.56 — even the closest variant shares barely half its entity set with the current model. These are materially different extractions, not cheaper approximations. Bi-encoder variants cluster low (0.44-0.46) at every scale, suggesting the architecture extracts systematically differently.

Why this matters for sequencing

Item 10 becomes the one decidable item on the list today, and moves up the suggested order accordingly. Throughput is now measured, and unlike the retrieval question, the harness already exists — src/ctra/rag/eval/ (NERExperiment, CHIA/BioNLP benchmarks, threshold sweeps) is exactly the tool for adjudicating uni-base / uni-small on F1. A 3-6x faster index would also make every later experiment cheaper.

Scripts and raw JSON are in notes/bench_rag_hardware/ (untracked — notes/ is gitignored).

🤖 Generated with Claude Code