[ancsingabor]

Review of docs/guide/07-measurement.md.

1. The 📐 on bakeoff.py is correct — but your memory isn't wrong either

harness/apologia_eval/ contains exactly: candidates.py, db.py, gold.py, hashing.py, metrics.py, preflight.py. No bakeoff.py, no score.py. So 📐 is right on both.

What has run is the pre-flight, and eval/reports/preflight.json has the receipts — all three candidates exported to ONNX:

model	family	exported	rank agreement@3
intfloat/multilingual-e5-large	xlm-roberta	✅	0.6
BAAI/bge-m3	xlm-roberta	✅	0.7
Qwen/Qwen3-Embedding-0.6B	qwen3	✅	0.1

That's real work with real output, which is why it keeps reading as "we built it". It just isn't the bake-off.

2. The chapter never pointed at the two modules it talks about most

It spends eleven lines on servability and names npm run eval:preflight — then lists four of the six existing modules and omits preflight.py and candidates.py entirely. The slate is where each model's pretraining family, prefix convention and Serving value live; the pre-flight is the gate that admits a candidate. Without them the harness reads as four helpers waiting on a bake-off.

Both are now in the code table and in the diagram, with preflight.py --> bakeoff.py labelled "admits a candidate" so the ordering is visible.

3. Your instinct about the outcome table was right

Only one thing in that table is open: which row the numbers select. Everything else is decided — ADR-023 is accepted, the query-embedding service's shape is settled, and Vercel's Python 3.12 runtime is a checked fact (ADR-023 pins the toolchain to it deliberately). The table now says that above the rows, because pre-registering the interpretations is the point and a reader couldn't tell that from four "if… then…" lines.

4. Row 4 contradicted the prose underneath it

an open-weights model wins but cannot be served → there is no query path with it at all

That was added on 2026-09-15 as the row the table was missing — and then designed out. A candidate with no serving route is admitted to nothing. candidates.py is blunt about it:

It is one that cannot be used at all, and it must not be able to win.

So it's now struck through and labelled prevented, not awaited, with a note that Serving.NONE exists to record such a model rather than silently drop it. The prose below already said this; the table was arguing with it.

5. Dropped the State column

Third chapter to lose one — status outside status.md. The diagram marks what doesn't exist, and the chapter heading carries 🚧. Added eval/reports/ while there, which the table omitted despite preflight.json living in it.

6. Dated the truncation measurement

"About 80% of Summa chunks exceed 2,048 characters" is ADR-023's, measured 2026-09-10. Restating a measurement undated in a guide chapter is how it quietly becomes a standing claim — it would move the moment ARTICLE_BUDGET_CHARS did.

Verified and unchanged

• three retrieval metrics in metrics.py: recall_at_k, full_recall_at_k, reciprocal_rank
• ten gold questions, eight scorable → 1/8 = the 12.5 points the confidence-interval answer cites ✅
• "a Summa chunk covers about 7 units": 23,326 / 3,453 = 6.75 ✅
• full-recall@k description matches the implementation
• The pre-flight/bake-off TL;DR bullet added in docs(guide/05): say why superseded documents are kept, and clear three stale locator examples #26 reads correctly in context

Verification

• npm run docs:lint — clean
• Diagram re-rendered
• Chapter 07: 145 prose lines

🤖 Generated with Claude Code