Skip to content

Fix review findings: per-arm Jev counters and corrected experiment docs - #37

Merged
yodablocks merged 3 commits into
masterfrom
fix/review-findings
Oct 5, 2026
Merged

yodablocks merged 3 commits into
masterfrom
fix/review-findings

Conversation

@yodablocks

Copy link
Copy Markdown
Owner

Fixes from a two-axis code review (standards and spec) of PRs #28 to #36. One code fix and documentation corrections. The standards findings about code style in experiments/ (type hints, docstrings, magic numbers, model names) are not fixed here: they turn on whether experiments/ should be exempt from the repo rules, which is the owner's decision.

Code

  • experiments/jev_rerank/run.py: one Jev client was shared across arms, so a later arm's fresh_calls, cost_usd and latency percentiles included the earlier arms. Each Jev arm now gets its own client. The affected results file (run-20261004T033321.json, the jev:v3 fields) is left as recorded and annotated in the experiment README; the per-wording figures quoted there were already right.

Docs

  • Model-name rule stated backwards. experiments/jev_rerank/README.md said the rule "bans model names inside rag/ and llm/". CLAUDE.md says "No hardcoded model names outside config.py and llm/ backends", so jev-latest and google/gemma-4-12b-qat in experiments/ break it. The README now says so plainly.
  • Jev's variation attached to the right number. The top-level table put the one-question run-to-run variation next to 36/38, an embedding-pool single run; the repeat was on the union pool. Reworded.
  • "About 5 questions behind Jev" is now "4 to 5", because Jev scored 38, 38 and 37 in three passes. Also in the local re-ranker README, where the 862 ms Jev latency is now labelled as measured on the embedding pool.
  • Hard-set caveats carried forward. Only h-003 to h-011 were checked against the PDFs; the French rewrites only through their originals; and h-001 and h-002 are the two gold questions that motivated the union merge, so the hard set is not independent of it.
  • Open items: the source finding now says "none of the sources checked (ECB, Banque de France, AMF)" instead of a blanket claim, and a sixth item records that the planned ingest-time tagging experiment was never run (and that "experiment 2" and "3" were reused as names).
  • Stale line: the local re-ranker README said LM Studio was "not on this host"; it is the same Mac.
  • The repeat-run call count (1,060) is now said to come from the run's printed output, since repeat.json does not store it.

No experiment was re-run. pytest is unaffected: nothing under quarq/ or tests/ changed.

@yodablocks
yodablocks merged commit d196449 into master Oct 5, 2026
4 checks passed
@yodablocks
yodablocks deleted the fix/review-findings branch October 5, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant