[yodablocks]

What

Every earlier Jev result (#28, #30) came from a response cache that returns one answer per (question, passage) pair, so Jev's run-to-run variation was never measured. experiments/jev_rerank/repeat.py scores the same pairs three times (pass 0 from the cache, passes 1 and 2 with the cache bypassed), using the union pool of 10 and wording v1 over the 38 gold and 15 hard questions. Cloud calls only; about $0.04. The bands were committed before the run.

Also a README correction (separate commit): the Experiments table said Jev's Hit@1 was 38/38 and 36/38, which I wrote from single cached runs.

Result

Jev is not deterministic.

• 281 of 490 pairs (57%) returned identical scores in all three passes; mean spread 0.010; 24 pairs (5%) spread by more than 0.05; one pair by 0.25.
• Hit@1 per pass: gold38 38, 38, 37; hard15 12, 12, 12.
• Two gold questions (b1-005, y-08) changed their top-1 page in at least one pass; none of the hard ones did.

The bands left a gap for "Hit@1 differs by exactly 1", which is what happened: not Stable, not Unstable. Plain reading: mild run-to-run variation, and the 38/38 should be read as 37 to 38. The direction of the earlier comparisons still holds: bge scores 33/38 on the embedding pool and on the union pool, and Jev has not been seen below 37 on the 38.

Caveats

• Three passes is a small sample of the variation.
• The script keeps only which questions changed top-1, not the per-pass top-1 pages, so which question lost the gold page in pass 2 is not recorded.
• Not tested: the other wordings (v2, v3), the embedding pool, more passes.
• 490 distinct pairs rather than 530: two hard questions repeat gold questions (same question and pool).

Changes outside experiments/

README: two table cells reworded to say Jev's results move by about one question between runs, and that 38/38 held in two of three passes.