Skip to content

test(aryabhatta): let the calibration corpus be somewhere else - #79

Open
adityak74 wants to merge 1 commit into
mainfrom
test/calibration-corpus-root
Open

test(aryabhatta): let the calibration corpus be somewhere else#79
adityak74 wants to merge 1 commit into
mainfrom
test/calibration-corpus-root

Conversation

@adityak74

Copy link
Copy Markdown
Contributor

Every run of the evidence-gathering harness so far has ended the same way:

NotEnoughEvidence { n: 21, required: 50 }

Rerunning cannot fix that. This repo has 22 directories with two or more .rs files in them, and the whole aviskaar monorepo has the same 22, because nothing else in it is Rust. The corpus was the binding constraint on the measurement — not the model, not the step budget, not the time.

ZORP_CAL_ROOT

Points the corpus at any directory. The cargo registry is the obvious one:

3,387 eligible directories
23,736 .rs files

Real third-party Rust, diverse in size and style. It is also a better corpus than our own code for the same reason a held-out set beats a training one — though it is worth being precise: this makes the corpus unseen by us, not unseen by the model. Published crates are exactly the kind of thing a model is likely to have read.

The sampling change is not cosmetic

Paths are sorted, so truncate(want) on 3,387 directories takes an alphabetical prefix — one letter's worth of crates, all of them structurally similar because they came from the same corner of the registry. On the old 22-directory corpus this was invisible; at this scale it would have quietly made the sample unrepresentative while the numbers looked fine.

Striding spans the corpus instead, stays deterministic so any machine draws the same sample, and prints what it sampled from rather than silently capping.

Smoke test

Against the registry with stealth/ox-alpha, n=3:

corpus: 3387 eligible directories, sampling every 1129th for 3

adler2-2.0.1/src                       said  442 in [ 441,  443] @0.98  truth  442  hit
jiff-0.2.35/src/fmt/friendly           said 6344 in [6340, 6348] @0.96  truth 6344  hit
rkyv-0.8.18/src/validation/shared      said  293 in [ 291,  295] @0.85  truth  293  hit

unusable 0
n=3 covered=3 coverage=Some(1.0)

Three crates from three different letters at three orders of magnitude — which is what the striding is for. Three exact point estimates, nothing unusable.

What this unblocks

n=60 is now reachable, so the report can return an actual GO/NO-GO instead of NotEnoughEvidence. That matters for a specific finding: the 22-directory run showed stated confidence anti-correlated with coverage above 0.93 (1.00 observed below it, 0.67–0.78 at and above). That result currently rests on bands of 1–9 samples each, and only one of them carries real weight. It needs a corpus this size before it is worth believing.

Test-only change, #[ignore]d as before. cargo test --workspace is unaffected.

https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4

Every run of this harness so far has ended in NotEnoughEvidence at n=21,
required 50, and rerunning cannot fix that. This repo has 22 directories
with two or more .rs files in them, and the whole aviskaar monorepo has
the same 22 because nothing else in it is Rust. The corpus was the
binding constraint on the measurement, not the model and not the budget.

ZORP_CAL_ROOT points the corpus at any directory. The cargo registry is
the obvious one to reach for: 3,387 eligible directories and 23,736
files of real third-party Rust. It is also a better corpus than our own
code for the same reason a held-out set is better than a training one,
though it is worth being clear that this does not make it unseen by the
model, only unseen by us.

The sampling changed with it, and that part is not cosmetic. Paths are
sorted, so taking the first n from 3,387 directories takes an
alphabetical prefix, which on a registry checkout is one letter's worth
of crates. Striding spans the corpus instead, stays deterministic so the
same machine and any other machine draw the same sample, and prints what
it sampled from rather than quietly capping.

Smoke tested against the registry with stealth/ox-alpha at n=3: adler2,
jiff and rkyv, 442, 6344 and 293 lines, three exact hits and nothing
unusable. Three different crates at three orders of magnitude, which is
the spread the striding is for.

Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant