test(aryabhatta): let the calibration corpus be somewhere else - #79
Open
adityak74 wants to merge 1 commit into
Open
test(aryabhatta): let the calibration corpus be somewhere else#79adityak74 wants to merge 1 commit into
adityak74 wants to merge 1 commit into
Conversation
Every run of this harness so far has ended in NotEnoughEvidence at n=21, required 50, and rerunning cannot fix that. This repo has 22 directories with two or more .rs files in them, and the whole aviskaar monorepo has the same 22 because nothing else in it is Rust. The corpus was the binding constraint on the measurement, not the model and not the budget. ZORP_CAL_ROOT points the corpus at any directory. The cargo registry is the obvious one to reach for: 3,387 eligible directories and 23,736 files of real third-party Rust. It is also a better corpus than our own code for the same reason a held-out set is better than a training one, though it is worth being clear that this does not make it unseen by the model, only unseen by us. The sampling changed with it, and that part is not cosmetic. Paths are sorted, so taking the first n from 3,387 directories takes an alphabetical prefix, which on a registry checkout is one letter's worth of crates. Striding spans the corpus instead, stays deterministic so the same machine and any other machine draw the same sample, and prints what it sampled from rather than quietly capping. Smoke tested against the registry with stealth/ox-alpha at n=3: adler2, jiff and rkyv, 442, 6344 and 293 lines, three exact hits and nothing unusable. Three different crates at three orders of magnitude, which is the spread the striding is for. Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every run of the evidence-gathering harness so far has ended the same way:
Rerunning cannot fix that. This repo has 22 directories with two or more
.rsfiles in them, and the whole aviskaar monorepo has the same 22, because nothing else in it is Rust. The corpus was the binding constraint on the measurement — not the model, not the step budget, not the time.ZORP_CAL_ROOTPoints the corpus at any directory. The cargo registry is the obvious one:
Real third-party Rust, diverse in size and style. It is also a better corpus than our own code for the same reason a held-out set beats a training one — though it is worth being precise: this makes the corpus unseen by us, not unseen by the model. Published crates are exactly the kind of thing a model is likely to have read.
The sampling change is not cosmetic
Paths are sorted, so
truncate(want)on 3,387 directories takes an alphabetical prefix — one letter's worth of crates, all of them structurally similar because they came from the same corner of the registry. On the old 22-directory corpus this was invisible; at this scale it would have quietly made the sample unrepresentative while the numbers looked fine.Striding spans the corpus instead, stays deterministic so any machine draws the same sample, and prints what it sampled from rather than silently capping.
Smoke test
Against the registry with
stealth/ox-alpha,n=3:Three crates from three different letters at three orders of magnitude — which is what the striding is for. Three exact point estimates, nothing unusable.
What this unblocks
n=60is now reachable, so the report can return an actual GO/NO-GO instead ofNotEnoughEvidence. That matters for a specific finding: the 22-directory run showed stated confidence anti-correlated with coverage above 0.93 (1.00 observed below it, 0.67–0.78 at and above). That result currently rests on bands of 1–9 samples each, and only one of them carries real weight. It needs a corpus this size before it is worth believing.Test-only change,
#[ignore]d as before.cargo test --workspaceis unaffected.https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4