test(aryabhatta): account for every attempt the calibration run samples - #88
Merged
Merged
Conversation
The 60-directory registry run said "unusable 25" and only 19 lines in the log said "unusable". The other six were never lost. Three discard paths printed three different phrasings, and the summary used the word from only one of them, so a grep for that word found 19 of 25. The six were one step limit and five `Outcome::Error`, all printed as "no answer (...)". Five of those were the model endpoint dropping the connection mid-run. The counter was already counted rather than derived, so it was right; the reporting around it was not. Every sampled directory now prints one line with the same prefix, scored or discarded, and every discard lands in one of eight named categories. `Tally` counts and prints in the same call, so an attempt cannot be counted without a line or printed without being counted. The summary states sampled, scored, and every category including the ones at zero, and the run asserts the categories sum to the sample before it prints a calibration report. The per attempt body moved into `record_attempt`, which takes the store write as a closure. That is what makes the accounting testable: the whole path runs with no model and no database, including a replay of the registry run's own 35 scored and 25 discarded. The two reasons PR #86 fixed keep their categories. Their counts should fall to zero, and a category that vanishes when it stops firing takes the evidence of the fix with it. A discard is never a way to raise the pass rate. Nothing invents an interval, an unreadable answer still contributes nothing to `n`, and a test says so. A discard now carries an excerpt of the model's answer next to the reason, capped and flattened to one line. The two bugs #86 fixed could not be replayed because only the error survived. That text is model authored: it goes to stdout and nowhere else, it is never written to the store, and no detector and nothing in the search layer can read it back. Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
adityak74
force-pushed
the
fix/calibration-attempt-accounting
branch
from
August 22, 2026 11:33
999a06f to
e85f17f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 60-directory registry run reported
unusable 25, but only 19 lines in thelog said
unusable. Six attempts looked like silent losses, and the summaryfigure was computed as sampled-minus-scored, so it counted them without knowing
what they were.
They were not lost. I was looking for the wrong word.
Three discard paths printed three different phrasings and the summary borrowed
the word from only one of them. The six are all there, under
no answer (...):So the harness was not dropping attempts. It was describing them in a
vocabulary that did not match its own total, which is worse in one specific
way: the count looked checkable and was not.
What changes
Every sampled directory is now scored or discarded into one of eight named
categories. A
Tallycounts and prints in the same call, so neither canhappen without the other, and the run asserts that scored plus discarded
equals sampled before it prints a calibration report.
The per-attempt work moved into
record_attempt, which takes the store writeas a closure. That is what lets the whole accounting path run in tests with no
model and no database.
Categories report zero rather than disappearing. PR #86 fixed both parser
reasons, so
no fenced json blockandnot the shape asked forshould nowfall to zero. A category that vanishes when it stops firing takes the evidence
of the fix with it.
A discard is never a way to raise the pass rate
This is the line that matters on a change to a measurement harness. Nothing
invents an interval, and an unreadable answer contributes nothing to
n.a_discarded_attempt_never_becomes_a_scored_oneexists to say so.Discards now carry the text that failed
The two parser bugs #86 fixed could not be replayed, because only the error
survived and the model's actual answer was never recorded. A discard now
carries an excerpt of the answer next to the reason.
That text is model-authored. It goes to stdout and nowhere else. It is never
written to the store, so no detector and nothing in the search layer can read
it back, which is the integrity rule in
CLAUDE.md.Why this matters for the go/no-go
nclearingMIN_CALIBRATION_N = 50is the current blocker on getting a realverdict instead of
NotEnoughEvidence. Six unexplained losses per sixty isdirectly material to that, and a harness whose own total cannot be reconciled
against its own output undermines every number downstream of it.
Verification
cargo test -p zorp-agent --features research --test evidence_calibration16 passed, 0 failed, 1 ignored (the live one). Among them
the_registry_run_that_seemed_to_lose_six_attempts_reconciles, whichreconstructs this exact discrepancy.
cargo fmt --all --checkclean.cargo clippy --workspace --exclude zorp-track --all-targets --locked -- -D warningsclean.Provenance
Written by a subagent that committed and pushed, then died on an API session
limit before it could open the PR or report results. I verified the branch
myself: I confirmed the six
no answerlines in the original log, ran thetests, and ran fmt and clippy. Its own final coherence read never happened,
so treat CI and review as the first full check.
https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4