Skip to content

test(aryabhatta): account for every attempt the calibration run samples - #88

Merged
adityak74 merged 1 commit into
mainfrom
fix/calibration-attempt-accounting
Aug 22, 2026
Merged

test(aryabhatta): account for every attempt the calibration run samples#88
adityak74 merged 1 commit into
mainfrom
fix/calibration-attempt-accounting

Conversation

@adityak74

Copy link
Copy Markdown
Contributor

The 60-directory registry run reported unusable 25, but only 19 lines in the
log said unusable. Six attempts looked like silent losses, and the summary
figure was computed as sampled-minus-scored, so it counted them without knowing
what they were.

They were not lost. I was looking for the wrong word.

Three discard paths printed three different phrasings and the summary borrowed
the word from only one of them. The six are all there, under no answer (...):

4  no answer (https...)                      network errors reaching the model
1  no answer (Connection reset by peer ...)  os error 54
1  no answer (step limit reached)

So the harness was not dropping attempts. It was describing them in a
vocabulary that did not match its own total, which is worse in one specific
way: the count looked checkable and was not.

What changes

Every sampled directory is now scored or discarded into one of eight named
categories
. A Tally counts and prints in the same call, so neither can
happen without the other, and the run asserts that scored plus discarded
equals sampled
before it prints a calibration report.

The per-attempt work moved into record_attempt, which takes the store write
as a closure. That is what lets the whole accounting path run in tests with no
model and no database.

Categories report zero rather than disappearing. PR #86 fixed both parser
reasons, so no fenced json block and not the shape asked for should now
fall to zero. A category that vanishes when it stops firing takes the evidence
of the fix with it.

A discard is never a way to raise the pass rate

This is the line that matters on a change to a measurement harness. Nothing
invents an interval, and an unreadable answer contributes nothing to n.
a_discarded_attempt_never_becomes_a_scored_one exists to say so.

Discards now carry the text that failed

The two parser bugs #86 fixed could not be replayed, because only the error
survived and the model's actual answer was never recorded. A discard now
carries an excerpt of the answer next to the reason.

That text is model-authored. It goes to stdout and nowhere else. It is never
written to the store, so no detector and nothing in the search layer can read
it back, which is the integrity rule in CLAUDE.md.

Why this matters for the go/no-go

n clearing MIN_CALIBRATION_N = 50 is the current blocker on getting a real
verdict instead of NotEnoughEvidence. Six unexplained losses per sixty is
directly material to that, and a harness whose own total cannot be reconciled
against its own output undermines every number downstream of it.

Verification

cargo test -p zorp-agent --features research --test evidence_calibration
16 passed, 0 failed, 1 ignored (the live one). Among them
the_registry_run_that_seemed_to_lose_six_attempts_reconciles, which
reconstructs this exact discrepancy.

cargo fmt --all --check clean.
cargo clippy --workspace --exclude zorp-track --all-targets --locked -- -D warnings clean.

Provenance

Written by a subagent that committed and pushed, then died on an API session
limit before it could open the PR or report results. I verified the branch
myself: I confirmed the six no answer lines in the original log, ran the
tests, and ran fmt and clippy. Its own final coherence read never happened,
so treat CI and review as the first full check.

https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4

The 60-directory registry run said "unusable 25" and only 19 lines in
the log said "unusable". The other six were never lost. Three discard
paths printed three different phrasings, and the summary used the word
from only one of them, so a grep for that word found 19 of 25.

The six were one step limit and five `Outcome::Error`, all printed as
"no answer (...)". Five of those were the model endpoint dropping the
connection mid-run. The counter was already counted rather than
derived, so it was right; the reporting around it was not.

Every sampled directory now prints one line with the same prefix,
scored or discarded, and every discard lands in one of eight named
categories. `Tally` counts and prints in the same call, so an attempt
cannot be counted without a line or printed without being counted. The
summary states sampled, scored, and every category including the ones
at zero, and the run asserts the categories sum to the sample before it
prints a calibration report.

The per attempt body moved into `record_attempt`, which takes the store
write as a closure. That is what makes the accounting testable: the
whole path runs with no model and no database, including a replay of
the registry run's own 35 scored and 25 discarded.

The two reasons PR #86 fixed keep their categories. Their counts should
fall to zero, and a category that vanishes when it stops firing takes
the evidence of the fix with it.

A discard is never a way to raise the pass rate. Nothing invents an
interval, an unreadable answer still contributes nothing to `n`, and a
test says so.

A discard now carries an excerpt of the model's answer next to the
reason, capped and flattened to one line. The two bugs #86 fixed could
not be replayed because only the error survived. That text is model
authored: it goes to stdout and nowhere else, it is never written to
the store, and no detector and nothing in the search layer can read it
back.

Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
@adityak74
adityak74 force-pushed the fix/calibration-attempt-accounting branch from 999a06f to e85f17f Compare August 22, 2026 11:33
@adityak74
adityak74 merged commit eb00f3b into main Aug 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant