fix(investigate): stop throwing away forecasts that were there - #86
Merged
Conversation
The 60-directory registry calibration run scored 35 of 60 attempts. The other 25 were discarded, and the corpus was not the reason: 11 failed with "EOF while parsing a value at line 1 column 0" and 8 with "no fenced json block in the forecast". Both are this file, and in both the forecast was present and readable. The EOF ones are an empty last fence. parse_forecast took the last block whatever it held, so a model that closed with an empty fence, or put its caveats in one, threw its own answer away and handed serde an empty string. It now reads backwards to the last block that is actually a forecast. Last-not-first still holds and still has its test; what changed is that last means the last forecast rather than the last block of any kind. The other 8 are a fence left open at the end. The forecast is asked for last, so a truncated answer loses its closing backticks and nothing else, and fenced_blocks dropped the body on the floor at end of input even though it parses. It is now kept. This loosens which block gets read and nothing else. is_forecast_shaped requires all four numbers present and finite, so a block that merely resembles a forecast is stepped over rather than repaired, and the block that does get selected still faces every coherence check that was there before. Two tests hold that line: one that a block missing fields is still refused, and one that an incoherent last forecast is reported rather than skipped in favour of an earlier one that happens to pass. Reading backwards must not become a way to shop for an answer. A forecast that cannot be read must never become one that was invented. What this does not do is claim a number. Whether these 19 attempts come back as scored forecasts can only be settled by rerunning, because the model's text was never logged, only the error it produced. Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
This was referenced Aug 22, 2026
adityak74
added a commit
that referenced
this pull request
Aug 22, 2026
The 60-directory registry run said "unusable 25" and only 19 lines in the log said "unusable". The other six were never lost. Three discard paths printed three different phrasings, and the summary used the word from only one of them, so a grep for that word found 19 of 25. The six were one step limit and five `Outcome::Error`, all printed as "no answer (...)". Five of those were the model endpoint dropping the connection mid-run. The counter was already counted rather than derived, so it was right; the reporting around it was not. Every sampled directory now prints one line with the same prefix, scored or discarded, and every discard lands in one of eight named categories. `Tally` counts and prints in the same call, so an attempt cannot be counted without a line or printed without being counted. The summary states sampled, scored, and every category including the ones at zero, and the run asserts the categories sum to the sample before it prints a calibration report. The per attempt body moved into `record_attempt`, which takes the store write as a closure. That is what makes the accounting testable: the whole path runs with no model and no database, including a replay of the registry run's own 35 scored and 25 discarded. The two reasons PR #86 fixed keep their categories. Their counts should fall to zero, and a category that vanishes when it stops firing takes the evidence of the fix with it. A discard is never a way to raise the pass rate. Nothing invents an interval, an unreadable answer still contributes nothing to `n`, and a test says so. A discard now carries an excerpt of the model's answer next to the reason, capped and flattened to one line. The two bugs #86 fixed could not be replayed because only the error survived. That text is model authored: it goes to stdout and nowhere else, it is never written to the store, and no detector and nothing in the search layer can read it back. Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 60-directory registry calibration run scored 35 of 60 attempts. The
report came back
NotEnoughEvidence { n: 35, required: 50 }, so the whole runstill cannot produce a GO or NO-GO.
The corpus was not the reason. All 25 discarded attempts failed in this file:
forecast block is not the shape asked for: EOF while parsing a value at line 1 column 0no fenced json block in the forecastIn both known cases the forecast was present and readable. The parser threw
it away.
The 11: an empty last fence
parse_forecasttook the last fenced block whatever it held:A model that closes with an empty fence, or puts its caveats in one, discarded
its own answer. The empty body reached serde as
"", which reports exactlyEOF while parsing a value at line 1 column 0. That error message is thesignature of an empty string, which is what made this findable.
It now reads backwards to the last block that is a forecast.
Last-not-first still holds. That rule exists so a model quoting the
requested shape while explaining itself does not get its example parsed as its
answer, and its test is untouched. What changed is that "last" now means the
last forecast rather than the last block of any kind.
The 8: a fence left open at the end
The forecast is asked for last, so a truncated answer loses its closing
backticks and nothing else.
fenced_blocksdropped the body at end of inputeven though it parses. It is now kept.
This loosens which block is read, and nothing else
The risk in a change like this is obvious: a parser that tries harder can
start inventing. It does not.
is_forecast_shapeddemands all four numbers present and finite. A blockthat merely resembles a forecast is stepped over, never repaired.
forecast that cannot be read "must never be worth faking", and that is
still true.
Two tests hold that line specifically:
a_block_that_is_not_a_forecast_is_still_refused— a block with onlyexpected_valueis refused, not completed.an_incoherent_last_forecast_is_reported_not_skipped— an incoherent finalforecast is reported, not skipped in favour of an earlier one that
happens to pass. Reading backwards must not become a way to shop for an
answer.
What I am not claiming
I cannot tell you these 19 attempts now score. The model's text was never
logged, only the error it produced, so the failing inputs are gone. The tests
reproduce both signatures from constructed cases that match the error messages
exactly, which is good evidence the diagnosis is right, and it is not the same
thing as a replay. Settling it takes a rerun.
And 6 of the 25 are still unexplained. 60 sampled, 35 scored, 19 printed a
reason. The remaining 6 produced no line at all and I could not find them in
the log. The
unusable 25counter is computed as sampled minus scored, so itcounts them without knowing what they were. That is a separate gap, in the
harness's reporting rather than here.
Verification
Five new tests, all watched failing first. The three targeting the bugs failed
with the real signatures; the two guard tests passed before the change, which
is what makes them guards.
cargo test -p zorp-agent --features research0 failed.cargo fmt --all --checkclean.cargo clippy --workspace --exclude zorp-track --all-targets --locked -- -D warningsclean.Note that CI's clippy job excludes
zorp-trackand does not enableresearch,so this file is not linted by CI. I linted it directly and it is clean.
zorp-trackhas two pre-existing lints under a current clippy, which is whythat crate is excluded and why the research-enabled invocation cannot run under
-D warningstoday. Not fixed here.https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4