Skip to content

fix(investigate): stop throwing away forecasts that were there - #86

Merged
adityak74 merged 1 commit into
mainfrom
fix/forecast-parsing
Aug 22, 2026
Merged

fix(investigate): stop throwing away forecasts that were there#86
adityak74 merged 1 commit into
mainfrom
fix/forecast-parsing

Conversation

@adityak74

Copy link
Copy Markdown
Contributor

The 60-directory registry calibration run scored 35 of 60 attempts. The
report came back NotEnoughEvidence { n: 35, required: 50 }, so the whole run
still cannot produce a GO or NO-GO.

The corpus was not the reason. All 25 discarded attempts failed in this file:

count error
11 forecast block is not the shape asked for: EOF while parsing a value at line 1 column 0
8 no fenced json block in the forecast
6 no line printed, unexplained, see below

In both known cases the forecast was present and readable. The parser threw
it away.

The 11: an empty last fence

parse_forecast took the last fenced block whatever it held:

let block = fenced_blocks(text).pop().ok_or(ForecastError::NoFencedBlock)?;

A model that closes with an empty fence, or puts its caveats in one, discarded
its own answer. The empty body reached serde as "", which reports exactly
EOF while parsing a value at line 1 column 0. That error message is the
signature of an empty string, which is what made this findable.

It now reads backwards to the last block that is a forecast.

Last-not-first still holds. That rule exists so a model quoting the
requested shape while explaining itself does not get its example parsed as its
answer, and its test is untouched. What changed is that "last" now means the
last forecast rather than the last block of any kind.

The 8: a fence left open at the end

The forecast is asked for last, so a truncated answer loses its closing
backticks and nothing else. fenced_blocks dropped the body at end of input
even though it parses. It is now kept.

This loosens which block is read, and nothing else

The risk in a change like this is obvious: a parser that tries harder can
start inventing. It does not.

  • is_forecast_shaped demands all four numbers present and finite. A block
    that merely resembles a forecast is stepped over, never repaired.
  • The selected block still faces every coherence check that was already there.
  • Nothing substitutes a default interval. The module's existing comment says a
    forecast that cannot be read "must never be worth faking", and that is
    still true.

Two tests hold that line specifically:

  • a_block_that_is_not_a_forecast_is_still_refused — a block with only
    expected_value is refused, not completed.
  • an_incoherent_last_forecast_is_reported_not_skipped — an incoherent final
    forecast is reported, not skipped in favour of an earlier one that
    happens to pass. Reading backwards must not become a way to shop for an
    answer.

What I am not claiming

I cannot tell you these 19 attempts now score. The model's text was never
logged, only the error it produced, so the failing inputs are gone. The tests
reproduce both signatures from constructed cases that match the error messages
exactly, which is good evidence the diagnosis is right, and it is not the same
thing as a replay. Settling it takes a rerun.

And 6 of the 25 are still unexplained. 60 sampled, 35 scored, 19 printed a
reason. The remaining 6 produced no line at all and I could not find them in
the log. The unusable 25 counter is computed as sampled minus scored, so it
counts them without knowing what they were. That is a separate gap, in the
harness's reporting rather than here.

Verification

Five new tests, all watched failing first. The three targeting the bugs failed
with the real signatures; the two guard tests passed before the change, which
is what makes them guards.

cargo test -p zorp-agent --features research 0 failed.
cargo fmt --all --check clean.
cargo clippy --workspace --exclude zorp-track --all-targets --locked -- -D warnings clean.

Note that CI's clippy job excludes zorp-track and does not enable research,
so this file is not linted by CI. I linted it directly and it is clean.
zorp-track has two pre-existing lints under a current clippy, which is why
that crate is excluded and why the research-enabled invocation cannot run under
-D warnings today. Not fixed here.

https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4

The 60-directory registry calibration run scored 35 of 60 attempts. The
other 25 were discarded, and the corpus was not the reason: 11 failed
with "EOF while parsing a value at line 1 column 0" and 8 with "no
fenced json block in the forecast". Both are this file, and in both the
forecast was present and readable.

The EOF ones are an empty last fence. parse_forecast took the last
block whatever it held, so a model that closed with an empty fence, or
put its caveats in one, threw its own answer away and handed serde an
empty string. It now reads backwards to the last block that is actually
a forecast. Last-not-first still holds and still has its test; what
changed is that last means the last forecast rather than the last block
of any kind.

The other 8 are a fence left open at the end. The forecast is asked for
last, so a truncated answer loses its closing backticks and nothing
else, and fenced_blocks dropped the body on the floor at end of input
even though it parses. It is now kept.

This loosens which block gets read and nothing else. is_forecast_shaped
requires all four numbers present and finite, so a block that merely
resembles a forecast is stepped over rather than repaired, and the block
that does get selected still faces every coherence check that was there
before. Two tests hold that line: one that a block missing fields is
still refused, and one that an incoherent last forecast is reported
rather than skipped in favour of an earlier one that happens to pass.
Reading backwards must not become a way to shop for an answer. A
forecast that cannot be read must never become one that was invented.

What this does not do is claim a number. Whether these 19 attempts come
back as scored forecasts can only be settled by rerunning, because the
model's text was never logged, only the error it produced.

Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
@adityak74
adityak74 merged commit 38b9bb5 into main Aug 22, 2026
7 checks passed
adityak74 added a commit that referenced this pull request Aug 22, 2026
The 60-directory registry run said "unusable 25" and only 19 lines in
the log said "unusable". The other six were never lost. Three discard
paths printed three different phrasings, and the summary used the word
from only one of them, so a grep for that word found 19 of 25.

The six were one step limit and five `Outcome::Error`, all printed as
"no answer (...)". Five of those were the model endpoint dropping the
connection mid-run. The counter was already counted rather than
derived, so it was right; the reporting around it was not.

Every sampled directory now prints one line with the same prefix,
scored or discarded, and every discard lands in one of eight named
categories. `Tally` counts and prints in the same call, so an attempt
cannot be counted without a line or printed without being counted. The
summary states sampled, scored, and every category including the ones
at zero, and the run asserts the categories sum to the sample before it
prints a calibration report.

The per attempt body moved into `record_attempt`, which takes the store
write as a closure. That is what makes the accounting testable: the
whole path runs with no model and no database, including a replay of
the registry run's own 35 scored and 25 discarded.

The two reasons PR #86 fixed keep their categories. Their counts should
fall to zero, and a category that vanishes when it stops firing takes
the evidence of the fix with it.

A discard is never a way to raise the pass rate. Nothing invents an
interval, an unreadable answer still contributes nothing to `n`, and a
test says so.

A discard now carries an excerpt of the model's answer next to the
reason, capped and flattened to one line. The two bugs #86 fixed could
not be replayed because only the error survived. That text is model
authored: it goes to stdout and nowhere else, it is never written to
the store, and no detector and nothing in the search layer can read it
back.

Claude-Session: https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant