feat(investigate): ask for confidence on a grid, so a band can be judged

[adityak74]

Stacked on #89. That PR is where required_band_n comes from, and it is
what makes this necessary. Merge #89 first.

The problem

A forecaster free to write any confidence spreads a run across bands that can
never individually be judged. From a live run, thirty scored forecasts:

@0.99  6      @0.96  5
@0.98  7      @0.95  4
@0.97  7      @0.93  1

Six bands, thickest holding seven. required_band_n asks for the larger of a
flat twenty and 1 / (1 - confidence):

band	rows needed	at the observed mix
0.99	100	~500 usable attempts
0.98	50
0.97	34

At roughly four minutes per attempt that is about forty-four hours of model
time, and no GO was reachable before then. The reason had nothing to do with
whether the forecaster was any good.

The change

The question is asked on a grid: 0.5, 0.7, 0.8, 0.9, 0.95.

Every level needs only the flat floor of twenty, which a run of ordinary size
reaches. A test asserts that against required_band_n rather than trusting the
arithmetic to stay true if someone edits the grid:

fn every_level_on_the_grid_is_judgeable_at_a_reachable_sample_size() {
    for level in CONFIDENCE_GRID {
        assert!(required_band_n(*level) <= MIN_BAND_N, ...);
    }
}

This is not a way to manufacture a GO

Worth being direct, because a change to a measurement that makes a verdict
easier to obtain deserves the suspicion.

• It does not make the forecaster better calibrated. A model that states
  0.9 and is right 60% of the time still fails, and now fails with enough rows
  behind the finding to mean it.
• It changes the precision claimed, not the shape of the question. The old
  prompt's own worked example was 0.80, which is on the grid.
• What it removes is a distinction the evidence could never support. Asking
  for a free number is asking to tell 0.97 from 0.98 with a sample that cannot
  tell them apart. Fixed grids are how calibration is measured everywhere else
  for exactly this reason.
• It can produce NO-GO just as easily. Thick bands are more capable of
  demonstrating miscalibration than thin ones, not less.

The grid is asked for, never enforced

parse_forecast still accepts any confidence in (0, 1), and
a_confidence_off_the_grid_is_still_parsed holds that open.

Filtering off-grid answers would discard real forecasts and bias the sample
toward the runs where the model happened to comply, which is a worse failure
than a thin band: it would be invisible.

Verification

cargo test -p zorp-agent --features research 0 failed (17 in
investigate::forecast, 3 new, all watched failing first on
cannot find value CONFIDENCE_GRID).
cargo fmt --all --check clean.

A 150-directory run on this grid is under way. I will report the curve and the
verdict it produces, whichever way it goes.

https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4

[adityak74]

Closing this. It does not work, and the run says so.

The model ignores the grid. With a binary I verified in release (the_prompt_offers_a_fixed_set_of_confidences passing, no recompile between build and launch), the first five forecasts came back 0.96, 0.96, 0.97, 0.99, 0.99. Five for five off-grid.

And it made the harness worse. no fenced json block in the forecast was 8 of 60 attempts on the original prompt, about 13%. On this prompt it was 7 of 15, about 47%. The added sentences after the JSON shape appear to degrade the response format, so the change cost usable forecasts while achieving nothing.

The reasoning in the PR body still stands: free-form confidences shatter the sample across bands nothing can judge. The fix is to bin at scoring time, which does not depend on the model complying with anything. That is #91, which I am rebasing onto #89 so it no longer carries this change.

Note for anyone reading the diff later: the arithmetic here was right and the intervention was wrong. Asking a model nicely is not a substitute for scoring the answer properly.

https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4

[adityak74]

Correcting one of the two reasons I gave for closing this.

I said the grid prompt raised no fenced json block in the forecast from about 13% of attempts to about 47%. That attribution was wrong. The run I restarted on the original prompt, from main, is showing 12 of 29 attempts with the same error, about 41%. Both prompts sit in the same range now, so the grid did not cause the increase.

What did change is not this PR. The earlier 60-directory run on the same original prompt saw 8 of 60, about 13%. Three candidates: the corpus sample differs (stride 56 at n=60 against stride 17 at n=200, so different directories), stealth/ox-alpha is a cloaked preview model and providers rotate what sits behind those names, or ordinary variance. I cannot separate them from the logs I have, and I am not going to guess.

The closure still stands, on the other reason alone. Five of five forecasts came back off-grid from a binary verified in release with no recompile between build and launch. The model does not honour the instruction, so the change buys nothing whatever its effect on formatting. #91 bins at scoring time and does not depend on the model complying.

Worth noting for whoever picks this up: PR #88 captures the model's raw answer next to each discard. Right now a no fenced json block at 41% is undiagnosable, because only the error survives and the text is gone. That is the same wall #86 hit.

https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4