fix(track): a calibration band is a bin sized by what it can judge

[adityak74]

Stacked on cal/confidence-grid (#90), which is stacked on cal/band-evidence (#89). Both must merge first. Targets cal/confidence-grid.

The problem

required_band_n is right, but nothing had asked what it does to a forecaster that writes free-form confidences. One band per distinct stated confidence shatters the sample. A registry run against stealth/ox-alpha, 35 scored forecasts:

0.93  n= 3  cov=1.000   needs  20   TOO THIN
0.95  n= 5  cov=1.000   needs  20   TOO THIN
0.96  n= 3  cov=0.333   needs  25   TOO THIN
0.97  n=11  cov=0.909   needs  34   TOO THIN
0.98  n= 9  cov=0.889   needs  50   TOO THIN
0.99  n= 4  cov=1.000   needs 100   TOO THIN

Nothing concluded, at any tolerance. The rows were not the problem: pooled they say a stated 0.968 against an observed 0.886, a forecaster overconfident by about eight points, which is a finding.

Fixed bins were checked and are worse. Cutting at 0.7, 0.85 and 0.95 gives three and thirty two, and the bin of three blocks the go at every tolerance forever, so one stray forecast at 0.93 holds the verdict hostage for the life of the record. A grid fixed in advance is a bet on the forecaster's habits.

The binning rule, and why

Rows already arrive ascending by stated confidence. Adjacent rows accumulate until the bin holds what required_band_n asks of its own mean, and then the bin closes. Two rules on top of that:

1. A whole group of identical stated confidences moves together. A bin never closes part way through one stated confidence. So every forecast claiming the same number is judged in one place, and no boundary depends on the order rows happened to be written in. The alternative, splitting a tie group on the query's tiebreak, is deterministic but allocates identical forecasts to different bins by something unrelated to what is being measured.
2. A leftover tail too small to close joins the bin before it, rather than being dropped or standing as its own perpetual BandTooThin. Standing alone is what made fixed bins fail.

CalibrationBand::confidence is now the mean of the stated confidences in the bin, never an edge, because the comparison has to be against what was claimed. When a bin holds one distinct stated confidence, mean_stated returns that number itself rather than dividing a float sum, because required_band_n is a step function and one ulp can move what a band is asked for by a whole row.

CalibrationBand gained parts: Vec<BandPart>, one entry per distinct stated confidence pooled into the bin, ascending, with its own n and covered. parts.first() and parts.last() are the bin's span, and the rest is what makes a pass auditable rather than asserted. The four calibration printers now print it.

Worth knowing. Written out, n >= ceil(n / (n - S)) for a bin of n rows whose stated confidences sum to S is exactly n - S >= 1: a bin is thick enough when a correct forecaster expects at least one miss in it. That form only rises as rows are added, and MIN_BAND_N is a count, which only rises too. That is the proof that merging a tail into a closed bin can never reopen it, and it is why a greedy left to right pass is well defined.

On the real run the rule closes one bin of 31 at the 0.98 group (mean 0.9648, needs 29), then the four rows at 0.99 cannot close and merge back: one band, n=35, mean stated 0.9677, observed 0.8857, gap 0.082. At a tolerance of 0.20 the only remaining reason is NotEnoughEvidence (35 < 50). At 0.05 it is a BandOutOfTolerance with that gap, which is the overconfidence the data really shows.

Does this make GO easier?

Yes, and it should be read as a loosening. A go that no tolerance could reach is now reachable. Three things convinced me it is the honest kind.

• What it removes is silence, not a finding. Every band in the motivating run was BandTooThin, which is the report saying it has not measured anything. Nothing that was a demonstrated miss became a pass by binning alone.
• A group that can stand alone still does. A stated confidence carrying the rows its own requirement asks for closes its own bin and is never averaged into a neighbour. a_stated_confidence_that_can_stand_alone_is_not_pooled pins that, and a_bin_thick_enough_to_show_a_miss_still_shows_one pins that a real miss still reports as one.
• The verdict is untouched. Every guarantee from fix(track): a band too thin to judge is its own no-go, never a miss #89 still holds with its tests: MIN_CALIBRATION_N, MIN_BAND_N, required_band_n, thinness checked before tolerance, NaN checked before comparison, a stated confidence outside (0, 1) being Unjudgeable, and the empty report answering NotEnoughEvidence. A report too small to fill one bin is one band that verdict calls BandTooThin.

What this can hide, and what limits it

Pooling averages. The pooled gap is the row-weighted mean of the gaps that went into the bin, so:

• Same-signed gaps cannot cancel. The average of two gaps in the same direction lies between them. Uniform overconfidence, the failure this whole report exists to catch, is exactly that case, so binning cannot make it disappear.
• Opposite-signed gaps can cancel, and the smaller group can carry real weight. 19 rows at a stated 0.50 all covered (gap +0.50), pooled with 20 at a stated 0.90 covering 0.60 (gap -0.30), read as a pooled gap of 0.09. The second group alone is a demonstrated miss today. That is a genuine loss and I am not going to pretend otherwise.

What limits it:

• Bins are the narrowest the requirement allows. A bin closes at the first group boundary where it can be judged, so only a group that arithmetically cannot be judged on its own is ever absorbed. The absorbed group's weight is capped by its own requirement: at most 19 rows at low confidence, more at high confidence, which is where this is weakest and worth watching.
• Nothing is hidden from the report, only from the verdict. parts carries every pooled stated confidence with its own n and covered, so the cancellation above is visible on the page and re-derivable by hand.
• There is no scheme that avoids this. Any rule that never drops a forecast and judges only judgeable aggregates has to pool the groups that cannot be judged. The alternative is dropping them, which biases the curve in silence and is worse.

Not negotiable, and tested

• Boundaries never see outcomes. bin_boundaries(&[f64]) takes the stated confidences and nothing else, so a boundary cannot be fitted to the answer. The signature is what makes it true; the_bins_do_not_move_when_every_outcome_flips is what keeps it true.
• Every scored row lands in exactly one bin. every_scored_forecast_lands_in_exactly_one_bin sums the bands' n and covered against the report's, over four shapes, and the_bins_are_a_partition_of_every_scored_row checks the boundaries directly.
• The binner closes on exactly the predicate verdict judges by, through the same required_band_n over the same mean_stated. A bin the binner called finished can never come back as BandTooThin.

Tests added

Test	What it pins
the_six_band_run_collapses_into_a_band_that_can_be_judged	The real 35 row run becomes one judgeable band; mean stated is the mean; at 0.20 the only reason left is NotEnoughEvidence; at 0.05 the pooled overconfidence is a finding
every_scored_forecast_lands_in_exactly_one_bin	Bands' n and covered sum to the report's, over four shapes
a_report_too_small_for_one_bin_is_still_too_thin	Five rows is one band and BandTooThin, not Go
the_bins_do_not_move_when_every_outcome_flips	Boundaries are outcome-independent, over two plans that mix hits and misses inside a stated confidence
a_stated_confidence_that_can_stand_alone_is_not_pooled	A judgeable group closes its own bin
a_bin_thick_enough_to_show_a_miss_still_shows_one	60 at a stated 0.80 covering 0.45 is still BandOutOfTolerance, gap 0.35
a_tail_that_cannot_close_never_reopens_the_bin_it_joins	Merging a tail keeps the bin closed
every_closed_bin_satisfies_the_rule_the_verdict_uses	Over six plans, every band meets required_band_n(band.confidence) and no closed bin comes back thin
reaching_the_flat_floor_is_not_enough_for_a_confident_bin	20 at a stated 0.98 clears MIN_BAND_N, needs 50, keeps filling
the_bins_are_a_partition_of_every_scored_row	No empty, overlapping or dropped bins, including a NaN and a stated 1.0
the_curve_is_one_point_per_band_in_ascending_order	Rewritten from the old per-confidence version; still pins the query's sort

Verification

Command	Result
cargo test -p zorp-track --lib calibration	47 passed, 0 failed
cargo test -p zorp-track	251 lib + 10 integration passed, 0 failed
cargo test -p zorp-agent --features research	598 + 32 + 22 + 4 + 3 + 4 + 9 + 4 passed, 0 failed (model-calling calibration tests stay #[ignore]d and were not run)
cargo fmt --all --check	clean
cargo clippy --workspace --exclude zorp-track --all-targets --locked -- -D warnings	clean, exit 0
cargo clippy -p zorp-track --all-targets	2 warnings, both pre-existing and not mine: prereg.rs:170 too many arguments, track.rs:344 contains() over iter().any()
rustup update	stable unchanged at 1.98.0; MSRV toolchain 1.95 present

Mutations run against my own guards

Mutation	Result
Close a bin on n >= MIN_BAND_N instead of required_band_n(mean)	2 tests fail: reaching_the_flat_floor_is_not_enough_for_a_confident_bin, every_closed_bin_satisfies_the_rule_the_verdict_uses
Drop the leftover tail instead of merging it	6 tests fail, including every_scored_forecast_lands_in_exactly_one_bin and the_bins_are_a_partition_of_every_scored_row
Report the bin's low edge instead of the mean	1 test fails: the_six_band_run_collapses_into_a_band_that_can_be_judged
Let the outcomes reorder the rows before binning	3 tests fail, including the_bins_do_not_move_when_every_outcome_flips

The first pass of the outcome-independence test did not catch the fourth mutation, because the six band run's fully-covered groups make a flip degenerate. The test now runs a second plan that mixes hits and misses inside each stated confidence, and it fails as it should.

Note on the machine

The disk hit 100% mid-run and every build failed with ENOSPC. I freed zorp/target/debug/incremental in the main checkout (pure cache, no process using it, 11G) and two duplicate libduckdb-sys build directories inside this agent's own worktree target. I did not touch target/release anywhere, the calibration run on .claude/worktrees/aryabhatta/target/release, port 7777, or ~/.config/zorp/. There is still roughly 350G of stale per-worktree debug caches on that volume and it will wedge again.

https://claude.ai/code/session_01KGPVQ8wUG7h36zashWYCp4