Skip to content

Add top-label histogram binning as a post-hoc confidence calibrator - #17

Open
Yasas-Sri wants to merge 1 commit into
nokia-applied-research:mainfrom
Yasas-Sri:calibrate-binning
Open

Yasas-Sri wants to merge 1 commit into
nokia-applied-research:mainfrom
Yasas-Sri:calibrate-binning

Conversation

@Yasas-Sri

@Yasas-Sri Yasas-Sri commented Oct 3, 2026 •

Copy link
Copy Markdown

Closes #8.

Adds anyjev/calibrate/binning.py: top-label histogram binning (Zadrozny and Elkan, 2001) as a standalone post-hoc confidence calibrator, with fit / apply / to_dict / from_dict. It works on any [N, K] probabilities AnyJev returns, fit on human labels or the model's own answers after reasoning.

  • Equal-mass bins over top-1 confidence, each mapped to its measured top-1 accuracy. Bins under min_count are merged into a neighbour.
  • The argmax is kept. The other options are scaled by one common factor, and the new top-1 is floored just above the rescaled runner-up, so a low bin value can't let the runner-up overtake it. Accuracy is exactly the input's.
  • No defaults on the fields: no bin table leaves probabilities unchanged, so HistogramBinning() without fitting raises.
  • Outputs are clipped to at least 1e-12 and renormalised, so a bin that was always right gives no exact 0 or 1, and a later log stays finite.

Tests (tests/test_binning.py): exact per-bin values on constructed sets, argmax kept and rows summing to 1 on random data (K = 2, 4, 20), the runner-up case, confidence 1.0, no exact 0/1 outputs, to_dict / from_dict round trip, bad input, and a held-out ECE check on an overconfident model (with a local ECE helper)

ruff check anyjev scripts space tests and pytest -q pass locally.

@Yasas-Sri Yasas-Sri changed the title Add top-label histogram binning as an L1 calibrator Add top-label histogram binning as a post-hoc confidence calibrator Oct 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

work on help-wanted slot: anyjev/calibrate/binning.py (L1 alternative)

1 participant