feat(metrics): add calibration diagnostics for probability forecasts

[stefan-jansen]

Summary

Adds ml4t.diagnostic.metrics.calibration, exported from ml4t.diagnostic.metrics, for scoring probability forecasts (model outputs or prediction-market prices) against binary outcomes, including outcomes correlated within an event.

Function	Returns
brier_score(forecasts, outcomes, *, weights=None)	float
log_loss(forecasts, outcomes, *, weights=None, eps=1e-15)	float (forecasts clipped to [eps, 1-eps])
brier_decomposition(forecasts, outcomes, *, bins=None, weights=None)	BrierDecomposition: reliability, resolution, uncertainty, within-bin variance and covariance, base rate, brier_skill_score
reliability_table(forecasts, outcomes, *, bins=PREDICTION_MARKET_BIN_EDGES, clusters=None, weights=None, confidence=0.95, n_bootstrap=1000, seed=0)	pl.DataFrame, one row per bin: n, n_clusters, weight, mean forecast, observed frequency, difference, CI, CI method
expected_calibration_error(...), max_calibration_error(...)	float
calibration_slope(forecasts, outcomes, *, spec="linear"|"logit", clusters=None, weights=None, confidence=0.95, eps=1e-3)	CalibrationSlope: a, b, SEs, CIs, p-value for b = 1

Design choices:

• Decomposition identity is exact. With bins=None each distinct forecast is a bin (Murphy, 1973), which suits tick-quoted prices. With explicit bins, the two Stephenson et al. (2008) within-bin terms are added so the components still sum to the Brier score.
• Reliability intervals: Wilson without clusters, Wilson on the Kish effective n with weights, and a percentile cluster bootstrap when clusters is given. The bootstrap resamples whole clusters across all bins using per-cluster sparse sums, so it scales to about 1M rows and 200k clusters with 1000 replicates in roughly 5 s. The library has no existing cluster bootstrap; its existing bootstraps are stationary/block resampling for time series.
• Slope errors: a sandwich estimator with the CR1 / HC1 small-sample factor. It matches statsmodels' OLS and GLM results to 1e-8 (tested). It is implemented directly because statsmodels warns that cluster covariance is not fully supported with GLM var_weights.
• Bin edges are rounded to 12 decimals, so computed edges such as np.linspace(0, 1, 11) still place a price of 0.3 in [0.3, 0.4).
• Missing values raise an error instead of being dropped.

Tests

tests/metrics/test_calibration.py (35 tests) checks the following:

• a planted favorite-longshot distortion (logit q = 1.5 logit p) is recovered by the reliability table, with intervals covering the true bin frequency, and by the slope;
• calibrated synthetic prices give b ≈ 1 with an interval covering 1;
• the decomposition identity holds for every binning, with and without weights;
• the cluster-bootstrap width grows by about √m for perfectly correlated clusters of size m, and singleton clusters reproduce the Wilson width;
• weights are equivalent to duplicated rows;
• the sandwich matches statsmodels.

Each of 14 planted bugs was confirmed to fail its targeted test, and every test passed again after the source was restored. The planted bugs: drop the within-bin covariance, resolution around 0.5, bootstrap over observations instead of clusters, wrong Wilson z, unweighted observed frequency, closed-left bins, no edge rounding, reversed difference sign, unweighted ECE, no log-loss clip, unaggregated cluster scores, logit spec without the transform, p-value against 0, and no small-sample factor.

Docs

New Methods page docs/methods/calibration.md. Its example is executed by test_documentation_examples. Also updates the nav, the methods index, the API reference members, the reference list, and metrics/AGENTS.md.

Dependency lock

A second commit updates uv.lock to pypdf 6.19.0, urllib3 2.8.0 and virtualenv 21.14.6. The security workflow PyPI audit found 19 known vulnerabilities in the previously locked versions, which failed every Audit job and blocked the required Build Package check. These are transitive dependencies and the lock change is independent of the calibration code. pip-audit now reports none, and the full suite still passes.

Local gates

• ruff check / ruff format --check: pass
• ty check: pass
• pytest tests/ -n auto --timeout 120: 5472 passed, 7 skipped
• mkdocs build --strict: pass
• check_documentation_links.py --site site: pass
• check_book_links.py: pass
• pre-commit run --all-files: pass

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.