Problem
Currently _apply_calibration in eval.py only calibrates market questions (anchoring to freeze value) and applies extremity clamping. Dataset questions pass through with no shrinkage toward the base rate.
LLMs tend to be overconfident on dataset questions — their predictions deviate too far from 0.5. This adds noise that hurts the Brier score.
Proposed Solution
Add a base-rate shrinkage step for dataset questions in _apply_calibration. This blends the model's prediction toward 0.5 (uninformative prior) with a tunable weight, reducing overconfidence on dataset questions where the model has less reliable signal than prediction markets.
Scope
eval.py — add dataset shrinkage constant and logic in _apply_calibration
Rationale
- Market questions already benefit from anchoring to market price (weight 0.91)
- Dataset questions have no such anchor — shrinkage toward base rate fills this gap
- Brier score is minimized when forecasts are well-calibrated; overconfidence is the primary LLM failure mode
Problem
Currently
_apply_calibrationineval.pyonly calibrates market questions (anchoring to freeze value) and applies extremity clamping. Dataset questions pass through with no shrinkage toward the base rate.LLMs tend to be overconfident on dataset questions — their predictions deviate too far from 0.5. This adds noise that hurts the Brier score.
Proposed Solution
Add a base-rate shrinkage step for dataset questions in
_apply_calibration. This blends the model's prediction toward 0.5 (uninformative prior) with a tunable weight, reducing overconfidence on dataset questions where the model has less reliable signal than prediction markets.Scope
eval.py— add dataset shrinkage constant and logic in_apply_calibrationRationale