Skip to content

Add dataset base-rate shrinkage to improve Brier Index #165

Description

@lukeinglis

Problem

Currently _apply_calibration in eval.py only calibrates market questions (anchoring to freeze value) and applies extremity clamping. Dataset questions pass through with no shrinkage toward the base rate.

LLMs tend to be overconfident on dataset questions — their predictions deviate too far from 0.5. This adds noise that hurts the Brier score.

Proposed Solution

Add a base-rate shrinkage step for dataset questions in _apply_calibration. This blends the model's prediction toward 0.5 (uninformative prior) with a tunable weight, reducing overconfidence on dataset questions where the model has less reliable signal than prediction markets.

Scope

  • eval.py — add dataset shrinkage constant and logic in _apply_calibration

Rationale

  • Market questions already benefit from anchoring to market price (weight 0.91)
  • Dataset questions have no such anchor — shrinkage toward base rate fills this gap
  • Brier score is minimized when forecasts are well-calibrated; overconfidence is the primary LLM failure mode

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions