Context
Once the parity repo has Naive Forecaster testing (validating that join_resolved_questions + score_forecasts match the leaderboard), this repo should stop re-testing scoring math and instead focus on testing correct usage of the parity package.
Tests to add/update
1. Zero missing forecasts (integration)
Run a round through eval.py --agent lab --round <ROUND> --raw and assert n_missing == 0. This validates that the multi-horizon extraction (#144) produces forecast keys that match all scoring keys from join_resolved_questions().
2. Per-horizon extraction (unit)
Call amulti_forecast on a dataset question with N resolution dates. Assert it returns exactly N probabilities, not 1. This validates the fix from PR #144.
3. verify_parity.py structural checks (existing)
Keep the existing checks: prompt template matching, scoring formula cross-check, dummy score = 50.0, resolution entry preservation, outcome diversity.
4. Remove scoring math tests from this repo
Tests that validate Brier score computation, Brier Index formula, or difficulty adjustment aggregation belong in the parity repo. This repo should trust the parity package for scoring correctness and only test that it passes the right inputs.
Related
Context
Once the parity repo has Naive Forecaster testing (validating that
join_resolved_questions+score_forecastsmatch the leaderboard), this repo should stop re-testing scoring math and instead focus on testing correct usage of the parity package.Tests to add/update
1. Zero missing forecasts (integration)
Run a round through
eval.py --agent lab --round <ROUND> --rawand assertn_missing == 0. This validates that the multi-horizon extraction (#144) produces forecast keys that match all scoring keys fromjoin_resolved_questions().2. Per-horizon extraction (unit)
Call
amulti_forecaston a dataset question with N resolution dates. Assert it returns exactly N probabilities, not 1. This validates the fix from PR #144.3. verify_parity.py structural checks (existing)
Keep the existing checks: prompt template matching, scoring formula cross-check, dummy score = 50.0, resolution entry preservation, outcome diversity.
4. Remove scoring math tests from this repo
Tests that validate Brier score computation, Brier Index formula, or difficulty adjustment aggregation belong in the parity repo. This repo should trust the parity package for scoring correctness and only test that it passes the right inputs.
Related