Skip to content

Update parity tests to focus on pipeline integration, not scoring correctness #145

Description

@lukeinglis

Context

Once the parity repo has Naive Forecaster testing (validating that join_resolved_questions + score_forecasts match the leaderboard), this repo should stop re-testing scoring math and instead focus on testing correct usage of the parity package.

Tests to add/update

1. Zero missing forecasts (integration)

Run a round through eval.py --agent lab --round <ROUND> --raw and assert n_missing == 0. This validates that the multi-horizon extraction (#144) produces forecast keys that match all scoring keys from join_resolved_questions().

2. Per-horizon extraction (unit)

Call amulti_forecast on a dataset question with N resolution dates. Assert it returns exactly N probabilities, not 1. This validates the fix from PR #144.

3. verify_parity.py structural checks (existing)

Keep the existing checks: prompt template matching, scoring formula cross-check, dummy score = 50.0, resolution entry preservation, outcome diversity.

4. Remove scoring math tests from this repo

Tests that validate Brier score computation, Brier Index formula, or difficulty adjustment aggregation belong in the parity repo. This repo should trust the parity package for scoring correctness and only test that it passes the right inputs.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions