[Meerav29]

Tracking: #19 (autopilot window 2). Slice 9 in docs/autopilot/queue.md.

What this slice does

Adds the Phase 3 season-level backtest (spec §7 season-level metrics, §8 Phase 3, §4 legacy benchmark). It is built and tested on the fixture. It has not been run on real data.

• gp backtest --season-level [--cutoffs 2,6,12]: for each held-out season and cutoff, it fits through season:cutoff (ADVI by default, --method nuts also works) and chains the rest of that season with slice-8's cli.forecast_season. The chain only gets rows up to the cutoff. The real calendar is read off the season's own R/S rows through the remaining_calendar loader seam (backtest.backtest_schedule).
• Each (season, cutoff) is scored against the real final standings (backtest.score_season_cutoff):
  • constructor-share Spearman, v2 exp_share against the actual share
  • Brier on the drivers' and constructors' title winner
  • coverage of the 80% interval on final points, for drivers and for constructors
• The legacy number at the matching cutoff (backtest.legacy_constructor_spearman) comes from features.build_features(early_rounds=cutoff) and the Ridge pipeline from train.py, trained only on earlier seasons. The static 0.88 is reported next to it as legacy_reference_spearman.
• backtest_summary.json gets a season_level block with per-cutoff means, failed_cutoffs, gate_valid, gate_invalid_reasons and v2_beats_legacy_spearman. The verdict is null in any of these cases: a cutoff failed (fit, chain, or no rounds left after it), a cutoff was never attempted, or a scored cutoff has no legacy number. If a race-level summary already exists in --out-dir, the block is merged into it and nothing is overwritten.
• Also writes season_backtest_metrics.csv and, for each cutoff, season_<Y>_r<c>_{drivers,constructors}_championship.csv.
• drivers_championship.csv and constructors_championship.csv (including gp season's output) gain two columns, p10_final_points and p90_final_points. The §6 columns are unchanged.

Review pack

About 25 minutes, most of it waiting on the fits.

1. Read the three 2026-10-23 entries in docs/autopilot/decisions.md (5 min). The one most likely to need your call is the legacy comparison. It trains on earlier seasons only, not LOSO, so its numbers will probably come out below 0.88.
2. Check that the legacy inputs exist locally: ls data/race_results.csv data/constructor_standings.csv. If they're missing, run PYTHONPATH=src python src/collect.py --start-year 2014 --end-year 2025 first. Without them every cutoff has no legacy number and the verdict is null, which is intended.
3. Run the real season-level backtest. This is the Phase 3 gate and only you run it:

   PYTHONPATH=src python -m cli backtest --seasons 2022-2025 --season-level --method nuts --n-trials 5000 --out-dir outputs/season_bt

   That is 12 fits (4 seasons × 3 cutoffs). NUTS uses 250/250 draws on 1 chain, so expect tens of minutes. Not timed. Use --method advi for a quicker first look.
4. Look at outputs/season_bt/backtest_summary.json → season_level (5 min):
   • gate_valid should be true and failed_cutoffs empty.
   • per_cutoff["2"].mean_constructor_spearman_v2 against mean_constructor_spearman_legacy.
   • mean_coverage80_* should be near 0.8.
   • v2_beats_legacy_spearman is the verdict.
5. Optional: spot-check one season_2024_r6_drivers_championship.csv against what you remember of 2024 after round 6.

Acceptance criteria

• gp backtest --season-level fits at cutoffs after rounds 2, 6 and 12 of each held-out season and chains the rest with slice-8's simulator. --cutoffs defaults to 2,6,12. The chain is cli.forecast_season, unchanged from slice-8. Evidence: test_cli_routes_season_level_to_the_season_harness asserts cutoffs == [2, 6, 12] and that the race-level path doesn't run. TestLoop::test_the_chain_sees_nothing_after_the_cutoff_but_the_real_calendar asserts the chain sees round.max() == 1 at cutoff 1 and gets the real remaining calendar (Saudi Arabian Grand Prix).
• Writes constructor-share Spearman per cutoff next to the legacy model's number at the matching cutoff, title-winner Brier for both championships, and 80% interval coverage on final points. These are the per_cutoff[c].mean_constructor_spearman_{v2,legacy}, mean_brier_{drivers,constructors}_title and mean_coverage80_{drivers,constructors} keys. Tests: test_scores_against_the_whole_season_not_the_cutoff, test_coverage_ignores_drivers_the_chain_did_not_simulate, test_championship_tables_carry_the_80pct_points_interval, test_legacy_trains_only_on_earlier_seasons.
• The season-level block follows the same gate_valid / null-verdict discipline, so a partial run or a failed cutoff can't report a verdict. Tests: test_a_partial_run_cannot_report_a_verdict, test_a_failed_cutoff_cannot_report_a_verdict_and_is_named, test_a_missing_legacy_number_cannot_report_a_verdict, TestLoop::test_a_failed_fit_is_recorded_and_voids_the_verdict, test_a_cutoff_with_nothing_left_to_chain_is_a_failure_not_a_skip. A losing v2 reports False, not None (test_v2_losing_to_legacy_is_false_not_none).
• Tested end to end on data/driver_rounds_fixture.csv with ADVI and tiny draw counts. The real backtest was not run. test_end_to_end_on_the_fixture_writes_a_season_level_block does a real ADVI fit (200 steps, 20 draws) through 2024:1, chains round 2 with 200 trials, scores it, and writes every file. It takes about 7 s once PyTensor has compiled. The fixture only has 2024 R1–R2, so this test uses cutoff 1. The default 2/6/12 cutoffs fail on it as no_rounds_after_cutoff, which the test above asserts.
• Existing tests still pass, and the new tests fail without the change.

Suite, run locally (Python 3.11.17 venv, pymc 5.28.5, arviz 0.23.4, pandas 2.3.3), PYTHONPATH=src pytest tests -q:

baseline on auto/queue d680929: 309 passed, 660 warnings in 66.10s
this branch:                    326 passed, 661 warnings in 25.89s

CI (Python 3.12) is the authoritative gate. I did not run 3.12 locally.

Mutations, each applied alone to tests/test_season_backtest.py (17 tests) and then reverted:

Mutation	Result
M1: chain gets the full driver_rounds instead of the rows up to the cutoff (leak)	1 failed
M2: failed cutoffs don't invalidate the gate	3 failed
M3: legacy uses LOSO (!= season) instead of earlier seasons	1 failed
M4: constructors table drops p10/p90_final_points	2 failed
M5: driver coverage over the whole table instead of the line-up	1 failed (after tightening the test, which passed M5 at first)
M6: unattempted cutoffs not counted	1 failed
M7: sprint weekends not flagged in the backtest schedule	1 failed

Decisions made without you

All three are recorded in docs/autopilot/decisions.md (2026-10-23):

1. Legacy number: a rolling Ridge (train.py pipeline, all FEATURE_COLS) on seasons before the held-out one, using features built at early_rounds=cutoff. The 0.88 is shown as a reference. Rejected: LOSO, the 0.88 method, which trains on future seasons. Also rejected: replaying LassoCV selection, weight tuning and the model pick for each fold.
2. Metric definitions:
   • Title Brier is the multi-class Brier summed over the field (0–2). Rejected: the per-driver mean, which shrinks with field size.
   • 80% coverage is an inclusive [p10, p90] on final points. Driver coverage counts only the cutoff line-up.
   • The calendar is read from the season's own rows. Rejected: FastF1's schedule (network, and it lists rounds with no result).
3. --season-level runs instead of the race-level backtest, not as well as it. Its block is merged into an existing summary in the same --out-dir. Rejected: coupling it to the 2.7 h race-level run.

Deliberately not included

• Running the real backtest. That is the owner's gate (review pack step 3). Phase 3 has not passed.
• Retiring confidence.py or the pkl. Both are untouched.
• Parallelising across seasons, and warm starts between cutoffs.
• A full replay of the legacy training procedure. See decision 1.
• Season-level calibration plots, and CRPS on final points.

How to revert

git revert <merge commit> on auto/queue. The change is additive:

• the new functions sit at the end of src/backtest.py
• cmd_backtest gets a new branch, and the old path is unchanged
• sim/season.py gains two output columns
• there is one new test file and the queue/decisions docs

No data files or schemas change.

---

Generated by Claude Code