[Meerav29]

Tracking: #19. Closes docs/phase2-status.md caveat 1 on the code side ("Calibration was not re-checked for post-quali"). Nothing here has been run against the real data.

What this slice does

gp backtest --post-quali now collects podium calibration for the post-quali arm, using the same rows and shape as the pre-weekend arm.

• calibration_podium_postquali.csv: one row per scored driver per round that post-quali scored. Its columns are p_podium_pred, podium_actual, season, round. calibration_podium.csv gets the same two trailing columns, so the original two columns keep their names and order.
• backtest_summary.json → calibration_podium: a binned table plus an ECE.
  • pre covers every scored round and is always written.
  • With --post-quali the block also has postquali and pre_paired. pre_paired is the pre arm restricted to the rounds post-quali scored, so you can compare pre_paired.ece and postquali.ece over the same races.
• ECE definition: ten equal-width bins on [0, 1], with p = 1.0 in the top bin and empty bins left out. ECE = Σ_b (n_b / N) · |mean predicted_b − observed podium rate_b|. It is in probability units, so 0 means perfectly calibrated and 0.02 means two points off on average.
• postquali_worse_rounds in the summary lists rounds where post-quali winner log loss exceeds pre-weekend's by more than 1.0 nats (POSTQUALI_WORSE_THRESHOLD_NATS), worst first. Each entry has season, round, log_loss_pre, log_loss_postquali, delta, grid_pole_driver, actual_winner. This is a list only. It feeds no verdict and claims no diagnosis.
• score_round adds actual_winner and grid_pole_driver to every row of backtest_metrics.csv. They are read from the race rows: finish_position == 1, and grid_position == 1, the same grid post-quali conditions on.

Diff: 252 added, 10 deleted across 4 files (src/backtest.py, one new test file, decisions.md, queue.md).

Review pack

About 20 minutes.

1. Read the two 2026-10-25 entries in docs/autopilot/decisions.md (3 min). They set the bin scheme and ECE, and the 1.0-nat threshold.
2. Read calibration_summary and postquali_worse_rounds in src/backtest.py (5 min). Together they are about 50 lines.
3. Run the new tests (2 min):
   PYTHONPATH=src pytest tests/test_postquali_calibration.py -q
4. Owner only, the real numbers. This does not need the 2.7 h run if you already have outputs/phase2_2022_2025/, but that directory predates this slice and has no post-quali calibration rows, so the post-quali ECE needs a re-run:
   PYTHONPATH=src python -m cli backtest --seasons 2022-2025 --rounds all --method nuts --post-quali
   Then compare calibration_podium.pre_paired with .postquali in backtest_summary.json. Check that postquali_worse_rounds contains 2022 R13, 2022 R14 and 2024 R21. A cheaper smoke test is --seasons 2025 --rounds coarse; the gate will report invalid, but the calibration block is still written.

Acceptance criteria

• The post-quali arm writes its own calibration_podium_postquali.csv in the same shape as the pre-weekend file. test_end_to_end_writes_both_calibration_files_and_both_summaries runs the real run_backtest(post_quali=True) on data/driver_rounds_fixture.csv with ADVI. It asserts that both CSVs exist with identical columns and the same row count. In this test, NUTS conditioning is swapped for a 200-step ADVI fit to keep it to seconds. test_the_post_quali_arm_adds_one_calibration_row_per_scored_driver covers the arm on its own (20 rows, 3 podium hits).
• The summary reports a binned table and an ECE for both arms. The ECE is defined above. test_binned_table_and_ece_match_a_hand_computation checks ECE = 2/5·0.05 + 3/5·0.45. Further tests cover a perfectly calibrated arm (ECE 0), p = 1.0 landing in the top bin, and an empty arm reporting ece: null rather than 0. The end-to-end test asserts the three blocks pre, pre_paired and postquali, each with bin counts summing to n.
• Rounds worse by more than a stated threshold are listed with grid-pole driver and actual winner. test_only_rounds_worse_by_more_than_the_threshold_are_listed_worst_first covers: deltas of +2.4 and +1.5 listed in that order; +1.0 (exactly at threshold), −0.5 and a NaN round excluded. test_post_quali_summary_carries_the_listing_and_its_threshold checks that the listing is JSON-serializable. test_score_round_names_the_pole_sitter_and_the_winner checks both names against the fixture's 2024 R1 rows.
• Tested on the fixture. No full backtest run.

Evidence, from a fresh Python 3.11 venv (pymc 5.28.5, arviz 0.23.4):

$ PYTHONPATH=src pytest tests -q
335 passed, 661 warnings in 76.55s

That is 326 before this slice (the slice-9 review count) plus 9 new tests.

Tests fail without the change. On base auto/queue, the new file errors at import because the new names don't exist. I also applied five mutations to this branch's src/backtest.py. Each produced 1 failed, 22 passed across the new file and test_backtest_postquali.py:

Mutation	Result
run_backtest no longer passes calibration_rows to the arm	1 failed
ECE as an unweighted mean over bins	1 failed
threshold >= instead of >	1 failed
grid_pole_driver always None	1 failed
no clip, so p = 1.0 falls off the top bin	1 failed

Not caught: restricting pre_paired to the post-quali rounds. The fixture backtest scores a single round (2024 R2), so pre and pre_paired are identical there, and replacing pre_paired with pre would pass. The code is a three-line merge on (season, round) in run_backtest. Please read it there.

Decisions made without you

Both are recorded in docs/autopilot/decisions.md, 2026-10-25.

1. Bins and ECE. I chose ten equal-width 0.1 bins, matching how phase2-status.md already reads the pre file ("the 0.5-0.6 bin"), and an n-weighted absolute-gap ECE. To make that ECE comparable across arms, I added pre_paired and the season, round columns. Rejected: quantile bins, which give each arm different edges so the tables can't be read side by side. Also rejected: comparing against all-rounds pre only, which would let a failed post-quali round change one side and not the other.
2. The "much worse" threshold is 1.0 nats, strictly greater. That means post-quali put about 2.7× less probability on the winner than pre-weekend did. It includes the three rounds phase2-status.md names (deltas 2.39, 2.24, 1.57). I can't tell from the sandbox how many of the other ten worse rounds clear it. Rejected: any positive delta, which lists all 13, most of them noise. Also rejected: a threshold relative to pre log loss, which is harder to read.

Deliberately not included

• Running the real backtest. That is the owner's gate.
• Any diagnosis of the confident misses. This slice only lists them.
• Calibration of p_win or p_points. The queue asks for the podium file's shape only.
• Retiring confidence.py or the pkl.

How to revert

git revert <merge commit> on auto/queue. The only shape change is two trailing columns on calibration_podium.csv and new keys in backtest_summary.json/backtest_metrics.csv, which nothing in the repo reads.

---

Generated by Claude Code