From 83ff7cfd5c66210f53e5e87e0cbdb83390d949c6 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 13:34:16 +0000 Subject: [PATCH 1/2] slice-9: season-level backtest harness (gp backtest --season-level) Fits at cutoffs 2/6/12 of each held-out season, chains the rest with slice-8's simulator, and scores constructor-share Spearman (next to a rolling legacy number at the same cutoff), title Brier for both championships and 80% interval coverage on final points. Writes a season_level block to backtest_summary.json with gate_valid / null verdict discipline. Tested end to end on the fixture with ADVI; the real backtest has not been run. Refs #19 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01JyCVWAmeHjqML4h9ugdbGQ --- docs/autopilot/decisions.md | 18 +++ docs/autopilot/queue.md | 2 +- src/backtest.py | 254 ++++++++++++++++++++++++++++++++++ src/cli.py | 13 ++ src/sim/season.py | 18 ++- tests/test_season_backtest.py | 251 +++++++++++++++++++++++++++++++++ 6 files changed, 551 insertions(+), 5 deletions(-) create mode 100644 tests/test_season_backtest.py diff --git a/docs/autopilot/decisions.md b/docs/autopilot/decisions.md index e3be7fb..290e923 100644 --- a/docs/autopilot/decisions.md +++ b/docs/autopilot/decisions.md @@ -172,3 +172,21 @@ Question: Which line-up races the remaining rounds, and how are ties on point Chosen: The last completed round's entrants. Drivers who scored earlier but are no longer entered keep their points and score nothing more. Team current points come from the recorded points on R/S rows, so they follow whoever drove the car. Ties go to more Grand Prix wins, then column order. Rejected: An explicit entry list (nothing in the repo maintains one) and the full FIA countback (second places onward). The countback adds a per-position tally for ties that fractional expected points almost never hit. Blast radius: `cli.forecast_season`, `sim.season.final_positions`. + +## 2026-10-23 — slice-9 +Question: §7 asks for constructor-share Spearman "comparable to the legacy 0.88 ... match the cutoff". The 0.88 is a leave-one-season-out CV at `EARLY_ROUNDS=2`, so it trains on seasons *after* the one it scores. What legacy number goes next to v2's at each cutoff? +Chosen: A rolling legacy number per (season, cutoff): `features.build_features(early_rounds=cutoff)`, then `train.py`'s Ridge pipeline (`SimpleImputer` + `Ridge(alpha=1.0)`) on all `FEATURE_COLS`, trained on seasons strictly before the held-out one. It sees what the v2 fit sees and nothing later. The 0.88 is reported beside it as `legacy_reference_spearman`, never scored. If the legacy inputs (`data/race_results.csv`, `data/constructor_standings.csv`) can't be read, the cutoff has no legacy number and the verdict is null. +Rejected: Reproducing the 0.88 method (LOSO). It would let the legacy model learn from 2024-2025 when scoring 2022, which the rolling v2 fit cannot do, so the comparison would favour legacy for a reason that isn't skill. Also rejected: replaying `train.py`'s LassoCV selection, rule-change weight tuning and Ridge-vs-XGBoost pick per fold. More faithful, but it is a nested CV per (season, cutoff), and the shipped pkl picks among those settings with all seasons in view, so the tuned choice leaks too. +Blast radius: `backtest.legacy_constructor_spearman`, `backtest._legacy_features`. Only the legacy column of the season-level block. Swapping in another legacy method is one function. + +## 2026-10-23 — slice-9 +Question: How are the title Brier and the 80% coverage defined, and how is the held-out season's calendar supplied? +Chosen: Title Brier is the multi-class Brier, summed over the field (0 = certain and right, 2 = certain and wrong), not the per-driver mean the race-level `_brier` uses: a per-driver mean shrinks with field size and makes every title forecast look near-perfect. Coverage counts an entity when its real final points lie in `[p10_final_points, p90_final_points]` inclusive, two columns now added to both championship tables. Driver coverage is over the cutoff round's line-up only; a driver who had already left has a fixed total that any interval covers. The calendar is read off the held-out season's own R/S rows through `remaining_calendar`'s loader seam, and the chain is handed only rows up to the cutoff. +Rejected: The mean Brier, for the reason above. Also rejected: FastF1's schedule for past seasons. Network-bound, and it would list cancelled or uncollected rounds that have no actual result to score against. +Blast radius: `backtest.score_season_cutoff`, `backtest.backtest_schedule`, and two added columns on `drivers_championship.csv` / `constructors_championship.csv` (§6 columns unchanged). + +## 2026-10-23 — slice-9 +Question: Does `--season-level` run alongside the race-level backtest or instead of it, and where does its block go? +Chosen: Instead of. `gp backtest --season-level` runs only the season harness and writes `season_level` into `backtest_summary.json`. If that file already exists in `--out-dir` (a race-level run pointed at the same dir), the block is merged in and the race-level keys are kept. A cutoff past a season's last round, a failed fit, or a failed chain is a named failed cutoff, and any failed or unattempted cutoff, or a scored cutoff with no legacy number, makes `v2_beats_legacy_spearman` null. +Rejected: Running both in one invocation. The race-level grid is the slow part (~2.7 h with `--post-quali`) and the season harness is 12 fits for 2022-2025; coupling them makes the cheap one wait on the expensive one. +Blast radius: `cli.cmd_backtest` (a new branch, the old path untouched) and `backtest.run_season_backtest`. diff --git a/docs/autopilot/queue.md b/docs/autopilot/queue.md index cc70102..c5df0e9 100644 --- a/docs/autopilot/queue.md +++ b/docs/autopilot/queue.md @@ -423,7 +423,7 @@ Acceptance: ## slice-9: season-level backtest harness -Status: todo +Status: in-progress Spec: §7 season-level metrics, §8 Phase 3, §4 (legacy benchmark). diff --git a/src/backtest.py b/src/backtest.py index d8bb799..8a2c5d9 100644 --- a/src/backtest.py +++ b/src/backtest.py @@ -507,3 +507,257 @@ def run_backtest(seasons: list[int], n_trials: int = 5000, method: str = "advi", print(json.dumps(summary, indent=2)) print(f"Saved -> {outdir}") return metrics_df + + +# --------------------------------------------------------------------------- +# Season-level backtest (spec sec 7 season-level metrics, sec 8 Phase 3) +# --------------------------------------------------------------------------- + +SEASON_CUTOFFS = (2, 6, 12) +# Legacy leave-one-season-out constructor-share Spearman at EARLY_ROUNDS=2 +# (spec sec 4). Reported beside the rolling legacy number, never scored. +LEGACY_REFERENCE_SPEARMAN = 0.88 + + +def backtest_schedule(driver_rounds: pd.DataFrame): + """A `sim.season.remaining_calendar` loader that reads a held-out season's + calendar off the rounds that actually ran, in FastF1's column names. A + round is a sprint weekend if it has S rows (as `cli._round_has_sprint`).""" + def loader(season: int) -> pd.DataFrame: + rows = driver_rounds[driver_rounds["season"] == season] + races = rows[rows["session_type"] == "R"].drop_duplicates("round").sort_values("round") + sprints = set(rows.loc[rows["session_type"] == "S", "round"]) + return pd.DataFrame({ + "RoundNumber": races["round"].astype(int).to_numpy(), + "EventName": races["event_name"].astype(str).to_numpy(), + "EventFormat": ["sprint_qualifying" if r in sprints else "conventional" + for r in races["round"]], + }) + return loader + + +def _champion(points: dict, wins: dict) -> str | None: + """Most points, ties to more Grand Prix wins (as `sim.season.final_positions`).""" + if not points: + return None + return max(points, key=lambda k: (points[k], wins.get(k, 0))) + + +def _title_brier(p_title: dict, champion: str | None) -> float: + """Multi-class Brier on the title winner: summed over the field, 0 to 2.""" + if champion is None: + return float("nan") + p = dict(p_title) + p.setdefault(champion, 0.0) + return float(sum((v - (1.0 if k == champion else 0.0)) ** 2 for k, v in p.items())) + + +def _coverage80(table: pd.DataFrame, key: str, actual: dict, among=None) -> float: + rows = table if among is None else table[table[key].isin(among)] + hits = [r.p10_final_points <= actual.get(getattr(r, key), 0.0) <= r.p90_final_points + for r in rows.itertuples(index=False)] + return float(np.mean(hits)) if hits else float("nan") + + +def _share_spearman(predicted: dict, actual: dict) -> float: + common = [k for k in predicted if k in actual] + if len(common) < 3: + return float("nan") + rho, _ = spearmanr([predicted[k] for k in common], [actual[k] for k in common]) + return float(rho) + + +def score_season_cutoff(drivers: pd.DataFrame, constructors: pd.DataFrame, + driver_rounds: pd.DataFrame, season: int, lineup: list) -> dict: + """Score one cutoff's championship tables against the season's real final + standings (spec sec 7). Driver coverage is over `lineup`, the drivers the + chain actually simulated: a driver who had already left keeps a fixed + total, which an interval covers trivially.""" + from cli import _season_so_far + + last = int(driver_rounds.loc[driver_rounds["season"] == season, "round"].max()) + drv_pts, team_pts, drv_wins, team_wins = _season_so_far(driver_rounds, season, last) + total = sum(team_pts.values()) + actual_share = {t: p / total for t, p in team_pts.items()} if total > 0 else {} + return { + "constructor_spearman_v2": _share_spearman( + dict(zip(constructors["team"], constructors["exp_share"])), actual_share), + "brier_drivers_title": _title_brier(dict(zip(drivers["driver"], drivers["p_title"])), + _champion(drv_pts, drv_wins)), + "brier_constructors_title": _title_brier(dict(zip(constructors["team"], constructors["p_title"])), + _champion(team_pts, team_wins)), + "coverage80_drivers": _coverage80(drivers, "driver", drv_pts, among=lineup), + "coverage80_constructors": _coverage80(constructors, "team", team_pts), + } + + +def legacy_constructor_spearman(features: pd.DataFrame, season: int) -> float: + """The legacy season-level model at a matching cutoff: `features` is + `features.build_features(early_rounds=cutoff)`. Trained on seasons + strictly before `season`, the same information the rolling v2 fit has, + with `train.py`'s Ridge pipeline on its full feature set.""" + from sklearn.impute import SimpleImputer + from sklearn.linear_model import Ridge + from sklearn.pipeline import Pipeline + from train import FEATURE_COLS + + known = features[features["season_points_share"].notna()] + train, test = known[known["year"] < season], known[known["year"] == season] + if train.empty or len(test) < 3: + return float("nan") + model = Pipeline([("imp", SimpleImputer()), ("reg", Ridge(alpha=1.0))]) + model.fit(train[FEATURE_COLS], train["season_points_share"]) + rho, _ = spearmanr(model.predict(test[FEATURE_COLS]), test["season_points_share"]) + return float(rho) + + +def _legacy_features(cutoff: int) -> pd.DataFrame: + from features import build_features + + return build_features(early_rounds=cutoff) + + +def season_level_summary(rows: pd.DataFrame, seasons: list, cutoffs: list) -> dict: + """The season-level block of `backtest_summary.json`. Same discipline as + `post_quali_summary`: every requested (season, cutoff) is accounted for, + and `v2_beats_legacy_spearman` is None unless the run is complete -- every + cutoff scored and a legacy number beside each one.""" + out: dict = {"seasons": list(seasons), "cutoffs": list(cutoffs), + "legacy_reference_spearman": LEGACY_REFERENCE_SPEARMAN, + "n_cutoffs_expected": len(seasons) * len(cutoffs)} + ok = rows[rows["status"] == "ok"] if len(rows) else rows + out["n_cutoffs_ok"] = len(ok) + attempted = {(int(r.season), int(r.cutoff)) for r in rows.itertuples()} if len(rows) else set() + missing = [f"{s}:{c} (not attempted)" for s in seasons for c in cutoffs if (s, c) not in attempted] + failed = missing + [f"{int(r.season)}:{int(r.cutoff)} ({r.status})" + for r in rows.itertuples() if r.status != "ok"] + out["failed_cutoffs"] = failed + + metrics = ["constructor_spearman_v2", "constructor_spearman_legacy", "brier_drivers_title", + "brier_constructors_title", "coverage80_drivers", "coverage80_constructors"] + per_cutoff = {} + for c in cutoffs: + at = ok[ok["cutoff"] == c] if len(ok) else ok + per_cutoff[str(c)] = {"n_seasons": len(at), + **{f"mean_{m}": (float(at[m].mean()) if len(at) and at[m].notna().any() + else None) for m in metrics}} + out["per_cutoff"] = per_cutoff + + no_legacy = (ok[ok["constructor_spearman_legacy"].isna()] if len(ok) else ok) + reasons = [] + if failed: + reasons.append(f"{len(failed)} of {out['n_cutoffs_expected']} cutoff(s) did not score") + if len(no_legacy): + reasons.append(f"{len(no_legacy)} scored cutoff(s) have no legacy number to compare against") + if len(ok) and ok["constructor_spearman_v2"].isna().any(): + reasons.append("a scored cutoff has no v2 Spearman (fewer than 3 teams in common)") + if not len(ok): + reasons.append("nothing scored") + out["gate_valid"] = not reasons + out["gate_invalid_reasons"] = reasons + if len(ok): + out["mean_constructor_spearman_v2"] = float(ok["constructor_spearman_v2"].mean()) + out["mean_constructor_spearman_legacy"] = (float(ok["constructor_spearman_legacy"].mean()) + if ok["constructor_spearman_legacy"].notna().any() + else None) + out["v2_beats_legacy_spearman"] = (bool(out["mean_constructor_spearman_v2"] + > out["mean_constructor_spearman_legacy"]) + if not reasons else None) + return out + + +def run_season_backtest(seasons: list[int], cutoffs=SEASON_CUTOFFS, n_trials: int = 5000, + method: str = "advi", advi_steps: int = 4000, advi_draws: int = 300, + out_dir: str | None = None, driver_rounds: pd.DataFrame | None = None, + driver_meta: pd.DataFrame | None = None, circuits: pd.DataFrame | None = None, + legacy_features=_legacy_features) -> pd.DataFrame: + """For each held-out season and cutoff: fit through `season:cutoff`, chain + the rest of that season with `cli.forecast_season`, and score the tables + against how the season really ended. The chain sees only rows up to the + cutoff; the real calendar is read off the season's own rows.""" + from cli import _entrants_for_round, _load_circuits, forecast_season + from model.pace import build_pace_data, fit_advi, fit_nuts + from model.reliability import build_reliability_data + + if driver_rounds is None: + driver_rounds = pd.read_csv(os.path.join(DATA_DIR, "driver_rounds.csv")) + meta_path = os.path.join(DATA_DIR, "driver_meta.csv") + driver_meta = pd.read_csv(meta_path) if os.path.exists(meta_path) else None + circuits = _load_circuits() if circuits is None else circuits + outdir = out_dir or os.path.join(OUTPUTS_DIR, f"backtest_{datetime.now():%Y%m%d_%H%M%S}") + os.makedirs(outdir, exist_ok=True) + loader = backtest_schedule(driver_rounds) + legacy_cache: dict = {} + rows = [] + t_start = time.time() + + for season in seasons: + season_rounds = set(driver_rounds.loc[(driver_rounds["season"] == season) + & (driver_rounds["session_type"] == "R"), "round"]) + for cutoff in cutoffs: + row = {"season": season, "cutoff": cutoff} + rows.append(row) + if cutoff not in season_rounds or cutoff >= max(season_rounds): + row["status"] = "no_rounds_after_cutoff" + continue + t0 = time.time() + print(f"[season-backtest] {season}: fitting through {season}:{cutoff} ({method}) ...") + try: + data = build_pace_data(driver_rounds, driver_meta, through_season=season, + through_round=cutoff) + idata = (fit_advi(data, n=advi_steps, draws=advi_draws) if method == "advi" + else fit_nuts(data, draws=250, tune=250, chains=1)) + reliability = build_reliability_data(driver_rounds, through_season=season, + through_round=cutoff) + except Exception as exc: + print(f" WARNING: fit failed for {season}:{cutoff}: {exc}") + row["status"] = "fit_failed" + continue + seen = driver_rounds[(driver_rounds["season"] < season) + | ((driver_rounds["season"] == season) & (driver_rounds["round"] <= cutoff))] + try: + drivers, constructors = forecast_season(idata, data, reliability, seen, circuits, + n_trials=n_trials, seed=cutoff, + schedule_loader=loader) + except Exception as exc: + print(f" WARNING: season chain failed for {season}:{cutoff}: {exc}") + row["status"] = "forecast_failed" + continue + for name, table in (("drivers", drivers), ("constructors", constructors)): + table.to_csv(os.path.join(outdir, f"season_{season}_r{cutoff}_{name}_championship.csv"), + index=False) + lineup = _entrants_for_round(seen, season, cutoff)["abbreviation"].tolist() + row.update(score_season_cutoff(drivers, constructors, driver_rounds, season, lineup)) + + if cutoff not in legacy_cache: + try: + legacy_cache[cutoff] = legacy_features(cutoff) + except Exception as exc: + print(f" WARNING: legacy features unavailable at cutoff {cutoff}: {exc}") + legacy_cache[cutoff] = None + feats = legacy_cache[cutoff] + row["constructor_spearman_legacy"] = (legacy_constructor_spearman(feats, season) + if feats is not None else float("nan")) + row["status"] = "ok" + row["elapsed_s"] = time.time() - t0 + + rows_df = pd.DataFrame(rows) + for col in ("constructor_spearman_v2", "constructor_spearman_legacy"): + if col not in rows_df.columns: + rows_df[col] = float("nan") + rows_df.to_csv(os.path.join(outdir, "season_backtest_metrics.csv"), index=False) + + block = season_level_summary(rows_df, seasons, list(cutoffs)) + block.update({"method": method, "n_trials": n_trials, "total_elapsed_s": time.time() - t_start}) + # Join a race-level summary already in this --out-dir rather than overwrite it. + summary_path = os.path.join(outdir, "backtest_summary.json") + summary = {} + if os.path.exists(summary_path): + with open(summary_path) as f: + summary = json.load(f) + summary["season_level"] = block + with open(summary_path, "w") as f: + json.dump(summary, f, indent=2) + print(json.dumps(block, indent=2)) + print(f"Saved -> {outdir}") + return rows_df diff --git a/src/cli.py b/src/cli.py index 5324b74..8175fd8 100644 --- a/src/cli.py +++ b/src/cli.py @@ -626,6 +626,14 @@ def cmd_backtest(args): if len(seasons) == 2: seasons = list(range(seasons[0], seasons[1] + 1)) + if args.season_level: + from backtest import run_season_backtest + + run_season_backtest(seasons=seasons, cutoffs=[int(c) for c in args.cutoffs.split(",")], + n_trials=args.n_trials, method=args.method, + advi_steps=args.advi_steps, out_dir=args.out_dir) + return + run_backtest( seasons=seasons, n_trials=args.n_trials, method=args.method, rounds=args.rounds, advi_steps=args.advi_steps, out_dir=args.out_dir, @@ -685,6 +693,11 @@ def build_parser() -> argparse.ArgumentParser: "against pre-weekend (Phase 2 gate); writes conditioning_log.csv") p_bt.add_argument("--post-quali-chains", type=int, default=2, help="chains for each round's post-quali conditioning refit") + p_bt.add_argument("--season-level", action="store_true", + help="Season-level backtest instead (spec sec 7, Phase 3): fit at each " + "--cutoffs round, chain the rest of the season, score both championships") + p_bt.add_argument("--cutoffs", default="2,6,12", + help="with --season-level: comma-separated rounds to fit through") p_bt.set_defaults(func=cmd_backtest) return parser diff --git a/src/sim/season.py b/src/sim/season.py index b356502..b4673b6 100644 --- a/src/sim/season.py +++ b/src/sim/season.py @@ -165,9 +165,16 @@ def _position_columns(pos: np.ndarray) -> dict: "p90_final_position": float(np.percentile(pos, 90))} +def _points_interval(points: np.ndarray) -> dict: + """The 80% interval on final points, which the season backtest scores (sec 7).""" + return {"p10_final_points": float(np.percentile(points, 10)), + "p90_final_points": float(np.percentile(points, 90))} + + def drivers_championship(total_points: np.ndarray, total_wins: np.ndarray, driver_ids: list, teams: list, current_points) -> pd.DataFrame: - """sec 6 `drivers_championship.csv`, plus `team` and `current_points`.""" + """sec 6 `drivers_championship.csv`, plus `team`, `current_points` and + the 80% interval on final points.""" pos = final_positions(total_points, total_wins) rows = [] for i, driver in enumerate(driver_ids): @@ -175,13 +182,15 @@ def drivers_championship(total_points: np.ndarray, total_wins: np.ndarray, drive "p_title": float(np.mean(pos[:, i] == 1)), "exp_final_points": float(np.mean(total_points[:, i])), **_position_columns(pos[:, i]), - "p_top3": float(np.mean(pos[:, i] <= 3))}) + "p_top3": float(np.mean(pos[:, i] <= 3)), + **_points_interval(total_points[:, i])}) return pd.DataFrame(rows).sort_values("exp_final_points", ascending=False).reset_index(drop=True) def constructors_championship(total_points: np.ndarray, total_wins: np.ndarray, team_names: list, current_points) -> pd.DataFrame: - """sec 6 `constructors_championship.csv`, plus `current_points`. + """sec 6 `constructors_championship.csv`, plus `current_points` and the + 80% interval on final points. `exp_share` is the per-trial share of all constructors' points, averaged.""" pos = final_positions(total_points, total_wins) totals = total_points.sum(axis=1, keepdims=True) @@ -192,5 +201,6 @@ def constructors_championship(total_points: np.ndarray, total_wins: np.ndarray, "p_title": float(np.mean(pos[:, j] == 1)), "exp_final_points": float(np.mean(total_points[:, j])), "exp_share": float(np.mean(share[:, j])), - **_position_columns(pos[:, j])}) + **_position_columns(pos[:, j]), + **_points_interval(total_points[:, j])}) return pd.DataFrame(rows).sort_values("exp_final_points", ascending=False).reset_index(drop=True) diff --git a/tests/test_season_backtest.py b/tests/test_season_backtest.py new file mode 100644 index 0000000..bd9c05f --- /dev/null +++ b/tests/test_season_backtest.py @@ -0,0 +1,251 @@ +"""Season-level backtest harness (spec sec 7 season-level metrics, sec 8 +Phase 3; autopilot slice-9). One end-to-end run on the fixture with a real +ADVI fit at tiny step counts; everything else is pure or stubbed.""" +import json +import os + +import numpy as np +import pandas as pd +import pytest + +import backtest +import cli +import model.pace +from backtest import ( + backtest_schedule, legacy_constructor_spearman, run_season_backtest, score_season_cutoff, + season_level_summary, +) +from sim.season import constructors_championship, drivers_championship, remaining_calendar +from train import FEATURE_COLS + +FIXTURE = os.path.join(os.path.dirname(__file__), "..", "data", "driver_rounds_fixture.csv") + + +def _fixture(): + return pd.read_csv(FIXTURE) + + +def _legacy_features(teams=("A", "B", "C", "D", "E"), years=(2022, 2023, 2024), seed=0): + """Synthetic `features.build_features` output: the target is a monotone + function of `early_points_share`, so a fitted Ridge ranks it perfectly.""" + rng = np.random.default_rng(seed) + rows = [] + for y in years: + share = rng.dirichlet(np.ones(len(teams))) + for t, s in zip(teams, share): + row = {c: 0.0 for c in FEATURE_COLS} + row.update(year=y, constructor=t, early_points_share=s, season_points_share=s) + rows.append(row) + return pd.DataFrame(rows) + + +# --------------------------------------------------------------------------- +# End to end on the fixture: real ADVI fit, real chain, real scoring +# --------------------------------------------------------------------------- + +def test_end_to_end_on_the_fixture_writes_a_season_level_block(tmp_path): + # a race-level summary already in the out dir is kept, not overwritten + (tmp_path / "backtest_summary.json").write_text(json.dumps({"n_rounds_scored": 7})) + rows = run_season_backtest([2024], cutoffs=[1], n_trials=200, advi_steps=200, advi_draws=20, + out_dir=str(tmp_path), driver_rounds=_fixture(), + legacy_features=lambda c: _legacy_features()) + + assert rows["status"].tolist() == ["ok"] + summary = json.loads((tmp_path / "backtest_summary.json").read_text()) + assert summary["n_rounds_scored"] == 7 + block = summary["season_level"] + assert block["gate_valid"] is True and block["gate_invalid_reasons"] == [] + assert isinstance(block["v2_beats_legacy_spearman"], bool) + assert block["per_cutoff"]["1"]["n_seasons"] == 1 + assert block["per_cutoff"]["1"]["mean_constructor_spearman_legacy"] == pytest.approx(1.0) + r = rows.iloc[0] + assert -1.0 <= r["constructor_spearman_v2"] <= 1.0 + assert 0.0 <= r["brier_drivers_title"] <= 2.0 and 0.0 <= r["brier_constructors_title"] <= 2.0 + assert 0.0 <= r["coverage80_drivers"] <= 1.0 + for f in ("season_backtest_metrics.csv", "season_2024_r1_drivers_championship.csv", + "season_2024_r1_constructors_championship.csv"): + assert (tmp_path / f).exists() + + +# --------------------------------------------------------------------------- +# The loop with the fit and chain stubbed: leak safety and failure statuses +# --------------------------------------------------------------------------- + +def _tables(teams=("Red Bull Racing", "Ferrari", "McLaren", "Mercedes")): + drivers = pd.DataFrame({"driver": ["VER"], "p_title": [1.0], "p10_final_points": [0.0], + "p90_final_points": [100.0]}) + constructors = pd.DataFrame({"team": list(teams), "p_title": [1.0] + [0.0] * (len(teams) - 1), + "exp_share": np.linspace(0.4, 0.1, len(teams)), + "p10_final_points": 0.0, "p90_final_points": 100.0}) + return drivers, constructors + + +class TestLoop: + def _run(self, monkeypatch, tmp_path, chain=None, fit=None, cutoffs=(1,)): + seen = {} + + def fake_chain(idata, data, reliability, driver_rounds, circuits, **kw): + seen["rounds"] = driver_rounds + seen["calendar"] = remaining_calendar(2024, 1, loader=kw["schedule_loader"]) + return (chain or _tables)() + + monkeypatch.setattr(model.pace, "fit_advi", fit or (lambda *a, **kw: "IDATA")) + monkeypatch.setattr(cli, "forecast_season", fake_chain) + rows = run_season_backtest([2024], cutoffs=list(cutoffs), n_trials=10, out_dir=str(tmp_path), + driver_rounds=_fixture(), + legacy_features=lambda c: _legacy_features()) + return rows, seen + + def test_the_chain_sees_nothing_after_the_cutoff_but_the_real_calendar(self, monkeypatch, tmp_path): + rows, seen = self._run(monkeypatch, tmp_path) + assert rows["status"].tolist() == ["ok"] + assert seen["rounds"]["round"].max() == 1 + assert seen["calendar"]["event_name"].tolist() == ["Saudi Arabian Grand Prix"] + + def test_a_failed_fit_is_recorded_and_voids_the_verdict(self, monkeypatch, tmp_path): + def boom(*a, **kw): + raise RuntimeError("diverged") + rows, _ = self._run(monkeypatch, tmp_path, fit=boom) + assert rows["status"].tolist() == ["fit_failed"] + block = json.loads((tmp_path / "backtest_summary.json").read_text())["season_level"] + assert block["failed_cutoffs"] == ["2024:1 (fit_failed)"] + assert block["v2_beats_legacy_spearman"] is None + + def test_a_failed_chain_is_recorded(self, monkeypatch, tmp_path): + def boom(): + raise RuntimeError("no schedule") + rows, _ = self._run(monkeypatch, tmp_path, chain=boom) + assert rows["status"].tolist() == ["forecast_failed"] + + def test_a_cutoff_with_nothing_left_to_chain_is_a_failure_not_a_skip(self, monkeypatch, tmp_path): + rows, _ = self._run(monkeypatch, tmp_path, cutoffs=(1, 2, 6)) + assert rows["status"].tolist() == ["ok", "no_rounds_after_cutoff", "no_rounds_after_cutoff"] + block = json.loads((tmp_path / "backtest_summary.json").read_text())["season_level"] + assert block["n_cutoffs_expected"] == 3 and block["n_cutoffs_ok"] == 1 + assert block["gate_valid"] is False and block["v2_beats_legacy_spearman"] is None + + +# --------------------------------------------------------------------------- +# Summary: gate_valid / null-verdict discipline +# --------------------------------------------------------------------------- + +def _rows(v2, legacy, statuses=None, cutoffs=None): + n = len(v2) + return pd.DataFrame({ + "season": [2022 + i // 3 for i in range(n)], "cutoff": cutoffs or [(2, 6, 12)[i % 3] for i in range(n)], + "status": statuses or ["ok"] * n, "constructor_spearman_v2": v2, + "constructor_spearman_legacy": legacy, "brier_drivers_title": [0.5] * n, + "brier_constructors_title": [0.4] * n, "coverage80_drivers": [0.8] * n, + "coverage80_constructors": [0.7] * n, + }) + + +def test_complete_run_reports_a_real_verdict_per_cutoff(): + out = season_level_summary(_rows([0.9, 0.8, 0.95], [0.85, 0.85, 0.85]), [2022], [2, 6, 12]) + assert out["gate_valid"] is True + assert out["v2_beats_legacy_spearman"] is True + assert out["per_cutoff"]["6"]["mean_constructor_spearman_v2"] == pytest.approx(0.8) + assert out["per_cutoff"]["6"]["mean_constructor_spearman_legacy"] == pytest.approx(0.85) + assert out["legacy_reference_spearman"] == 0.88 + + +def test_v2_losing_to_legacy_is_false_not_none(): + out = season_level_summary(_rows([0.5, 0.5, 0.5], [0.9, 0.9, 0.9]), [2022], [2, 6, 12]) + assert out["v2_beats_legacy_spearman"] is False + + +def test_a_partial_run_cannot_report_a_verdict(): + # season 2023 was requested but never attempted + out = season_level_summary(_rows([0.9, 0.9, 0.9], [0.1, 0.1, 0.1]), [2022, 2023], [2, 6, 12]) + assert out["v2_beats_legacy_spearman"] is None + assert "2023:2 (not attempted)" in out["failed_cutoffs"] + assert out["n_cutoffs_expected"] == 6 and out["n_cutoffs_ok"] == 3 + + +def test_a_failed_cutoff_cannot_report_a_verdict_and_is_named(): + rows = _rows([0.9, np.nan, 0.9], [0.1, np.nan, 0.1], statuses=["ok", "fit_failed", "ok"]) + out = season_level_summary(rows, [2022], [2, 6, 12]) + assert out["gate_valid"] is False and out["v2_beats_legacy_spearman"] is None + assert out["failed_cutoffs"] == ["2022:6 (fit_failed)"] + + +def test_a_missing_legacy_number_cannot_report_a_verdict(): + out = season_level_summary(_rows([0.9, 0.9, 0.9], [0.1, np.nan, 0.1]), [2022], [2, 6, 12]) + assert out["v2_beats_legacy_spearman"] is None + assert any("legacy" in r for r in out["gate_invalid_reasons"]) + + +# --------------------------------------------------------------------------- +# Scoring against the real final standings +# --------------------------------------------------------------------------- + +def test_scores_against_the_whole_season_not_the_cutoff(): + dr = _fixture() + final = dr[dr["session_type"] == "R"].groupby("abbreviation")["points"].sum() + leader = final.idxmax() # VER, both rounds + drivers = pd.DataFrame({"driver": [leader, "PER"], "p_title": [0.75, 0.25], + "p10_final_points": [final[leader], 0.0], + "p90_final_points": [final[leader], 1.0]}) + _, constructors = _tables() + out = score_season_cutoff(drivers, constructors, dr, 2024, lineup=[leader, "PER"]) + assert out["brier_drivers_title"] == pytest.approx(0.25 ** 2 + 0.25 ** 2) + # leader's interval is exactly the true total (covered); PER's misses + assert out["coverage80_drivers"] == 0.5 + assert out["brier_constructors_title"] == 0.0 + + +def test_coverage_ignores_drivers_the_chain_did_not_simulate(): + dr = _fixture() + drivers = pd.DataFrame({"driver": ["VER", "XXX"], "p_title": [1.0, 0.0], + "p10_final_points": [0.0, 5.0], "p90_final_points": [100.0, 10.0]}) + out = score_season_cutoff(drivers, _tables()[1], dr, 2024, lineup=["VER"]) + assert out["coverage80_drivers"] == 1.0 + + +def test_championship_tables_carry_the_80pct_points_interval(): + pts = np.tile(np.arange(100.0), (2, 1)).T # 100 trials, two entries + d = drivers_championship(pts, np.zeros_like(pts), ["A", "B"], ["T", "T"], [0.0, 0.0]) + c = constructors_championship(pts, np.zeros_like(pts), ["T", "U"], [0.0, 0.0]) + for table in (d, c): + assert table["p10_final_points"].tolist() == pytest.approx([9.9, 9.9]) + assert table["p90_final_points"].tolist() == pytest.approx([89.1, 89.1]) + + +def test_schedule_is_read_off_the_rounds_that_ran_with_sprints_flagged(): + dr = _fixture() + sprint = dr[(dr["round"] == 2) & (dr["session_type"] == "R")].assign(session_type="S") + cal = remaining_calendar(2024, 0, loader=backtest_schedule(pd.concat([dr, sprint]))) + assert cal["round"].tolist() == [1, 2] + assert cal["has_sprint"].tolist() == [False, True] + + +# --------------------------------------------------------------------------- +# Legacy at the matching cutoff +# --------------------------------------------------------------------------- + +def test_legacy_trains_only_on_earlier_seasons(): + feats = _legacy_features(years=(2021, 2022, 2023, 2024)) + assert legacy_constructor_spearman(feats, 2023) == pytest.approx(1.0) + # poisoning a later season's target cannot move an earlier season's number + poisoned = feats.copy() + later = poisoned["year"] == 2024 + poisoned.loc[later, "season_points_share"] = -poisoned.loc[later, "early_points_share"] * 50 + assert legacy_constructor_spearman(poisoned, 2023) == legacy_constructor_spearman(feats, 2023) + + +def test_legacy_without_earlier_seasons_has_no_number(): + assert np.isnan(legacy_constructor_spearman(_legacy_features(years=(2024,)), 2024)) + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + +def test_cli_routes_season_level_to_the_season_harness(monkeypatch): + calls = {} + monkeypatch.setattr(backtest, "run_season_backtest", lambda **kw: calls.update(kw)) + monkeypatch.setattr(backtest, "run_backtest", lambda **kw: pytest.fail("race-level ran")) + args = cli.build_parser().parse_args(["backtest", "--season-level", "--seasons", "2022-2023"]) + args.func(args) + assert calls["seasons"] == [2022, 2023] and calls["cutoffs"] == [2, 6, 12] + assert cli.build_parser().parse_args(["backtest"]).season_level is False From 28615d96e2c79742769dfbdbfcac2eeb1aaff079 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 13:35:06 +0000 Subject: [PATCH 2/2] queue: slice-9 in-review (PR #22) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01JyCVWAmeHjqML4h9ugdbGQ --- docs/autopilot/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/autopilot/queue.md b/docs/autopilot/queue.md index c5df0e9..4054a44 100644 --- a/docs/autopilot/queue.md +++ b/docs/autopilot/queue.md @@ -423,7 +423,7 @@ Acceptance: ## slice-9: season-level backtest harness -Status: in-progress +Status: in-review — PR #22. Spec: §7 season-level metrics, §8 Phase 3, §4 (legacy benchmark).