diff --git a/AUTOPILOT.md b/AUTOPILOT.md index d421325..6fc9ea2 100644 --- a/AUTOPILOT.md +++ b/AUTOPILOT.md @@ -8,13 +8,13 @@ | Day (ET) | Slice | |---|---| -| Tue Oct 20 (4pm, one-off) | 7: Phase 2 carry-forwards (overtaking call-site test, per-circuit fit, Miami quali fallback) | +| Tue Oct 20 (afternoon, one-off) | 7: Phase 2 carry-forwards (overtaking call-site test, per-circuit fit, Miami quali fallback) | | Wed Oct 21 | 8: season chaining and `gp season` | | Fri Oct 23 | 9: season-level backtest harness (built and tested on the fixture, **not run**) | | **Sat Oct 24, 8pm** | **US GP (2026 R18) post-quali forecast.** One-off, runs from the `forecast-data` branch and posts to #19 | | Sun Oct 25 | 10: post-quali calibration | -Builds run around 9am ET, review around 2pm ET, and the digest around 7pm ET. Tuesday's one-off build gets a one-off 9pm review so Wednesday doesn't halt. +Builds run around 9am ET, review around 2pm ET, and the digest around 7pm ET. Tuesday's one-off build gets an extra 6:30pm review so Wednesday doesn't halt. ## Review in this order when you're back diff --git a/docs/autopilot/decisions.md b/docs/autopilot/decisions.md index 7f56a9b..4f0f438 100644 --- a/docs/autopilot/decisions.md +++ b/docs/autopilot/decisions.md @@ -148,3 +148,57 @@ Question: `RaceTrialInputs.dnf_prob` is the fold of both channels, so the res Chosen: Carry the channels as two *optional* fields beside `dnf_prob`, which stays what `simulate_positions` draws against. A caller holding only a combined probability leaves them None and gets the pre-slice-6 flat scaling rather than an invented split. `cli.forecast_weekend` fills them; `backtest.forecast_race_challenger` does not, because it resolves the Grand Prix alone through `simulate_positions` and has no sprint to price. Rejected: Making the channels the primary fields and deriving `dnf_prob` as a computed property. Tidier, and it removes the chance of the fold disagreeing with its parts — but a dataclass property cannot be passed to the constructor, so every existing `RaceTrialInputs(dnf_prob=...)` call site and four test modules would have to be rewritten, for no change in behaviour. Also rejected: a single `incident_share` scalar, which assumes one split for the whole field when the mechanical channel is per team and the incident channel per driver. Blast radius: Two fields on `RaceTrialInputs` and two keyword arguments in `cli.forecast_weekend`. The risk worth naming: `dnf_prob` and the channels are now two statements of the same quantity and nothing enforces that they agree — a caller could set channels inconsistent with the fold and the Grand Prix would use one while the sprint used the other. `tests/test_cli.py::test_both_channels_reach_the_resolver_and_fold_to_dnf_prob` asserts the production path keeps them consistent; the dataclass itself does not. + +## 2026-10-20 — slice-7 +Question: The criterion allows either fitting only the forecast round's circuit or caching the full fit per pre-round window. Which? +Chosen: Fit only the forecast circuit: `build_overtaking_data(..., event_name=...)`, passed by both call sites through a new `cli._round_event_name`. One fit per forecast, whatever the window holds. Omitting `event_name` still fits the whole window, so the slice-4 tests and any diagnostic caller are unchanged. +Rejected: A cache keyed on the pre-round window. Each backtest round has a different window, so a cache only helps when the same round is forecast twice in one process, and it adds invalidation questions (a mutated `driver_rounds` frame) for no gain on the backtest path the carry-forward was about. +Blast radius: One keyword on `build_overtaking_data`, two call sites. The fitted value for the forecast circuit is identical either way (`tests/test_overtaking_wiring.py::TestSingleCircuitFit` asserts equality with the full fit), so no forecast number moves. + +## 2026-10-20 — slice-7 +Question: How should `collect_v2` flag a quali row whose time came from the session laps (2025 Miami), and which laps count? +Chosen: `q_segment = "laps"` on the row plus a WARNING log naming the drivers. Per row: only a row with a `Position` and no Q1/Q2/Q3 time falls back, to that driver's fastest lap with `Deleted` laps excluded. Pole is still the official Q3→Q2→Q1 minimum wherever any official time exists; lap times set pole only when the session has none at all. +Rejected: A new `time_source` column. Clearer, but it changes the `driver_rounds.csv` schema (COLUMNS, the validator, every existing file) for one round in nine seasons; `q_segment` has no readers in `src/` or `tests/`, so overloading it is free. Also rejected: triggering only when the whole session lacks times. Simpler, but the criterion is per row, and a partial upstream gap would then still drop drivers silently. +Blast radius: `_fastest_session_laps` and the fallback block in `collect_v2._build_quali_rows`. Fully-timed sessions are byte-for-byte unchanged (the laps are never read; a test asserts it). Lap-derived times can order near-ties differently from the official classification (NOR/PIA/ANT in Miami sit within 0.002s); `finish_position` still comes from the results, so the grid is unaffected. + +## 2026-10-21 — slice-8 +Question: §3.4 says carry the team random walk forward. It does not say whether the other posterior terms (driver skill, noise scales, both DNF hazards) are redrawn per round or held for the season. +Chosen: One posterior draw per trial, held for every remaining round. Only `car` steps between rounds, by the pace model's own `SIGMA_DEV` (doubled for a reset year's first rounds, as in `_step_sigma_scale`). The incident hazard's grid slot is the season-start posterior-mean quali ranking (`cli._hazard_grid`, pre-weekend rule). +Rejected: Redrawing the parameters each round. That treats parameter uncertainty as round-to-round noise, so it averages out over a season and makes title odds too confident. +Blast radius: `sim.season.simulate_season`, `cli.forecast_season`. Nothing in `gp race` or the backtest changes. + +## 2026-10-21 — slice-8 +Question: Which line-up races the remaining rounds, and how are ties on points broken? +Chosen: The last completed round's entrants. Drivers who scored earlier but are no longer entered keep their points and score nothing more. Team current points come from the recorded points on R/S rows, so they follow whoever drove the car. Ties go to more Grand Prix wins, then column order. +Rejected: An explicit entry list (nothing in the repo maintains one) and the full FIA countback (second places onward). The countback adds a per-position tally for ties that fractional expected points almost never hit. +Blast radius: `cli.forecast_season`, `sim.season.final_positions`. + +## 2026-10-23 — slice-9 +Question: §7 asks for constructor-share Spearman "comparable to the legacy 0.88 ... match the cutoff". The 0.88 is a leave-one-season-out CV at `EARLY_ROUNDS=2`, so it trains on seasons *after* the one it scores. What legacy number goes next to v2's at each cutoff? +Chosen: A rolling legacy number per (season, cutoff): `features.build_features(early_rounds=cutoff)`, then `train.py`'s Ridge pipeline (`SimpleImputer` + `Ridge(alpha=1.0)`) on all `FEATURE_COLS`, trained on seasons strictly before the held-out one. It sees what the v2 fit sees and nothing later. The 0.88 is reported beside it as `legacy_reference_spearman`, never scored. If the legacy inputs (`data/race_results.csv`, `data/constructor_standings.csv`) can't be read, the cutoff has no legacy number and the verdict is null. +Rejected: Reproducing the 0.88 method (LOSO). It would let the legacy model learn from 2024-2025 when scoring 2022, which the rolling v2 fit cannot do, so the comparison would favour legacy for a reason that isn't skill. Also rejected: replaying `train.py`'s LassoCV selection, rule-change weight tuning and Ridge-vs-XGBoost pick per fold. More faithful, but it is a nested CV per (season, cutoff), and the shipped pkl picks among those settings with all seasons in view, so the tuned choice leaks too. +Blast radius: `backtest.legacy_constructor_spearman`, `backtest._legacy_features`. Only the legacy column of the season-level block. Swapping in another legacy method is one function. + +## 2026-10-23 — slice-9 +Question: How are the title Brier and the 80% coverage defined, and how is the held-out season's calendar supplied? +Chosen: Title Brier is the multi-class Brier, summed over the field (0 = certain and right, 2 = certain and wrong), not the per-driver mean the race-level `_brier` uses: a per-driver mean shrinks with field size and makes every title forecast look near-perfect. Coverage counts an entity when its real final points lie in `[p10_final_points, p90_final_points]` inclusive, two columns now added to both championship tables. Driver coverage is over the cutoff round's line-up only; a driver who had already left has a fixed total that any interval covers. The calendar is read off the held-out season's own R/S rows through `remaining_calendar`'s loader seam, and the chain is handed only rows up to the cutoff. +Rejected: The mean Brier, for the reason above. Also rejected: FastF1's schedule for past seasons. Network-bound, and it would list cancelled or uncollected rounds that have no actual result to score against. +Blast radius: `backtest.score_season_cutoff`, `backtest.backtest_schedule`, and two added columns on `drivers_championship.csv` / `constructors_championship.csv` (§6 columns unchanged). + +## 2026-10-23 — slice-9 +Question: Does `--season-level` run alongside the race-level backtest or instead of it, and where does its block go? +Chosen: Instead of. `gp backtest --season-level` runs only the season harness and writes `season_level` into `backtest_summary.json`. If that file already exists in `--out-dir` (a race-level run pointed at the same dir), the block is merged in and the race-level keys are kept. A cutoff past a season's last round, a failed fit, or a failed chain is a named failed cutoff, and any failed or unattempted cutoff, or a scored cutoff with no legacy number, makes `v2_beats_legacy_spearman` null. +Rejected: Running both in one invocation. The race-level grid is the slow part (~2.7 h with `--post-quali`) and the season harness is 12 fits for 2022-2025; coupling them makes the cheap one wait on the expensive one. +Blast radius: `cli.cmd_backtest` (a new branch, the old path untouched) and `backtest.run_season_backtest`. + +## 2026-10-25 — slice-10 +Question: The queue asks for "an expected calibration error (or similar single number, defined in the PR)" for both arms, but not which bins, which rounds, or what file shape. +Chosen: Ten equal-width bins on [0, 1] (p = 1.0 in the top bin), the same 0.1 bins `docs/phase2-status.md` already reads the pre-weekend file with. ECE = n-weighted mean of |mean predicted - observed podium rate| over the non-empty bins, in probability units. The summary gets `calibration_podium.pre` (every scored round), and with `--post-quali` also `postquali` and `pre_paired` (pre restricted to the rounds post-quali scored), so the two ECEs compare the same races. Both CSVs carry `season, round` after the original two columns, which is what makes `pre_paired` computable. +Rejected: Equal-count (quantile) bins. They would put different bin edges on each arm and make the two tables impossible to read side by side. Also rejected: comparing post-quali against the all-rounds pre table only, which would let a failed post-quali round change one side and not the other. +Blast radius: `backtest.calibration_summary`, `_podium_calibration`, `run_backtest`'s summary. `calibration_podium.csv` gains two trailing columns; the existing two keep their names and order. No forecast moves. + +## 2026-10-25 — slice-10 +Question: The queue asks for rounds where post-quali log loss is worse than pre-weekend "by more than a stated threshold", and leaves the threshold to the PR. +Chosen: 1.0 nats (`POSTQUALI_WORSE_THRESHOLD_NATS`), strictly greater. That is post-quali putting about 2.7x less probability on the actual winner than pre-weekend did. On the Oct 4 run it would include the three rounds `phase2-status.md` names (deltas 2.39, 2.24, 1.57). How many of the other 10 worse rounds clear it is unknown from the sandbox, which has no `backtest_metrics.csv`. The grid-pole driver and the actual winner are read off the race rows (`grid_position == 1`, `finish_position == 1`) and added to every row of `backtest_metrics.csv`. +Rejected: Any positive delta. It would list all 13 worse rounds, most of them noise. Also rejected: a threshold relative to the round's pre log loss, which is harder to read and not what the queue asked for. +Blast radius: One constant and `backtest.postquali_worse_rounds`; listing only, feeds no gate. diff --git a/docs/autopilot/queue.md b/docs/autopilot/queue.md index 1d28092..e821853 100644 --- a/docs/autopilot/queue.md +++ b/docs/autopilot/queue.md @@ -337,7 +337,26 @@ includes `--grid-from-quali` and the post-quali backtest arm. ## slice-7: Phase 2 carry-forwards -Status: todo +Status: merged — PR #20 merged by the review routine 2026-10-20 (`b14a307`). + +All four criteria were verified against the diff rather than against the PR +body. CI (`test`) was green on `3bd3b22`; the diff is 403 lines, inside the +600-line review cap. The review re-ran the full suite on the PR head +independently (Python 3.13, pymc 6.3.2, arviz 1.3.0): 297 passed. Five of the +PR's seven mutations were re-applied independently and each reproduced its +stated failure count: cli call site 3, backtest call site 3, `event_name` +ignored 6, no laps fallback 5, deleted laps counted 3. Nothing in the body was +found overstated. + +The backtest has not been run against this slice. `TestSingleCircuitFit` +asserts that the one-circuit fit equals that circuit's entry in the full fit, so +no forecast number should move. The Miami fallback is tested only against a +mocked session. Whether `repair_round.py 2025 6 Q` now matches the hand-repaired +rows is the owner's check (review pack step 4). + +Carried forward, not blocking: `q_segment="laps"` overloads a column that +currently has no readers. A future reader that treats `q_segment` as Q1/Q2/Q3 +has to handle the flag (decision 2, 2026-10-20). Spec: `docs/phase2-status.md` "What is still open"; slice-4 notes above. @@ -360,7 +379,24 @@ Acceptance: ## slice-8: season chaining and `gp season` -Status: todo +Status: merged — PR #21 merged by the review routine 2026-10-21 (`e69f8c3`). + +All six criteria were verified against the diff rather than against the PR +body. CI (`test`) was green on `503aded`; the diff is 594 lines, inside the +600-line review cap, with the CLI wiring included. The review re-ran the full +suite on the PR head independently (Python 3.11, fresh venv): 309 passed. +Three mutations were applied independently: stepping from the fitted car +instead of the running one (1 failed), dropping one driver per team from the +constructor sum (2 failed), and a loader that returns nothing (5 failed). + +The backtest has not been run against this slice, and the real +`fastf1.get_event_schedule` path is untested (every test stubs the loader). +Review pack step 4 (`gp season` on a real fit, check `current_points` against +the real 2026 standings) is the owner's check. + +Carried forward, not blocking: `final_positions` orders on +`points * 100 + wins`, which assumes points come in steps of at least 0.5 and +wins stay under 50. True for every current points table. Spec: `docs/grid-prophet-v2-spec.md` §3.4, §6 (`gp season`, the two championship CSVs), §8 Phase 3. @@ -387,7 +423,25 @@ Acceptance: ## slice-9: season-level backtest harness -Status: todo +Status: merged — PR #22 merged by the review routine 2026-10-23 (`9bef025`). + +All four criteria were verified against the diff rather than against the PR +body. CI (`test`) was green on `28615d9`; the diff is 556 lines, inside the +600-line review cap. The review re-ran the full suite on the PR head +independently (Python 3.11, fresh venv): 326 passed. Five mutations were +applied independently, and each one failed a test: the chain is handed the full +`driver_rounds` (a leak), failed cutoffs no longer void the gate, the title +Brier is a mean instead of a sum, the legacy model trains on every other season +(LOSO), and the CLI ignores `--cutoffs`. + +**The real season-level backtest has not been run.** That is the Phase 3 gate, +and only the owner runs it (PR #22, review pack step 3). The legacy column is a +rolling Ridge on earlier seasons, not the 0.88 LOSO method (decision +2026-10-23), so expect it to come in below 0.88. + +Carried forward, not blocking: the fixture only has 2024 R1-R2, so the +end-to-end test runs at cutoff 1. The default 2/6/12 cutoffs have only been +exercised as `no_rounds_after_cutoff` failures. Spec: §7 season-level metrics, §8 Phase 3, §4 (legacy benchmark). @@ -410,7 +464,7 @@ Acceptance: ## slice-10: post-quali calibration -Status: todo +Status: merged — PR #23, squash-merged into `auto/queue` as c9ae98b by the review routine 2026-10-25. Spec: `docs/phase2-status.md` caveat 1 ("Calibration was not re-checked for post-quali"). diff --git a/src/backtest.py b/src/backtest.py index b1c15d0..86673e0 100644 --- a/src/backtest.py +++ b/src/backtest.py @@ -49,6 +49,13 @@ COARSE_ROUNDS = [2, 4, 6, 9, 12, 16, 20] EPS = 1e-3 +# Podium calibration bins: ten equal-width bins on [0, 1], the same 0.1 bins +# docs/phase2-status.md reads the pre-weekend file with. +CALIBRATION_N_BINS = 10 +# A round is listed as a post-quali "confident miss" when its winner log loss is +# worse than pre-weekend's by more than this many nats (~2.7x less probability +# on the actual winner). Listing only; it feeds no verdict. +POSTQUALI_WORSE_THRESHOLD_NATS = 1.0 def _log_loss_winner(p_win: dict, actual_winner: str) -> float: @@ -139,8 +146,11 @@ def score_round(forecast: pd.DataFrame, driver_rounds: pd.DataFrame, season: int p_win_pole = _pole_wins_baseline(actual) p_win_pace = _pace_rating_baseline(driver_rounds, actual, season, round_) + pole = actual[actual["grid_position"] == 1] result = { "season": season, "round": round_, "n_drivers": len(actual), + "actual_winner": actual_winner, + "grid_pole_driver": pole.iloc[0]["abbreviation"] if not pole.empty else None, "log_loss_winner_model": _log_loss_winner(p_win_model, actual_winner), "log_loss_winner_pole": _log_loss_winner(p_win_pole, actual_winner), "log_loss_winner_pace_rating": _log_loss_winner(p_win_pace, actual_winner), @@ -182,7 +192,7 @@ def forecast_race_challenger(challenger_model, reliability_data, driver_rounds: combined_dnf_prob, sample_incident_dnf_prob, sample_mechanical_dnf_prob, ) from sim.race import RaceTrialInputs, simulate_positions, summarize_trials, load_points_table - from cli import _entrants_for_round, _circuit_info + from cli import _entrants_for_round, _circuit_info, _round_event_name from model.overtaking import build_overtaking_data rng = np.random.default_rng(seed) @@ -190,7 +200,8 @@ def forecast_race_challenger(challenger_model, reliability_data, driver_rounds: # Same fitted overtaking parameter the Bayesian path uses, off the same # pre-round window -- "same race resolver" above has to stay true. overtaking_data = build_overtaking_data(driver_rounds, circuits, - before_season=season, before_round=round_) + before_season=season, before_round=round_, + event_name=_round_event_name(driver_rounds, season, round_)) circuit_type, overtaking_difficulty = _circuit_info(circuits, driver_rounds, season, round_, overtaking_data) @@ -248,6 +259,64 @@ def _calibration_bins(forecast: pd.DataFrame, actual: pd.DataFrame, col: str, th return out +CALIBRATION_COLUMNS = ["p_podium_pred", "podium_actual", "season", "round"] + + +def _podium_calibration(forecast: pd.DataFrame, actual: pd.DataFrame, season: int, + round_: int) -> list[tuple]: + """One (p_podium_pred, podium_actual, season, round) row per scored driver. + Both arms write this shape, so their files can be compared row for row.""" + return [(p, hit, season, round_) for p, hit in _calibration_bins( + forecast, actual, "p_podium", lambda a: set(a[a["finish_position"] <= 3]["abbreviation"]))] + + +def calibration_summary(calib: pd.DataFrame, n_bins: int = CALIBRATION_N_BINS) -> dict: + """Binned reliability table and expected calibration error for one arm. + + Bins are equal-width on [0, 1] (p = 1.0 falls in the top bin); empty bins + are left out of the table. ECE is the n-weighted mean of + |mean predicted - observed rate| over the non-empty bins: 0 is perfectly + calibrated, and it is in probability units, so 0.02 means the forecast is + off by two points on average where its predictions actually fall.""" + if calib.empty: + return {"n": 0, "ece": None, "bins": []} + p = calib["p_podium_pred"].astype(float).to_numpy() + hit = calib["podium_actual"].astype(bool).to_numpy() + idx = np.clip(np.floor(p * n_bins).astype(int), 0, n_bins - 1) + bins, ece = [], 0.0 + for b in range(n_bins): + mask = idx == b + n = int(mask.sum()) + if not n: + continue + mean_pred, rate = float(p[mask].mean()), float(hit[mask].mean()) + ece += n / len(p) * abs(mean_pred - rate) + bins.append({"bin_lo": b / n_bins, "bin_hi": (b + 1) / n_bins, "n": n, + "mean_pred": mean_pred, "observed_rate": rate}) + return {"n": len(p), "ece": float(ece), "bins": bins} + + +def postquali_worse_rounds(metrics_df: pd.DataFrame, + threshold: float = POSTQUALI_WORSE_THRESHOLD_NATS) -> list[dict]: + """Rounds where post-quali winner log loss is worse than pre-weekend's by + more than `threshold` nats, worst first, with the grid-pole driver and the + actual winner. Raw material for "do the confident misses share a cause"; + no cause is inferred here.""" + cols = {"log_loss_winner_model", "log_loss_winner_postquali", "season", "round"} + if not cols <= set(metrics_df.columns): + return [] + both = metrics_df.dropna(subset=["log_loss_winner_postquali"]) + delta = both["log_loss_winner_postquali"] - both["log_loss_winner_model"] + worse = both.assign(delta=delta)[delta > threshold].sort_values("delta", ascending=False) + return [{"season": int(r["season"]), "round": int(r["round"]), + "log_loss_pre": float(r["log_loss_winner_model"]), + "log_loss_postquali": float(r["log_loss_winner_postquali"]), + "delta": float(r["delta"]), + "grid_pole_driver": r.get("grid_pole_driver"), + "actual_winner": r.get("actual_winner")} + for _, r in worse.iterrows()] + + POST_QUALI_METRICS = ("log_loss_winner", "brier_podium", "brier_points", "spearman") @@ -298,6 +367,8 @@ def post_quali_summary(metrics_df: pd.DataFrame, conditioning_log: pd.DataFrame, out["gate_invalid_reasons"] = reasons out["postquali_beats_pre_log_loss"] = (bool(out["postquali_log_loss_delta"] < 0) if not reasons else None) + out["postquali_worse_threshold_nats"] = POSTQUALI_WORSE_THRESHOLD_NATS + out["postquali_worse_rounds"] = postquali_worse_rounds(both) if len(conditioning_log): from model.conditioning import fallback_rate, regressions out["fallback_rate"] = fallback_rate(conditioning_log) @@ -309,13 +380,15 @@ def post_quali_summary(metrics_df: pd.DataFrame, conditioning_log: pd.DataFrame, def _post_quali_arm(prior_idata, driver_rounds, driver_meta, circuits, reliability_data, season: int, round_: int, n_trials: int, chains: int, - conditioning_records: list, outdir: str) -> dict: + conditioning_records: list, outdir: str, + calibration_rows: list | None = None) -> dict: """Condition on round `round_`'s quali (warm-started NUTS, spec sec 6), forecast from the real grid, score it. Always returns a `postquali_status` -- "ok" with the `*_postquali` metric keys, or "conditioning_failed" / "forecast_failed" / "unscored" with nothing else. A failure is printed and counted by `post_quali_summary`; the round has no post-quali score, never a - silent zero. Only a round that conditioned writes a conditioning record.""" + silent zero. Only a round that conditioned writes a conditioning record, and + only a scored round adds rows to `calibration_rows`.""" from dataclasses import asdict from cli import forecast_race from model.conditioning import append_record, condition_on_quali @@ -341,6 +414,9 @@ def _post_quali_arm(prior_idata, driver_rounds, driver_meta, circuits, reliabili if scored is None: print(f" WARNING: post-quali forecast for {season}:{round_} could not be scored") return {"postquali_status": "unscored"} + if calibration_rows is not None: + calibration_rows.extend(_podium_calibration( + forecast, _actual_outcomes(driver_rounds, season, round_), season, round_)) out = {f"{m}_postquali": scored[f"{m}_model"] for m in POST_QUALI_METRICS} out["postquali_status"] = "ok" return out @@ -361,6 +437,7 @@ def run_backtest(seasons: list[int], n_trials: int = 5000, method: str = "advi", results = [] calibration_rows = [] + calibration_rows_postquali = [] conditioning_records = [] t_start = time.time() @@ -446,22 +523,19 @@ def run_backtest(seasons: list[int], n_trials: int = 5000, method: str = "advi", metrics.update(_post_quali_arm( idata, driver_rounds, driver_meta, circuits, reliability_data, season, rr, n_trials, post_quali_chains, conditioning_records, outdir, + calibration_rows=calibration_rows_postquali, )) if metrics is not None: metrics["fit_elapsed_s"] = fit_elapsed results.append(metrics) - actual = _actual_outcomes(driver_rounds, season, rr) - calibration_rows.extend( - (p, hit) for p, hit in _calibration_bins( - forecast, actual, "p_podium", lambda a: set(a[a["finish_position"] <= 3]["abbreviation"]) - ) - ) + calibration_rows.extend(_podium_calibration( + forecast, _actual_outcomes(driver_rounds, season, rr), season, rr)) total_elapsed = time.time() - t_start print(f"[backtest] total elapsed: {total_elapsed:.1f}s ({total_elapsed / 60:.1f} min)") metrics_df = pd.DataFrame(results) - calib_df = pd.DataFrame(calibration_rows, columns=["p_podium_pred", "podium_actual"]) + calib_df = pd.DataFrame(calibration_rows, columns=CALIBRATION_COLUMNS) os.makedirs(outdir, exist_ok=True) metrics_df.to_csv(os.path.join(outdir, "backtest_metrics.csv"), index=False) @@ -470,6 +544,7 @@ def run_backtest(seasons: list[int], n_trials: int = 5000, method: str = "advi", summary = { "seasons": seasons, "n_rounds_scored": len(metrics_df), "rounds_mode": rounds, "method": method, "n_trials": n_trials, "total_elapsed_s": total_elapsed, + "calibration_podium": {"pre": calibration_summary(calib_df)}, } if not metrics_df.empty: summary.update({ @@ -500,9 +575,273 @@ def run_backtest(seasons: list[int], n_trials: int = 5000, method: str = "advi", summary["post_quali"] = True summary.update(post_quali_summary(metrics_df, pd.DataFrame(conditioning_records), rounds_mode=rounds)) + calib_post = pd.DataFrame(calibration_rows_postquali, columns=CALIBRATION_COLUMNS) + calib_post.to_csv(os.path.join(outdir, "calibration_podium_postquali.csv"), index=False) + # pre restricted to the rounds post-quali scored, so the two ECEs are + # over the same races and differ only in the forecast + paired = calib_df.merge(calib_post[["season", "round"]].drop_duplicates(), + on=["season", "round"]) + summary["calibration_podium"].update({ + "pre_paired": calibration_summary(paired), + "postquali": calibration_summary(calib_post), + }) with open(os.path.join(outdir, "backtest_summary.json"), "w") as f: json.dump(summary, f, indent=2) print(json.dumps(summary, indent=2)) print(f"Saved -> {outdir}") return metrics_df + + +# --------------------------------------------------------------------------- +# Season-level backtest (spec sec 7 season-level metrics, sec 8 Phase 3) +# --------------------------------------------------------------------------- + +SEASON_CUTOFFS = (2, 6, 12) +# Legacy leave-one-season-out constructor-share Spearman at EARLY_ROUNDS=2 +# (spec sec 4). Reported beside the rolling legacy number, never scored. +LEGACY_REFERENCE_SPEARMAN = 0.88 + + +def backtest_schedule(driver_rounds: pd.DataFrame): + """A `sim.season.remaining_calendar` loader that reads a held-out season's + calendar off the rounds that actually ran, in FastF1's column names. A + round is a sprint weekend if it has S rows (as `cli._round_has_sprint`).""" + def loader(season: int) -> pd.DataFrame: + rows = driver_rounds[driver_rounds["season"] == season] + races = rows[rows["session_type"] == "R"].drop_duplicates("round").sort_values("round") + sprints = set(rows.loc[rows["session_type"] == "S", "round"]) + return pd.DataFrame({ + "RoundNumber": races["round"].astype(int).to_numpy(), + "EventName": races["event_name"].astype(str).to_numpy(), + "EventFormat": ["sprint_qualifying" if r in sprints else "conventional" + for r in races["round"]], + }) + return loader + + +def _champion(points: dict, wins: dict) -> str | None: + """Most points, ties to more Grand Prix wins (as `sim.season.final_positions`).""" + if not points: + return None + return max(points, key=lambda k: (points[k], wins.get(k, 0))) + + +def _title_brier(p_title: dict, champion: str | None) -> float: + """Multi-class Brier on the title winner: summed over the field, 0 to 2.""" + if champion is None: + return float("nan") + p = dict(p_title) + p.setdefault(champion, 0.0) + return float(sum((v - (1.0 if k == champion else 0.0)) ** 2 for k, v in p.items())) + + +def _coverage80(table: pd.DataFrame, key: str, actual: dict, among=None) -> float: + rows = table if among is None else table[table[key].isin(among)] + hits = [r.p10_final_points <= actual.get(getattr(r, key), 0.0) <= r.p90_final_points + for r in rows.itertuples(index=False)] + return float(np.mean(hits)) if hits else float("nan") + + +def _share_spearman(predicted: dict, actual: dict) -> float: + common = [k for k in predicted if k in actual] + if len(common) < 3: + return float("nan") + rho, _ = spearmanr([predicted[k] for k in common], [actual[k] for k in common]) + return float(rho) + + +def score_season_cutoff(drivers: pd.DataFrame, constructors: pd.DataFrame, + driver_rounds: pd.DataFrame, season: int, lineup: list) -> dict: + """Score one cutoff's championship tables against the season's real final + standings (spec sec 7). Driver coverage is over `lineup`, the drivers the + chain actually simulated: a driver who had already left keeps a fixed + total, which an interval covers trivially.""" + from cli import _season_so_far + + last = int(driver_rounds.loc[driver_rounds["season"] == season, "round"].max()) + drv_pts, team_pts, drv_wins, team_wins = _season_so_far(driver_rounds, season, last) + total = sum(team_pts.values()) + actual_share = {t: p / total for t, p in team_pts.items()} if total > 0 else {} + return { + "constructor_spearman_v2": _share_spearman( + dict(zip(constructors["team"], constructors["exp_share"])), actual_share), + "brier_drivers_title": _title_brier(dict(zip(drivers["driver"], drivers["p_title"])), + _champion(drv_pts, drv_wins)), + "brier_constructors_title": _title_brier(dict(zip(constructors["team"], constructors["p_title"])), + _champion(team_pts, team_wins)), + "coverage80_drivers": _coverage80(drivers, "driver", drv_pts, among=lineup), + "coverage80_constructors": _coverage80(constructors, "team", team_pts), + } + + +def legacy_constructor_spearman(features: pd.DataFrame, season: int) -> float: + """The legacy season-level model at a matching cutoff: `features` is + `features.build_features(early_rounds=cutoff)`. Trained on seasons + strictly before `season`, the same information the rolling v2 fit has, + with `train.py`'s Ridge pipeline on its full feature set.""" + from sklearn.impute import SimpleImputer + from sklearn.linear_model import Ridge + from sklearn.pipeline import Pipeline + from train import FEATURE_COLS + + known = features[features["season_points_share"].notna()] + train, test = known[known["year"] < season], known[known["year"] == season] + if train.empty or len(test) < 3: + return float("nan") + model = Pipeline([("imp", SimpleImputer()), ("reg", Ridge(alpha=1.0))]) + model.fit(train[FEATURE_COLS], train["season_points_share"]) + rho, _ = spearmanr(model.predict(test[FEATURE_COLS]), test["season_points_share"]) + return float(rho) + + +def _legacy_features(cutoff: int) -> pd.DataFrame: + from features import build_features + + return build_features(early_rounds=cutoff) + + +def season_level_summary(rows: pd.DataFrame, seasons: list, cutoffs: list) -> dict: + """The season-level block of `backtest_summary.json`. Same discipline as + `post_quali_summary`: every requested (season, cutoff) is accounted for, + and `v2_beats_legacy_spearman` is None unless the run is complete -- every + cutoff scored and a legacy number beside each one.""" + out: dict = {"seasons": list(seasons), "cutoffs": list(cutoffs), + "legacy_reference_spearman": LEGACY_REFERENCE_SPEARMAN, + "n_cutoffs_expected": len(seasons) * len(cutoffs)} + ok = rows[rows["status"] == "ok"] if len(rows) else rows + out["n_cutoffs_ok"] = len(ok) + attempted = {(int(r.season), int(r.cutoff)) for r in rows.itertuples()} if len(rows) else set() + missing = [f"{s}:{c} (not attempted)" for s in seasons for c in cutoffs if (s, c) not in attempted] + failed = missing + [f"{int(r.season)}:{int(r.cutoff)} ({r.status})" + for r in rows.itertuples() if r.status != "ok"] + out["failed_cutoffs"] = failed + + metrics = ["constructor_spearman_v2", "constructor_spearman_legacy", "brier_drivers_title", + "brier_constructors_title", "coverage80_drivers", "coverage80_constructors"] + per_cutoff = {} + for c in cutoffs: + at = ok[ok["cutoff"] == c] if len(ok) else ok + per_cutoff[str(c)] = {"n_seasons": len(at), + **{f"mean_{m}": (float(at[m].mean()) if len(at) and at[m].notna().any() + else None) for m in metrics}} + out["per_cutoff"] = per_cutoff + + no_legacy = (ok[ok["constructor_spearman_legacy"].isna()] if len(ok) else ok) + reasons = [] + if failed: + reasons.append(f"{len(failed)} of {out['n_cutoffs_expected']} cutoff(s) did not score") + if len(no_legacy): + reasons.append(f"{len(no_legacy)} scored cutoff(s) have no legacy number to compare against") + if len(ok) and ok["constructor_spearman_v2"].isna().any(): + reasons.append("a scored cutoff has no v2 Spearman (fewer than 3 teams in common)") + if not len(ok): + reasons.append("nothing scored") + out["gate_valid"] = not reasons + out["gate_invalid_reasons"] = reasons + if len(ok): + out["mean_constructor_spearman_v2"] = float(ok["constructor_spearman_v2"].mean()) + out["mean_constructor_spearman_legacy"] = (float(ok["constructor_spearman_legacy"].mean()) + if ok["constructor_spearman_legacy"].notna().any() + else None) + out["v2_beats_legacy_spearman"] = (bool(out["mean_constructor_spearman_v2"] + > out["mean_constructor_spearman_legacy"]) + if not reasons else None) + return out + + +def run_season_backtest(seasons: list[int], cutoffs=SEASON_CUTOFFS, n_trials: int = 5000, + method: str = "advi", advi_steps: int = 4000, advi_draws: int = 300, + out_dir: str | None = None, driver_rounds: pd.DataFrame | None = None, + driver_meta: pd.DataFrame | None = None, circuits: pd.DataFrame | None = None, + legacy_features=_legacy_features) -> pd.DataFrame: + """For each held-out season and cutoff: fit through `season:cutoff`, chain + the rest of that season with `cli.forecast_season`, and score the tables + against how the season really ended. The chain sees only rows up to the + cutoff; the real calendar is read off the season's own rows.""" + from cli import _entrants_for_round, _load_circuits, forecast_season + from model.pace import build_pace_data, fit_advi, fit_nuts + from model.reliability import build_reliability_data + + if driver_rounds is None: + driver_rounds = pd.read_csv(os.path.join(DATA_DIR, "driver_rounds.csv")) + meta_path = os.path.join(DATA_DIR, "driver_meta.csv") + driver_meta = pd.read_csv(meta_path) if os.path.exists(meta_path) else None + circuits = _load_circuits() if circuits is None else circuits + outdir = out_dir or os.path.join(OUTPUTS_DIR, f"backtest_{datetime.now():%Y%m%d_%H%M%S}") + os.makedirs(outdir, exist_ok=True) + loader = backtest_schedule(driver_rounds) + legacy_cache: dict = {} + rows = [] + t_start = time.time() + + for season in seasons: + season_rounds = set(driver_rounds.loc[(driver_rounds["season"] == season) + & (driver_rounds["session_type"] == "R"), "round"]) + for cutoff in cutoffs: + row = {"season": season, "cutoff": cutoff} + rows.append(row) + if cutoff not in season_rounds or cutoff >= max(season_rounds): + row["status"] = "no_rounds_after_cutoff" + continue + t0 = time.time() + print(f"[season-backtest] {season}: fitting through {season}:{cutoff} ({method}) ...") + try: + data = build_pace_data(driver_rounds, driver_meta, through_season=season, + through_round=cutoff) + idata = (fit_advi(data, n=advi_steps, draws=advi_draws) if method == "advi" + else fit_nuts(data, draws=250, tune=250, chains=1)) + reliability = build_reliability_data(driver_rounds, through_season=season, + through_round=cutoff) + except Exception as exc: + print(f" WARNING: fit failed for {season}:{cutoff}: {exc}") + row["status"] = "fit_failed" + continue + seen = driver_rounds[(driver_rounds["season"] < season) + | ((driver_rounds["season"] == season) & (driver_rounds["round"] <= cutoff))] + try: + drivers, constructors = forecast_season(idata, data, reliability, seen, circuits, + n_trials=n_trials, seed=cutoff, + schedule_loader=loader) + except Exception as exc: + print(f" WARNING: season chain failed for {season}:{cutoff}: {exc}") + row["status"] = "forecast_failed" + continue + for name, table in (("drivers", drivers), ("constructors", constructors)): + table.to_csv(os.path.join(outdir, f"season_{season}_r{cutoff}_{name}_championship.csv"), + index=False) + lineup = _entrants_for_round(seen, season, cutoff)["abbreviation"].tolist() + row.update(score_season_cutoff(drivers, constructors, driver_rounds, season, lineup)) + + if cutoff not in legacy_cache: + try: + legacy_cache[cutoff] = legacy_features(cutoff) + except Exception as exc: + print(f" WARNING: legacy features unavailable at cutoff {cutoff}: {exc}") + legacy_cache[cutoff] = None + feats = legacy_cache[cutoff] + row["constructor_spearman_legacy"] = (legacy_constructor_spearman(feats, season) + if feats is not None else float("nan")) + row["status"] = "ok" + row["elapsed_s"] = time.time() - t0 + + rows_df = pd.DataFrame(rows) + for col in ("constructor_spearman_v2", "constructor_spearman_legacy"): + if col not in rows_df.columns: + rows_df[col] = float("nan") + rows_df.to_csv(os.path.join(outdir, "season_backtest_metrics.csv"), index=False) + + block = season_level_summary(rows_df, seasons, list(cutoffs)) + block.update({"method": method, "n_trials": n_trials, "total_elapsed_s": time.time() - t_start}) + # Join a race-level summary already in this --out-dir rather than overwrite it. + summary_path = os.path.join(outdir, "backtest_summary.json") + summary = {} + if os.path.exists(summary_path): + with open(summary_path) as f: + summary = json.load(f) + summary["season_level"] = block + with open(summary_path, "w") as f: + json.dump(summary, f, indent=2) + print(json.dumps(block, indent=2)) + print(f"Saved -> {outdir}") + return rows_df diff --git a/src/cli.py b/src/cli.py index b6b103c..8175fd8 100644 --- a/src/cli.py +++ b/src/cli.py @@ -6,7 +6,7 @@ Implemented: `fit`, `race --mode pre`, `race --mode post-quali` (Phase 2, spec sec 6 conditioning), sprint weekends (sec 3.4), and `backtest` (the -sec 7 evaluation). `season` is Phase 3. +sec 7 evaluation), and `season` (sec 3.4 chaining, Phase 3). """ from __future__ import annotations @@ -243,6 +243,12 @@ def _quali_grid_for_round(driver_rounds: pd.DataFrame, season: int, round_: int, return values.rank(method="first", na_option="bottom").to_numpy(dtype=int) +def _round_event_name(driver_rounds: pd.DataFrame, season: int, round_: int): + """The round's event name, or None when the round has no rows.""" + rows = driver_rounds[(driver_rounds["season"] == season) & (driver_rounds["round"] == round_)] + return rows.iloc[0]["event_name"] if not rows.empty else None + + def _circuit_info(circuits: pd.DataFrame, driver_rounds: pd.DataFrame, season: int, round_: int, overtaking_data=None) -> tuple[str, float]: """(circuit_type, overtaking_difficulty) for one round. @@ -257,7 +263,7 @@ def _circuit_info(circuits: pd.DataFrame, driver_rounds: pd.DataFrame, season: i rows = driver_rounds[(driver_rounds["season"] == season) & (driver_rounds["round"] == round_)] if rows.empty: return "mixed", DEFAULT_DIFFICULTY - event_name = rows.iloc[0]["event_name"] + event_name = _round_event_name(driver_rounds, season, round_) circuit_type = rows.iloc[0]["circuit_type"] if pd.notna(rows.iloc[0]["circuit_type"]) else "mixed" match = circuits[circuits["event_name"] == event_name] hand_rating = float(match.iloc[0]["overtaking_difficulty"]) if not match.empty else None @@ -303,7 +309,8 @@ def forecast_weekend(idata, data, reliability_data, driver_rounds: pd.DataFrame, # model on a parameter that already saw the answer. if overtaking_data is None: overtaking_data = build_overtaking_data(driver_rounds, circuits, - before_season=season, before_round=round_) + before_season=season, before_round=round_, + event_name=_round_event_name(driver_rounds, season, round_)) circuit_type, overtaking_difficulty = _circuit_info(circuits, driver_rounds, season, round_, overtaking_data) round_idx = data.round_to_idx.get((season, round_)) # None for a genuinely future round -> use latest @@ -462,6 +469,152 @@ def cmd_race(args): print(f"Saved -> {sprint_path}") +# --------------------------------------------------------------------------- +# gp season +# --------------------------------------------------------------------------- + +def _season_so_far(driver_rounds: pd.DataFrame, season: int, through_round: int): + """Points and Grand Prix wins already scored in `season` up to and + including `through_round`, as ({driver: pts}, {team: pts}, {driver: wins}, + {team: wins}). Read from the recorded `points` of R and S rows, so team + totals follow whoever actually drove the car that weekend.""" + run = driver_rounds[(driver_rounds["season"] == season) & (driver_rounds["round"] <= through_round) + & (driver_rounds["session_type"].isin(["R", "S"]))] + pts = run.assign(points=run["points"].fillna(0.0)) + won = run[(run["session_type"] == "R") & (run["finish_position"] == 1)] + return (pts.groupby("abbreviation")["points"].sum().to_dict(), + pts.groupby("team")["points"].sum().to_dict(), + won.groupby("abbreviation").size().to_dict(), + won.groupby("team").size().to_dict()) + + +def forecast_season(idata, data, reliability_data, driver_rounds: pd.DataFrame, circuits: pd.DataFrame, + n_trials: int = 10_000, seed: int = 0, schedule_loader=None): + """Chain the rest of the fitted season (spec sec 3.4). + Returns `(drivers_championship, constructors_championship)`. + + The season and the last completed round are the fit's own cutoff, so the + chain starts exactly where the posterior ends. Entrants for the remaining + rounds are the last completed round's line-up.""" + from model.overtaking import build_overtaking_data, circuit_difficulty + from model.pace import CIRCUIT_TYPES + from model.reliability import sample_incident_dnf_prob, sample_mechanical_dnf_prob + from sim.race import load_points_table + from sim.season import ( + SeasonRound, SeasonTrialInputs, constructors_championship, drivers_championship, + remaining_calendar, simulate_season, step_sigma_for_round, + ) + + season, through = max(data.round_to_idx, key=data.round_to_idx.get) + calendar = remaining_calendar(season, through, loader=schedule_loader) + rng = np.random.default_rng(seed) + + entrants = _entrants_for_round(driver_rounds, season, through) + driver_ids, teams = entrants["abbreviation"].tolist(), entrants["team"].tolist() + drv_pts, team_pts, drv_wins, team_wins = _season_so_far(driver_rounds, season, through) + team_names = sorted(set(teams) | set(team_pts)) + + post = idata.posterior + n_posterior = post.sizes["chain"] * post.sizes["draw"] + draw_idx = rng.integers(0, n_posterior, size=n_trials) + + def flat(name): + arr = np.asarray(post[name]) + return arr.reshape(n_posterior, *arr.shape[2:])[draw_idx] + + car_all = np.asarray(post["car"].isel(round=data.round_to_idx[(season, through)]) + .transpose("chain", "draw", "team")).reshape(n_posterior, -1)[draw_idx] + car0_mean = flat("car0").mean(axis=1) + fitted_teams = list(post["car"].coords["team"].values) + car = np.column_stack([car_all[:, fitted_teams.index(t)] if t in fitted_teams else car0_mean + for t in team_names]) + + fitted_drivers = list(post["driver_skill"].coords["driver"].values) + skill_all, offset_all, track_all = flat("driver_skill"), flat("quali_offset"), flat("driver_track") + n_drivers = len(driver_ids) + skill, offset = np.zeros((n_trials, n_drivers)), np.zeros((n_trials, n_drivers)) + track = np.zeros((n_trials, n_drivers, len(CIRCUIT_TYPES))) + for i, d in enumerate(driver_ids): + if d in fitted_drivers: # an unseen driver keeps the field-average zeros + j = fitted_drivers.index(d) + skill[:, i], offset[:, i], track[:, i] = skill_all[:, j], offset_all[:, j], track_all[:, j] + + # Hazards are posterior parameters: one draw per trial, held for the season. + team_mech = {t: sample_mechanical_dnf_prob(reliability_data, t, season, n_trials, rng) + for t in sorted(set(teams))} + mech = np.column_stack([team_mech[t] for t in teams]) + quali_pace = car[:, [team_names.index(t) for t in teams]] + skill + offset + hazard_grid = _hazard_grid(None, quali_pace) + incident = np.column_stack([sample_incident_dnf_prob(reliability_data, d, hazard_grid[i], n_trials, rng) + for i, d in enumerate(driver_ids)]) + + overtaking = build_overtaking_data(driver_rounds, circuits, before_season=season, before_round=through + 1) + rounds = [] + for row in calendar.itertuples(index=False): + match = circuits[circuits["event_name"] == row.event_name] + ctype = match.iloc[0]["circuit_type"] if not match.empty else "mixed" + hand = float(match.iloc[0]["overtaking_difficulty"]) if not match.empty else None + rounds.append(SeasonRound( + round=int(row.round), has_sprint=bool(row.has_sprint), + circuit_type_idx=CIRCUIT_TYPES.index(ctype) if ctype in CIRCUIT_TYPES else 0, + overtaking_difficulty=circuit_difficulty(overtaking, row.event_name, hand), + step_sigma=step_sigma_for_round(season, int(row.round)))) + + inputs = SeasonTrialInputs( + driver_ids=driver_ids, teams=teams, team_names=team_names, car=car, + driver_skill=skill, driver_track=track, quali_offset=offset, + race_noise_nu=flat("race_nu"), race_noise_sigma=flat("race_sigma"), + quali_noise_sigma=flat("quali_sigma"), + mechanical_dnf_prob=mech, incident_dnf_prob=incident) + outcome = simulate_season(inputs, rounds, load_points_table("grand_prix", season), + load_points_table("sprint", season), rng) + + # Drivers who scored this season but are not on the current line-up keep + # their points and score nothing more. + all_drivers = driver_ids + sorted(set(drv_pts) - set(driver_ids)) + last_team = (driver_rounds[driver_rounds["season"] == season] + .sort_values("round").groupby("abbreviation")["team"].last()) + all_teams = teams + [last_team[d] for d in all_drivers[n_drivers:]] + future = np.zeros((n_trials, len(all_drivers))) + future_wins = np.zeros_like(future) + future[:, :n_drivers], future_wins[:, :n_drivers] = outcome.driver_points, outcome.driver_wins + cur_d = np.array([drv_pts.get(d, 0.0) for d in all_drivers]) + cur_dw = np.array([drv_wins.get(d, 0) for d in all_drivers]) + cur_t = np.array([team_pts.get(t, 0.0) for t in team_names]) + cur_tw = np.array([team_wins.get(t, 0) for t in team_names]) + drivers = drivers_championship(cur_d + future, cur_dw + future_wins, all_drivers, all_teams, cur_d) + constructors = constructors_championship(cur_t + outcome.team_points, cur_tw + outcome.team_wins, + team_names, cur_t) + return drivers, constructors + + +def cmd_season(args): + from sim.season import ScheduleUnavailable + + fit_dir = args.fit_dir + if fit_dir is None: + if not os.path.exists(LATEST_POINTER): + print("No fit found. Run `gp fit --through ...` first, or pass --fit-dir.", file=sys.stderr) + sys.exit(1) + fit_dir = open(LATEST_POINTER).read().strip() + idata, data, reliability_data, _ = load_fit(fit_dir) + try: + drivers, constructors = forecast_season(idata, data, reliability_data, _load_driver_rounds(), + _load_circuits(), n_trials=args.n_trials, seed=args.seed) + except ScheduleUnavailable as exc: + print(str(exc), file=sys.stderr) + sys.exit(1) + + out_dir = args.out_dir or fit_dir + os.makedirs(out_dir, exist_ok=True) + for name, table in (("drivers_championship.csv", drivers), + ("constructors_championship.csv", constructors)): + path = os.path.join(out_dir, name) + table.to_csv(path, index=False) + print(table.to_string(index=False)) + print(f"Saved -> {path}\n") + + # --------------------------------------------------------------------------- # gp backtest # --------------------------------------------------------------------------- @@ -473,6 +626,14 @@ def cmd_backtest(args): if len(seasons) == 2: seasons = list(range(seasons[0], seasons[1] + 1)) + if args.season_level: + from backtest import run_season_backtest + + run_season_backtest(seasons=seasons, cutoffs=[int(c) for c in args.cutoffs.split(",")], + n_trials=args.n_trials, method=args.method, + advi_steps=args.advi_steps, out_dir=args.out_dir) + return + run_backtest( seasons=seasons, n_trials=args.n_trials, method=args.method, rounds=args.rounds, advi_steps=args.advi_steps, out_dir=args.out_dir, @@ -513,6 +674,13 @@ def build_parser() -> argparse.ArgumentParser: "pit-lane starts; a sprint's own grid is unaffected)") p_race.set_defaults(func=cmd_race) + p_season = sub.add_parser("season", help="Chain the rest of the fitted season: both championships") + p_season.add_argument("--fit-dir", default=None) + p_season.add_argument("--out-dir", default=None) + p_season.add_argument("--n-trials", type=int, default=10_000) + p_season.add_argument("--seed", type=int, default=0) + p_season.set_defaults(func=cmd_season) + p_bt = sub.add_parser("backtest", help="Rolling backtest (spec sec 7)") p_bt.add_argument("--seasons", default="2022-2025") p_bt.add_argument("--rounds", default="coarse", choices=["coarse", "all"]) @@ -525,6 +693,11 @@ def build_parser() -> argparse.ArgumentParser: "against pre-weekend (Phase 2 gate); writes conditioning_log.csv") p_bt.add_argument("--post-quali-chains", type=int, default=2, help="chains for each round's post-quali conditioning refit") + p_bt.add_argument("--season-level", action="store_true", + help="Season-level backtest instead (spec sec 7, Phase 3): fit at each " + "--cutoffs round, chain the rest of the season, score both championships") + p_bt.add_argument("--cutoffs", default="2,6,12", + help="with --season-level: comma-separated rounds to fit through") p_bt.set_defaults(func=cmd_backtest) return parser diff --git a/src/collect_v2.py b/src/collect_v2.py index ee3df4b..63fb165 100644 --- a/src/collect_v2.py +++ b/src/collect_v2.py @@ -133,6 +133,34 @@ def _weather_summary(session) -> tuple[bool | None, float | None]: return None, None +# `q_segment` on a quali row whose time came from the session laps rather +# than the official Q1/Q2/Q3 columns. Nothing reads q_segment as a segment +# name, so the flag needs no schema change. +LAPS_FALLBACK_SEGMENT = "laps" + + +def _fastest_session_laps(session) -> dict: + """abbreviation -> fastest non-deleted lap in seconds, off the session laps. + + The fallback for a quali whose results carry positions but no Q1/Q2/Q3 + times (2025 R6 Miami upstream). Deleted laps (track limits) are excluded, + so a driver whose only lap was struck off still gets no time. + """ + try: + laps = session.laps + except Exception: + log.warning(" session laps unavailable for %s %s", getattr(session, "event", "?"), + getattr(session, "name", "?"), exc_info=True) + return {} + if laps is None or laps.empty or "LapTime" not in laps.columns: + return {} + laps = laps.dropna(subset=["LapTime"]) + if "Deleted" in laps.columns: + laps = laps[laps["Deleted"].fillna(False) != True] # noqa: E712 -- object dtype + fastest = laps.groupby("Driver")["LapTime"].min() + return {str(d): t / np.timedelta64(1, "s") for d, t in fastest.items()} + + def _build_quali_rows(year: int, rnd: int, event_name: str, circuit_type: str) -> list[dict]: rows = [] try: @@ -147,22 +175,47 @@ def _build_quali_rows(year: int, rnd: int, event_name: str, circuit_type: str) - return rows is_wet, air_temp = _weather_summary(session) - pole_time = None - for q_col in ("Q3", "Q2", "Q1"): - col = results[q_col].dropna() - if not col.empty: - pole_time = col.min() / np.timedelta64(1, "s") - break - + best_by_row = [] for _, r in results.iterrows(): - best = None - q_segment = None + best, q_segment = None, None for q_col in ("Q3", "Q2", "Q1"): t = r.get(q_col) if pd.notna(t): best = t / np.timedelta64(1, "s") q_segment = q_col break + best_by_row.append((best, q_segment)) + + # A classified row with no Q time is the Miami 2025 gap: fall back to the + # driver's fastest session lap, and say so in the row and the log. + needs_laps = [i for i, (_, r) in enumerate(results.iterrows()) + if best_by_row[i][0] is None and pd.notna(r.get("Position"))] + if needs_laps: + lap_times = _fastest_session_laps(session) + filled = [] + for i in needs_laps: + abbr = str(results.iloc[i].get("Abbreviation", "")) + if abbr in lap_times: + best_by_row[i] = (lap_times[abbr], LAPS_FALLBACK_SEGMENT) + filled.append(abbr) + if filled: + log.warning(" %d Round %d quali: no Q1/Q2/Q3 time for %s -- best_lap_s taken from " + "the fastest session lap (q_segment=%r)", year, rnd, ", ".join(filled), + LAPS_FALLBACK_SEGMENT) + + # Pole is the official one wherever a Q column has times, exactly as + # before; lap-derived times set it only when the session has none at all. + pole_time = None + for q_col in ("Q3", "Q2", "Q1"): + col = results[q_col].dropna() + if not col.empty: + pole_time = col.min() / np.timedelta64(1, "s") + break + timed = [b for b, _ in best_by_row if b is not None] + if pole_time is None and timed: + pole_time = min(timed) + + for (_, r), (best, q_segment) in zip(results.iterrows(), best_by_row): rows.append({ "season": year, "round": rnd, "event_name": event_name, "circuit_type": circuit_type, "session_type": "Q", diff --git a/src/model/overtaking.py b/src/model/overtaking.py index db416db..151413f 100644 --- a/src/model/overtaking.py +++ b/src/model/overtaking.py @@ -179,15 +179,22 @@ def fit_circuit(rows: pd.DataFrame, event_name: str, hand_rating: float | None) def build_overtaking_data(driver_rounds: pd.DataFrame, circuits: pd.DataFrame | None = None, before_season: int | None = None, - before_round: int | None = None) -> OvertakingData: + before_round: int | None = None, + event_name: str | None = None) -> OvertakingData: """Fit every circuit with enough history in the window. `before_season`/`before_round` cut the window to rounds strictly *before* that round, which is what keeps a backtest honest: the round being forecast must not be in its own overtaking fit. + + `event_name` restricts the fit to that one circuit. A forecast consumes + only its own circuit's parameter, so fitting the rest of the calendar on + every call made a backtest pay rounds x circuits fits (slice-7). """ ratings = hand_ratings(circuits) df = _usable_race_rows(driver_rounds) + if event_name is not None and not df.empty: + df = df[df["event_name"].astype(str) == str(event_name)] if before_season is not None: cutoff = (before_season, before_round if before_round is not None else 0) keep = [(s, r) < cutoff for s, r in zip(df["season"], df["round"])] diff --git a/src/sim/race.py b/src/sim/race.py index cffc06c..a744abd 100644 --- a/src/sim/race.py +++ b/src/sim/race.py @@ -252,16 +252,21 @@ def simulate_weekend(inputs: RaceTrialInputs, rng: np.random.Generator, sprint_positions=sprint_positions, sprint_dnf=sprint_dnf) -def summarize_trials(positions: np.ndarray, dnf: np.ndarray, driver_ids: list, teams: list, - points_table: dict) -> pd.DataFrame: - """Turn (n_trials, n_drivers) position/dnf arrays into race_forecast.csv rows.""" - n_trials, n_drivers = positions.shape +def points_for_trials(positions: np.ndarray, dnf: np.ndarray, points_table: dict) -> np.ndarray: + """(n_trials, n_drivers) points scored, zero for a DNF.""" + n_drivers = positions.shape[1] points_by_pos = np.zeros(n_drivers + 1) for pos, pts in points_table.items(): if 1 <= pos <= n_drivers: points_by_pos[pos] = pts - trial_points = points_by_pos[positions] - trial_points = np.where(dnf, 0.0, trial_points) + return np.where(dnf, 0.0, points_by_pos[positions]) + + +def summarize_trials(positions: np.ndarray, dnf: np.ndarray, driver_ids: list, teams: list, + points_table: dict) -> pd.DataFrame: + """Turn (n_trials, n_drivers) position/dnf arrays into race_forecast.csv rows.""" + n_trials, n_drivers = positions.shape + trial_points = points_for_trials(positions, dnf, points_table) rows = [] for i, driver in enumerate(driver_ids): diff --git a/src/sim/season.py b/src/sim/season.py new file mode 100644 index 0000000..b4673b6 --- /dev/null +++ b/src/sim/season.py @@ -0,0 +1,206 @@ +"""Season chaining (spec sec 3.4): both championships from one set of trials. + +Each trial starts from one posterior draw of `car[t]` at the last fitted round +and, before each future round, steps every team's car by the pace model's +random walk (sec 3.1). The step lands on the running value, not the fitted +one, so the wander compounds over the rest of the year. Each round runs +through `sim.race.simulate_weekend` (a sprint shares its Grand Prix's pace +draw). Team points are the sum of that team's drivers' points in the same +trial, so the two championships are consistent by construction. + +Everything else that is a posterior parameter (driver skill, track +interaction, quali offset, noise scales, both DNF hazards) is drawn once per +trial and held for the season: only `car` has a random walk in sec 3.1. + +The remaining calendar comes from FastF1 through `remaining_calendar`'s +`loader` seam. If it cannot be read the command fails; it never invents one. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +import numpy as np +import pandas as pd + +from sim.race import RaceTrialInputs, points_for_trials, simulate_weekend + + +class ScheduleUnavailable(RuntimeError): + """The season's event schedule could not be read. `gp season` stops.""" + + +def _fastf1_schedule(season: int) -> pd.DataFrame: + import fastf1 + + return fastf1.get_event_schedule(season, include_testing=False) + + +def remaining_calendar(season: int, after_round: int, loader=None) -> pd.DataFrame: + """Rounds after `after_round` as `round, event_name, has_sprint`, from a + FastF1-shaped `loader(season)`. Sprints are read off `EventFormat` as in + `collect_v2`. An unreadable or empty schedule raises; an empty result + only ever means the season is over.""" + loader = loader or _fastf1_schedule + try: + schedule = loader(season) + except Exception as exc: + raise ScheduleUnavailable( + f"Could not read the {season} event schedule from FastF1 ({exc}). " + "gp season will not guess the remaining calendar.") from exc + needed = {"RoundNumber", "EventName", "EventFormat"} + if schedule is None or len(schedule) == 0 or not needed.issubset(schedule.columns): + raise ScheduleUnavailable( + f"The {season} event schedule came back empty or without {sorted(needed)}. " + "gp season will not guess the remaining calendar.") + round_numbers = schedule["RoundNumber"].astype(int) + rounds = schedule[round_numbers > max(after_round, 0)].sort_values("RoundNumber") + return pd.DataFrame({ + "round": rounds["RoundNumber"].astype(int).to_numpy(), + "event_name": rounds["EventName"].astype(str).to_numpy(), + "has_sprint": rounds["EventFormat"].astype(str).str.lower().str.contains("sprint").to_numpy(), + }) + + +def step_sigma_for_round(season: int, round_: int) -> float: + """Random-walk step into `round_`, as `model.pace._step_sigma_scale` within a season.""" + from model.pace import ( + RULE_CHANGE_EARLY_ROUNDS, RULE_CHANGE_EARLY_SCALE, RULE_CHANGE_SEASONS, SIGMA_DEV, + ) + + early = season in RULE_CHANGE_SEASONS and round_ - 1 < RULE_CHANGE_EARLY_ROUNDS + return SIGMA_DEV * (RULE_CHANGE_EARLY_SCALE if early else 1.0) + + +@dataclass +class SeasonRound: + round: int + has_sprint: bool + circuit_type_idx: int # column of `SeasonTrialInputs.driver_track` + overtaking_difficulty: float + step_sigma: float # sd of the car random-walk step into this round + + +@dataclass +class SeasonTrialInputs: + """One posterior draw per trial, held for the whole season except `car`.""" + driver_ids: list + teams: list # per driver, len n_drivers + team_names: list # columns of `car` + car: np.ndarray # (n_trials, n_teams) at the last fitted round + driver_skill: np.ndarray # (n_trials, n_drivers) + driver_track: np.ndarray # (n_trials, n_drivers, n_circuit_types) + quali_offset: np.ndarray # (n_trials, n_drivers) + race_noise_nu: np.ndarray # (n_trials,) + race_noise_sigma: np.ndarray # (n_trials,) + quali_noise_sigma: np.ndarray # (n_trials,) + mechanical_dnf_prob: np.ndarray # (n_trials, n_drivers) + incident_dnf_prob: np.ndarray # (n_trials, n_drivers) + + +@dataclass +class SeasonOutcome: + """Simulated rounds only; `cli` adds the points already scored.""" + driver_points: np.ndarray # (n_trials, n_drivers) + driver_wins: np.ndarray # (n_trials, n_drivers), Grand Prix wins + team_points: np.ndarray # (n_trials, n_teams) + team_wins: np.ndarray # (n_trials, n_teams) + car_path: np.ndarray # (n_trials, n_rounds, n_teams), car at each simulated round + + +def simulate_season(inputs: SeasonTrialInputs, rounds: list, gp_points: dict, sprint_points: dict, + rng: np.random.Generator) -> SeasonOutcome: + """Chain `rounds` in order within every trial (sec 3.4).""" + from model.reliability import combined_dnf_prob + + n_trials, n_drivers = inputs.driver_skill.shape + team_col = np.array([inputs.team_names.index(t) for t in inputs.teams]) + dnf_prob = combined_dnf_prob(inputs.mechanical_dnf_prob, inputs.incident_dnf_prob) + + car = np.array(inputs.car, dtype=float, copy=True) + car_path = np.zeros((n_trials, len(rounds), car.shape[1])) + points = np.zeros((n_trials, n_drivers)) + wins = np.zeros((n_trials, n_drivers)) + for k, rnd in enumerate(rounds): + # The step lands on the running value: this is the carry-forward. + car = car + rng.normal(0.0, rnd.step_sigma, size=car.shape) + car_path[:, k] = car + car_d = car[:, team_col] + race_inputs = RaceTrialInputs( + driver_ids=inputs.driver_ids, + race_pace=car_d + inputs.driver_skill + inputs.driver_track[:, :, rnd.circuit_type_idx], + race_noise_nu=inputs.race_noise_nu, race_noise_sigma=inputs.race_noise_sigma, + quali_pace=car_d + inputs.driver_skill + inputs.quali_offset, + quali_noise_sigma=inputs.quali_noise_sigma, + dnf_prob=dnf_prob, + mechanical_dnf_prob=inputs.mechanical_dnf_prob, + incident_dnf_prob=inputs.incident_dnf_prob, + overtaking_difficulty=rnd.overtaking_difficulty, + ) + outcome = simulate_weekend(race_inputs, rng, has_sprint=rnd.has_sprint) + points += points_for_trials(outcome.positions, outcome.dnf, gp_points) + wins += (outcome.positions == 1) & ~outcome.dnf + if outcome.has_sprint: + points += points_for_trials(outcome.sprint_positions, outcome.sprint_dnf, sprint_points) + + membership = np.zeros((n_drivers, len(inputs.team_names))) + membership[np.arange(n_drivers), team_col] = 1.0 + return SeasonOutcome(driver_points=points, driver_wins=wins, + team_points=points @ membership, team_wins=wins @ membership, + car_path=car_path) + + +def final_positions(points: np.ndarray, wins: np.ndarray) -> np.ndarray: + """1-indexed championship position per trial; points ties go to more GP wins.""" + # Points come in steps of >= 0.5, wins are < 50 a season, so this key + # orders by points first and wins second without a per-trial lexsort. + key = np.asarray(points, dtype=float) * 100.0 + np.asarray(wins, dtype=float) + order = np.argsort(-key, axis=1, kind="stable") + return np.argsort(order, axis=1, kind="stable") + 1 + + +def _position_columns(pos: np.ndarray) -> dict: + return {"p10_final_position": float(np.percentile(pos, 10)), + "p50_final_position": float(np.percentile(pos, 50)), + "p90_final_position": float(np.percentile(pos, 90))} + + +def _points_interval(points: np.ndarray) -> dict: + """The 80% interval on final points, which the season backtest scores (sec 7).""" + return {"p10_final_points": float(np.percentile(points, 10)), + "p90_final_points": float(np.percentile(points, 90))} + + +def drivers_championship(total_points: np.ndarray, total_wins: np.ndarray, driver_ids: list, + teams: list, current_points) -> pd.DataFrame: + """sec 6 `drivers_championship.csv`, plus `team`, `current_points` and + the 80% interval on final points.""" + pos = final_positions(total_points, total_wins) + rows = [] + for i, driver in enumerate(driver_ids): + rows.append({"driver": driver, "team": teams[i], "current_points": float(current_points[i]), + "p_title": float(np.mean(pos[:, i] == 1)), + "exp_final_points": float(np.mean(total_points[:, i])), + **_position_columns(pos[:, i]), + "p_top3": float(np.mean(pos[:, i] <= 3)), + **_points_interval(total_points[:, i])}) + return pd.DataFrame(rows).sort_values("exp_final_points", ascending=False).reset_index(drop=True) + + +def constructors_championship(total_points: np.ndarray, total_wins: np.ndarray, team_names: list, + current_points) -> pd.DataFrame: + """sec 6 `constructors_championship.csv`, plus `current_points` and the + 80% interval on final points. + `exp_share` is the per-trial share of all constructors' points, averaged.""" + pos = final_positions(total_points, total_wins) + totals = total_points.sum(axis=1, keepdims=True) + share = np.divide(total_points, totals, out=np.zeros_like(total_points), where=totals > 0) + rows = [] + for j, team in enumerate(team_names): + rows.append({"team": team, "current_points": float(current_points[j]), + "p_title": float(np.mean(pos[:, j] == 1)), + "exp_final_points": float(np.mean(total_points[:, j])), + "exp_share": float(np.mean(share[:, j])), + **_position_columns(pos[:, j]), + **_points_interval(total_points[:, j])}) + return pd.DataFrame(rows).sort_values("exp_final_points", ascending=False).reset_index(drop=True) diff --git a/tests/test_collect_quali_fallback.py b/tests/test_collect_quali_fallback.py new file mode 100644 index 0000000..cdf4732 --- /dev/null +++ b/tests/test_collect_quali_fallback.py @@ -0,0 +1,131 @@ +"""`collect_v2` quali rows when the results carry positions but no Q1/Q2/Q3 +times -- the 2025 R6 Miami gap, repaired by hand for the Phase 2 backtest +(docs/phase2-status.md, "Data"). Autopilot slice-7. + +FastF1 is mocked throughout: no network, no cache. +""" + +import logging + +import numpy as np +import pandas as pd +import pytest + +import collect_v2 +from collect_v2 import LAPS_FALLBACK_SEGMENT, _build_quali_rows + + +def _td(seconds): + return pd.Timedelta(seconds=seconds) if seconds is not None else pd.NaT + + +class _StubQuali: + def __init__(self, results, laps): + self.results = results + self.laps = laps + self.weather_data = pd.DataFrame({"AirTemp": [25.0], "Rainfall": [False]}) + self.event, self.name = "Miami Grand Prix", "Qualifying" + + def load(self, **kwargs): + pass + + +def _results(q_times): + """q_times: abbreviation -> (Q1, Q2, Q3) in seconds or None, in position order.""" + rows = [] + for pos, (abbr, (q1, q2, q3)) in enumerate(q_times.items(), start=1): + rows.append({"Abbreviation": abbr, "FullName": abbr, "TeamName": "T", + "Position": float(pos), "Q1": _td(q1), "Q2": _td(q2), "Q3": _td(q3)}) + return pd.DataFrame(rows) + + +def _laps(rows): + """rows: (driver, lap_time_s, deleted).""" + return pd.DataFrame([{"Driver": d, "LapTime": _td(t), "Deleted": deleted} + for d, t, deleted in rows]) + + +MIAMI_LAPS = _laps([ + ("NOR", 86.400, False), ("NOR", 86.269, False), + ("PIA", 86.271, False), ("PIA", 85.900, True), # struck off: must not count + ("ANT", 86.271, False), ("ANT", None, False), +]) + + +def _rows(monkeypatch, results, laps): + monkeypatch.setattr(collect_v2.fastf1, "get_session", lambda *a: _StubQuali(results, laps)) + return {r["abbreviation"]: r for r in _build_quali_rows(2025, 6, "Miami Grand Prix", "mixed")} + + +class TestNoQTimesFallsBackToSessionLaps: + RESULTS = _results({"NOR": (None, None, None), "PIA": (None, None, None), + "ANT": (None, None, None)}) + + def test_best_lap_is_the_fastest_non_deleted_session_lap(self, monkeypatch): + rows = _rows(monkeypatch, self.RESULTS, MIAMI_LAPS) + assert rows["NOR"]["best_lap_s"] == pytest.approx(86.269) + assert rows["PIA"]["best_lap_s"] == pytest.approx(86.271) + assert rows["ANT"]["best_lap_s"] == pytest.approx(86.271) + + def test_fallback_rows_are_flagged_and_classified(self, monkeypatch): + rows = _rows(monkeypatch, self.RESULTS, MIAMI_LAPS) + assert {r["q_segment"] for r in rows.values()} == {LAPS_FALLBACK_SEGMENT} + assert all(r["classified"] for r in rows.values()) + + def test_gaps_are_to_the_fastest_lap_derived_time(self, monkeypatch): + rows = _rows(monkeypatch, self.RESULTS, MIAMI_LAPS) + assert rows["NOR"]["gap_to_best_s"] == pytest.approx(0.0) + assert rows["PIA"]["gap_to_best_s"] == pytest.approx(0.002) + + def test_positions_are_kept_from_the_results(self, monkeypatch): + rows = _rows(monkeypatch, self.RESULTS, MIAMI_LAPS) + assert [rows[d]["finish_position"] for d in ("NOR", "PIA", "ANT")] == [1.0, 2.0, 3.0] + + def test_the_fallback_is_logged_with_the_drivers(self, monkeypatch, caplog): + with caplog.at_level(logging.WARNING, logger=collect_v2.log.name): + _rows(monkeypatch, self.RESULTS, MIAMI_LAPS) + assert any("fastest session lap" in m and "NOR" in m for m in caplog.messages) + + def test_a_driver_with_only_a_deleted_lap_stays_untimed(self, monkeypatch): + laps = _laps([("NOR", 86.3, False), ("PIA", 85.9, True), ("ANT", 86.4, False)]) + rows = _rows(monkeypatch, self.RESULTS, laps) + assert rows["PIA"]["best_lap_s"] is None + assert rows["PIA"]["q_segment"] is None and not rows["PIA"]["classified"] + + +class TestTimedSessionsAreUnchanged: + def test_official_times_win_and_laps_are_not_read(self, monkeypatch): + results = _results({"VER": (87.0, 86.5, 86.1), "NOR": (87.1, 86.6, 86.2)}) + + class NoLaps(_StubQuali): + @property + def laps(self): + raise AssertionError("laps read for a session with Q times") + + @laps.setter + def laps(self, value): + pass + + monkeypatch.setattr(collect_v2.fastf1, "get_session", lambda *a: NoLaps(results, None)) + rows = {r["abbreviation"]: r for r in _build_quali_rows(2024, 1, "X", "mixed")} + assert rows["VER"]["best_lap_s"] == pytest.approx(86.1) + assert rows["NOR"]["q_segment"] == "Q3" + assert rows["NOR"]["gap_to_best_s"] == pytest.approx(0.1) + + def test_partial_gap_fills_only_the_missing_row_and_keeps_official_pole(self, monkeypatch): + results = _results({"VER": (87.0, 86.5, 86.1), "NOR": (None, None, None)}) + laps = _laps([("VER", 85.0, False), ("NOR", 86.3, False)]) + rows = _rows(monkeypatch, results, laps) + assert rows["VER"]["best_lap_s"] == pytest.approx(86.1) + assert rows["VER"]["q_segment"] == "Q3" + assert rows["NOR"]["q_segment"] == LAPS_FALLBACK_SEGMENT + assert rows["NOR"]["gap_to_best_s"] == pytest.approx(0.2) + + def test_an_unclassified_row_without_times_is_left_alone(self, monkeypatch): + results = _results({"VER": (87.0, 86.5, 86.1)}) + results = pd.concat([results, pd.DataFrame([{ + "Abbreviation": "SAR", "FullName": "SAR", "TeamName": "T", "Position": np.nan, + "Q1": pd.NaT, "Q2": pd.NaT, "Q3": pd.NaT}])], ignore_index=True) + laps = _laps([("SAR", 88.0, False)]) + rows = _rows(monkeypatch, results, laps) + assert rows["SAR"]["best_lap_s"] is None and rows["SAR"]["q_segment"] is None diff --git a/tests/test_overtaking_wiring.py b/tests/test_overtaking_wiring.py new file mode 100644 index 0000000..fd97d0c --- /dev/null +++ b/tests/test_overtaking_wiring.py @@ -0,0 +1,162 @@ +"""The fitted overtaking parameter reaches both production forecasts, and each +forecast fits only its own circuit (autopilot slice-7, slice-4 carry-forwards). + +PR #10's review found that `build_overtaking_data` could be deleted from +either call site (`cli.forecast_weekend`, `backtest.forecast_race_challenger`) +with the whole suite green: the tests handed the fit in at the `_circuit_info` +seam, so nothing checked that a forecast builds one for itself. The fixture +below makes the fit depart from the hand rating, so a forecast that silently +falls back to `data/circuits.csv` is caught. +""" + +import numpy as np +import pandas as pd +import pytest + +from cli import forecast_weekend +from model.overtaking import build_overtaking_data +from test_cli import DRIVERS, _fake_fit, _sprint_weekend_rounds + +HAND_RATING = 4.5 +HISTORY_EVENTS = ["Bahrain Grand Prix", "Monaco Grand Prix", "Italian Grand Prix", + "Japanese Grand Prix", "Brazilian Grand Prix"] +CIRCUITS = pd.DataFrame({ + "event_name": HISTORY_EVENTS, "circuit_type": ["mixed"] * len(HISTORY_EVENTS), + "overtaking_difficulty": [HAND_RATING] * len(HISTORY_EVENTS), +}) + + +def _locked_history(events=HISTORY_EVENTS, rounds_per_event=6, n_drivers=8): + """2023 races where nobody passes: the fastest car starts last and + finishes last. Only a very low difficulty explains that, so every + circuit's fit lands well below the 4.5 hand rating.""" + rows = [] + rnd = 0 + for event in events: + for _ in range(rounds_per_event): + rnd += 1 + for i in range(n_drivers): + grid = float(n_drivers - i) + rows.append(dict( + season=2023, round=rnd, event_name=event, circuit_type="mixed", + session_type="R", abbreviation=f"H{i}", team="Alpha" if i % 2 else "Beta", + grid_position=grid, finish_position=grid, classified=True, + status="Finished", dnf_cause=None, + gap_to_winner_median_clean_air_s=1.0 * i, is_wet=False, + )) + return pd.DataFrame(rows) + + +def _driver_rounds(): + return pd.concat([_locked_history(), _sprint_weekend_rounds()], ignore_index=True) + + +def _fitted_bahrain(driver_rounds): + fit = build_overtaking_data(driver_rounds, CIRCUITS, before_season=2024, + before_round=2).fit_for("Bahrain Grand Prix") + assert fit is not None + # The fixture is only useful if the fit really departs from the rating. + assert abs(fit.shrunk_difficulty - HAND_RATING) > 1.0 + return fit.shrunk_difficulty + + +def _count_fits(monkeypatch): + import model.overtaking + + fitted = [] + real = model.overtaking.fit_circuit + + def counting(rows, event_name, hand_rating): + fitted.append(event_name) + return real(rows, event_name, hand_rating) + + monkeypatch.setattr(model.overtaking, "fit_circuit", counting) + return fitted + + +def _run_forecast_weekend(monkeypatch, driver_rounds): + import sim.race + from model.reliability import build_reliability_data + + captured = {} + real = sim.race.simulate_weekend + + def spy(inputs, *a, **kw): + captured["inputs"] = inputs + return real(inputs, *a, **kw) + + monkeypatch.setattr(sim.race, "simulate_weekend", spy) + idata, data = _fake_fit(driver_rounds) + forecast_weekend(idata, data, build_reliability_data(driver_rounds), driver_rounds, + CIRCUITS, season=2024, round_=2, n_trials=200, seed=0) + return captured["inputs"] + + +def _run_challenger(monkeypatch, driver_rounds): + import model.challenger + import sim.race + from backtest import forecast_race_challenger + from model.reliability import build_reliability_data + + captured = {} + real = sim.race.simulate_positions + + def spy(inputs, *a, **kw): + captured["inputs"] = inputs + return real(inputs, *a, **kw) + + def fake_pace(model, team, driver, circuit_type, trp, drp, n, rng): + return rng.normal(DRIVERS.index(driver) * 0.3, 0.1, n) + + monkeypatch.setattr(sim.race, "simulate_positions", spy) + monkeypatch.setattr(model.challenger, "sample_pace", fake_pace) + forecast_race_challenger(None, build_reliability_data(driver_rounds), driver_rounds, + CIRCUITS, season=2024, round_=2, n_trials=200, seed=0) + return captured["inputs"] + + +RUNNERS = {"forecast_weekend": _run_forecast_weekend, + "forecast_race_challenger": _run_challenger} + + +@pytest.mark.parametrize("call_site", sorted(RUNNERS)) +class TestBothCallSitesFitTheirOwnCircuit: + def test_the_resolver_gets_the_fitted_value_not_the_hand_rating(self, monkeypatch, call_site): + driver_rounds = _driver_rounds() + inputs = RUNNERS[call_site](monkeypatch, driver_rounds) + assert inputs.overtaking_difficulty == pytest.approx(_fitted_bahrain(driver_rounds)) + assert inputs.overtaking_difficulty != pytest.approx(HAND_RATING) + + def test_one_circuit_is_fitted_however_many_the_window_holds(self, monkeypatch, call_site): + """Five circuits in the window, one fit per forecast. Before slice-7 + this counted every circuit with history (five here).""" + driver_rounds = _driver_rounds() + fitted = _count_fits(monkeypatch) + RUNNERS[call_site](monkeypatch, driver_rounds) + assert fitted == ["Bahrain Grand Prix"] + + def test_the_fit_count_does_not_grow_with_the_window(self, monkeypatch, call_site): + small = pd.concat([_locked_history(HISTORY_EVENTS[:1]), _sprint_weekend_rounds()], + ignore_index=True) + counts = [] + for driver_rounds in (small, _driver_rounds()): + fitted = _count_fits(monkeypatch) + RUNNERS[call_site](monkeypatch, driver_rounds) + counts.append(len(fitted)) + assert counts == [1, 1] + + +class TestSingleCircuitFit: + def test_restricting_to_one_event_matches_that_event_in_the_full_fit(self): + driver_rounds = _locked_history() + full = build_overtaking_data(driver_rounds, CIRCUITS) + one = build_overtaking_data(driver_rounds, CIRCUITS, event_name="Monaco Grand Prix") + assert set(one.by_circuit) == {"Monaco Grand Prix"} + assert one.fit_for("Monaco Grand Prix") == full.fit_for("Monaco Grand Prix") + + def test_an_event_with_no_history_gets_no_fit(self): + one = build_overtaking_data(_locked_history(), CIRCUITS, event_name="Las Vegas Grand Prix") + assert one.by_circuit == {} and one.skipped == {} + + def test_no_event_still_fits_the_whole_window(self): + assert len(build_overtaking_data(_locked_history(), CIRCUITS).by_circuit) == len(HISTORY_EVENTS) diff --git a/tests/test_postquali_calibration.py b/tests/test_postquali_calibration.py new file mode 100644 index 0000000..cff4d3a --- /dev/null +++ b/tests/test_postquali_calibration.py @@ -0,0 +1,146 @@ +"""Post-quali calibration (autopilot slice-10; docs/phase2-status.md caveat 1): +a podium calibration file per arm, a binned table and an ECE for both arms in +the summary, and the rounds where post-quali did much worse than pre-weekend.""" +import json +import os +import shutil + +import numpy as np +import pandas as pd +import pytest + +import backtest +import model.pace +from backtest import ( + CALIBRATION_COLUMNS, calibration_summary, post_quali_summary, postquali_worse_rounds, + score_round, +) + +FIXTURE = os.path.join(os.path.dirname(__file__), "..", "data", "driver_rounds_fixture.csv") + + +def _calib(pairs): + return pd.DataFrame([(p, h, 2024, 1) for p, h in pairs], columns=CALIBRATION_COLUMNS) + + +# --- calibration_summary ---------------------------------------------------- + +def test_binned_table_and_ece_match_a_hand_computation(): + # bin [0.0, 0.1): preds 0.05, 0.05, hits 0/2 -> |0.05 - 0| = 0.05, weight 2/5 + # bin [0.5, 0.6): preds 0.55 x3, hits 3/3 -> 0.45, weight 3/5 + out = calibration_summary(_calib([(0.05, False), (0.05, False), + (0.55, True), (0.55, True), (0.55, True)])) + assert out["n"] == 5 + assert [(b["bin_lo"], b["n"]) for b in out["bins"]] == [(0.0, 2), (0.5, 3)] + assert out["bins"][1]["mean_pred"] == pytest.approx(0.55) + assert out["bins"][1]["observed_rate"] == 1.0 + assert out["ece"] == pytest.approx(2 / 5 * 0.05 + 3 / 5 * 0.45) + + +def test_a_perfectly_calibrated_arm_has_zero_ece(): + pairs = [(0.25, True)] + [(0.25, False)] * 3 + [(0.75, True)] * 3 + [(0.75, False)] + assert calibration_summary(_calib(pairs))["ece"] == pytest.approx(0.0) + + +def test_p_of_one_falls_in_the_top_bin_and_empty_bins_are_omitted(): + out = calibration_summary(_calib([(1.0, True), (0.0, False)])) + assert [(b["bin_lo"], b["bin_hi"]) for b in out["bins"]] == [(0.0, 0.1), (0.9, 1.0)] + + +def test_an_empty_arm_reports_no_ece_rather_than_zero(): + assert calibration_summary(_calib([])) == {"n": 0, "ece": None, "bins": []} + + +# --- the confident-miss listing -------------------------------------------- + +def _metrics(pre, post): + n = len(pre) + return pd.DataFrame({ + "season": [2022] * n, "round": list(range(1, n + 1)), + "log_loss_winner_model": pre, "log_loss_winner_postquali": post, + "grid_pole_driver": [f"P{i}" for i in range(n)], + "actual_winner": [f"W{i}" for i in range(n)], + "brier_podium_model": 0.1, "brier_points_model": 0.2, "spearman_model": 0.5, + "brier_podium_postquali": 0.1, "brier_points_postquali": 0.2, "spearman_postquali": 0.5, + }) + + +def test_only_rounds_worse_by_more_than_the_threshold_are_listed_worst_first(): + # deltas: +2.4, +1.0 (at threshold: not listed), -0.5, +1.5, NaN (no post score) + m = _metrics([1.5, 1.0, 1.0, 1.0, 1.0], [3.9, 2.0, 0.5, 2.5, float("nan")]) + out = postquali_worse_rounds(m, threshold=1.0) + assert [(r["round"], r["grid_pole_driver"], r["actual_winner"]) for r in out] == [ + (1, "P0", "W0"), (4, "P3", "W3")] + assert out[0]["delta"] == pytest.approx(2.4) + assert out[0]["log_loss_pre"] == 1.5 and out[0]["log_loss_postquali"] == 3.9 + + +def test_post_quali_summary_carries_the_listing_and_its_threshold(): + out = post_quali_summary(_metrics([1.0, 1.0], [3.0, 0.5]), + pd.DataFrame({"fell_back": [False, False], "regression": [False, False]})) + assert out["postquali_worse_threshold_nats"] == backtest.POSTQUALI_WORSE_THRESHOLD_NATS + assert [r["round"] for r in out["postquali_worse_rounds"]] == [1] + json.dumps(out) # the listing must survive into backtest_summary.json + + +def test_score_round_names_the_pole_sitter_and_the_winner(): + dr = pd.read_csv(FIXTURE) + race = dr[(dr["season"] == 2024) & (dr["round"] == 1) & (dr["session_type"] == "R")] + forecast = pd.DataFrame({"driver": race["abbreviation"], "p_win": 0.05, "p_podium": 0.15, + "p_points": 0.5, "exp_points": np.arange(len(race), 0, -1)}) + out = score_round(forecast, dr, 2024, 1) + assert out["grid_pole_driver"] == race.loc[race["grid_position"] == 1, "abbreviation"].iloc[0] + assert out["actual_winner"] == race.loc[race["finish_position"] == 1, "abbreviation"].iloc[0] + + +# --- the arm's calibration rows ---------------------------------------------- + +def test_the_post_quali_arm_adds_one_calibration_row_per_scored_driver(monkeypatch, tmp_path): + import cli + import model.conditioning as conditioning + from model.conditioning import ConditioningRecord + + dr = pd.read_csv(FIXTURE) + race = dr[(dr["season"] == 2024) & (dr["round"] == 2) & (dr["session_type"] == "R")] + forecast = pd.DataFrame({"driver": race["abbreviation"], "p_win": 0.05, "p_podium": 0.2, + "p_points": 0.5, "exp_points": np.arange(len(race), 0, -1)}) + record = ConditioningRecord(season=2024, round=2, path="warm_start", fell_back=False, + early_season=True, warm_start_s=1.0, fallback_s=0.0, total_s=1.0, + warm_target_s=120.0, applied_target_s=300.0, regression=False) + monkeypatch.setattr(conditioning, "condition_on_quali", lambda *a, **kw: ("I", "D", record)) + monkeypatch.setattr(cli, "forecast_race", lambda *a, **kw: forecast) + rows = [] + out = backtest._post_quali_arm("PRIOR", dr, None, None, "REL", 2024, 2, 100, 1, [], + str(tmp_path), calibration_rows=rows) + assert out["postquali_status"] == "ok" + assert len(rows) == len(race) + assert sum(hit for _, hit, _, _ in rows) == 3 + assert {(s, r) for _, _, s, r in rows} == {(2024, 2)} + + +# --- end to end on the fixture --------------------------------------------- + +def test_end_to_end_writes_both_calibration_files_and_both_summaries(monkeypatch, tmp_path): + data_dir = tmp_path / "data" + data_dir.mkdir() + shutil.copy(FIXTURE, data_dir / "driver_rounds.csv") + monkeypatch.setattr(backtest, "DATA_DIR", str(data_dir)) + # conditioning is NUTS; swap in a tiny ADVI fit so this runs in seconds + real_advi = model.pace.fit_advi + monkeypatch.setattr(model.pace, "fit_nuts", + lambda data, **kw: real_advi(data, n=200, draws=20)) + out_dir = tmp_path / "out" + backtest.run_backtest([2024], n_trials=200, method="advi", rounds="all", advi_steps=200, + out_dir=str(out_dir), post_quali=True, post_quali_chains=1) + + pre = pd.read_csv(out_dir / "calibration_podium.csv") + post = pd.read_csv(out_dir / "calibration_podium_postquali.csv") + assert list(pre.columns) == list(post.columns) == CALIBRATION_COLUMNS + assert len(post) == len(pre) > 0 + summary = json.loads((out_dir / "backtest_summary.json").read_text()) + calib = summary["calibration_podium"] + assert set(calib) == {"pre", "pre_paired", "postquali"} + for arm in calib.values(): + assert arm["n"] == len(pre) and 0.0 <= arm["ece"] <= 1.0 + assert sum(b["n"] for b in arm["bins"]) == arm["n"] + assert isinstance(summary["postquali_worse_rounds"], list) diff --git a/tests/test_season.py b/tests/test_season.py new file mode 100644 index 0000000..2fb2318 --- /dev/null +++ b/tests/test_season.py @@ -0,0 +1,212 @@ +"""Season chaining and `gp season` (spec sec 3.4, sec 6; autopilot slice-8).""" + +import numpy as np +import pandas as pd +import pytest + +import sim.season +from sim.season import ( + ScheduleUnavailable, SeasonRound, SeasonTrialInputs, final_positions, remaining_calendar, + simulate_season, +) + +GP = {1: 25, 2: 18, 3: 15, 4: 12, 5: 10, 6: 8, 7: 6, 8: 4, 9: 2, 10: 1} +SPRINT = {1: 8, 2: 7, 3: 6, 4: 5, 5: 4, 6: 3, 7: 2, 8: 1} + + +def _inputs(n_trials=2000, n_teams=5, dnf=0.0, seed=0): + """Synthetic posterior: two drivers per team, every trial's car identical + at the start, so any spread in `car_path` is the random walk's own.""" + rng = np.random.default_rng(seed) + n_drivers = 2 * n_teams + team_names = [f"T{j}" for j in range(n_teams)] + shape = (n_trials, n_drivers) + return SeasonTrialInputs( + driver_ids=[f"D{i}" for i in range(n_drivers)], + teams=[team_names[i // 2] for i in range(n_drivers)], + team_names=team_names, + car=np.tile(np.linspace(0.0, 1.0, n_teams), (n_trials, 1)), + driver_skill=rng.normal(0.0, 0.1, size=shape), + driver_track=np.zeros((*shape, 4)), + quali_offset=np.zeros(shape), + race_noise_nu=np.full(n_trials, 5.0), race_noise_sigma=np.full(n_trials, 0.3), + quali_noise_sigma=np.full(n_trials, 0.2), + mechanical_dnf_prob=np.full(shape, dnf), incident_dnf_prob=np.full(shape, dnf), + ) + + +def _rounds(n, sprint_rounds=(), step=0.08): + return [SeasonRound(round=k, has_sprint=k in sprint_rounds, circuit_type_idx=0, + overtaking_difficulty=3.0, step_sigma=step) for k in range(1, n + 1)] + + +class TestRandomWalkCarriesForward: + def test_late_round_car_draw_is_wider_than_early(self): + out = simulate_season(_inputs(), _rounds(12), GP, SPRINT, np.random.default_rng(1)) + early = out.car_path[:, 0].std(axis=0) + late = out.car_path[:, -1].std(axis=0) + # Compounding drift: sd after k steps is step * sqrt(k). Re-drawing each + # round's step from the fitted value would leave every round at one step. + assert np.all(late > 2.5 * early) + np.testing.assert_allclose(late, 0.08 * np.sqrt(12), rtol=0.1) + + def test_rounds_draw_pace_from_the_wandered_car(self): + # A huge step makes the order random by the last round even though T0's + # fitted car starts a full second clear: proof the resolver reads the + # running car, not `inputs.car`. + inputs = _inputs(n_trials=500) + calm = simulate_season(inputs, _rounds(3, step=0.0), GP, SPRINT, np.random.default_rng(2)) + wild = simulate_season(inputs, _rounds(3, step=5.0), GP, SPRINT, np.random.default_rng(2)) + assert calm.team_points[:, 0].mean() > wild.team_points[:, 0].mean() + 20 + + +class TestPoints: + def test_sprint_rounds_score_their_own_table(self): + out = simulate_season(_inputs(n_trials=50), _rounds(3, sprint_rounds=(2,)), GP, SPRINT, + np.random.default_rng(3)) + # No DNFs: every trial hands out three GP tables and one sprint table. + np.testing.assert_allclose(out.driver_points.sum(axis=1), 3 * 101 + 36) + + def test_sprint_goes_through_simulate_weekend(self, monkeypatch): + calls = [] + real = sim.season.simulate_weekend + + def spy(inputs, rng, has_sprint=False, **kw): + calls.append(has_sprint) + return real(inputs, rng, has_sprint=has_sprint, **kw) + + monkeypatch.setattr(sim.season, "simulate_weekend", spy) + simulate_season(_inputs(n_trials=10), _rounds(3, sprint_rounds=(3,)), GP, SPRINT, + np.random.default_rng(0)) + assert calls == [False, False, True] + + def test_constructor_points_are_the_sum_of_their_drivers_per_trial(self): + inputs = _inputs(n_trials=300, dnf=0.1) + out = simulate_season(inputs, _rounds(4, sprint_rounds=(2,)), GP, SPRINT, + np.random.default_rng(4)) + for j, team in enumerate(inputs.team_names): + cols = [i for i, t in enumerate(inputs.teams) if t == team] + np.testing.assert_array_equal(out.team_points[:, j], out.driver_points[:, cols].sum(axis=1)) + np.testing.assert_array_equal(out.team_wins[:, j], out.driver_wins[:, cols].sum(axis=1)) + + +def test_final_positions_break_points_ties_on_wins(): + points = np.array([[100.0, 100.0, 50.0]]) + wins = np.array([[1, 3, 0]]) + np.testing.assert_array_equal(final_positions(points, wins), [[2, 1, 3]]) + + +def _schedule(rounds=6, sprint=(3,)): + return pd.DataFrame({ + "RoundNumber": list(range(0, rounds + 1)), + "EventName": ["Pre-Season Testing"] + [f"Event {r}" for r in range(1, rounds + 1)], + "EventFormat": ["testing"] + ["sprint_qualifying" if r in sprint else "conventional" + for r in range(1, rounds + 1)], + }) + + +class TestRemainingCalendar: + def test_rounds_after_cutoff_with_sprint_flags(self): + cal = remaining_calendar(2026, 2, loader=lambda s: _schedule()) + assert cal["round"].tolist() == [3, 4, 5, 6] + assert cal["has_sprint"].tolist() == [True, False, False, False] + + def test_unreadable_schedule_fails_rather_than_inventing_one(self): + def broken(season): + raise ConnectionError("no network") + + with pytest.raises(ScheduleUnavailable, match="will not guess"): + remaining_calendar(2026, 2, loader=broken) + + +# --------------------------------------------------------------------------- +# gp season end to end, on test_cli's fake posterior +# --------------------------------------------------------------------------- + +def _fit_inputs(): + from model.reliability import build_reliability_data + from test_cli import _fake_fit, _sprint_weekend_rounds + + driver_rounds = _sprint_weekend_rounds() + driver_rounds["points"] = 0.0 + is_race = driver_rounds["session_type"] == "R" + driver_rounds.loc[is_race, "points"] = driver_rounds.loc[is_race, "finish_position"].map(GP).fillna(0.0) + idata, data = _fake_fit(driver_rounds) + reliability = build_reliability_data(driver_rounds, through_season=2024, through_round=2) + circuits = pd.DataFrame({"event_name": ["Bahrain Grand Prix"], "circuit_type": ["mixed"], + "overtaking_difficulty": [4.5]}) + return idata, data, reliability, driver_rounds, circuits + + +def test_forecast_season_tables(monkeypatch): + from cli import forecast_season + + idata, data, reliability, driver_rounds, circuits = _fit_inputs() + drivers, constructors = forecast_season(idata, data, reliability, driver_rounds, circuits, + n_trials=400, seed=0, + schedule_loader=lambda s: _schedule(rounds=5)) + for col in ["driver", "p_title", "exp_final_points", "p10_final_position", + "p50_final_position", "p90_final_position", "p_top3"]: + assert col in drivers.columns + for col in ["team", "p_title", "exp_final_points", "exp_share", "p10_final_position", + "p50_final_position", "p90_final_position"]: + assert col in constructors.columns + + # current points: d1 won both rounds already run + current = drivers.set_index("driver")["current_points"] + assert current["d1"] == 50.0 and current["d2"] == 36.0 + assert (drivers["exp_final_points"] >= drivers["current_points"]).all() + assert drivers["p_title"].sum() == pytest.approx(1.0) + assert constructors["exp_share"].sum() == pytest.approx(1.0) + # same trials: constructor expectations are their drivers' expectations summed + by_team = drivers.groupby("team")["exp_final_points"].sum() + for team, exp in constructors.set_index("team")["exp_final_points"].items(): + assert exp == pytest.approx(by_team[team]) + + +def test_season_finished_means_standings_are_final(): + from cli import forecast_season + + idata, data, reliability, driver_rounds, circuits = _fit_inputs() + drivers, _ = forecast_season(idata, data, reliability, driver_rounds, circuits, n_trials=50, + schedule_loader=lambda s: _schedule(rounds=2)) + leader = drivers.iloc[0] + assert leader["driver"] == "d1" and leader["p_title"] == 1.0 + np.testing.assert_allclose(drivers["exp_final_points"], drivers["current_points"]) + + +class TestCmdSeason: + def _args(self, tmp_path): + import argparse + + return argparse.Namespace(fit_dir=str(tmp_path), out_dir=str(tmp_path), n_trials=50, seed=0) + + def _patch(self, monkeypatch, schedule): + import cli + + idata, data, reliability, driver_rounds, circuits = _fit_inputs() + monkeypatch.setattr(cli, "load_fit", lambda d: (idata, data, reliability, None)) + monkeypatch.setattr(cli, "_load_driver_rounds", lambda: driver_rounds) + monkeypatch.setattr(cli, "_load_circuits", lambda: circuits) + monkeypatch.setattr(sim.season, "_fastf1_schedule", schedule) + + def test_writes_both_championship_csvs(self, monkeypatch, tmp_path): + import cli + + self._patch(monkeypatch, lambda s: _schedule(rounds=4)) + cli.cmd_season(self._args(tmp_path)) + assert pd.read_csv(tmp_path / "drivers_championship.csv").shape[0] == 4 + assert pd.read_csv(tmp_path / "constructors_championship.csv").shape[0] == 2 + + def test_unreadable_schedule_exits_with_a_message(self, monkeypatch, tmp_path, capsys): + import cli + + def broken(season): + raise ConnectionError("rate limited") + + self._patch(monkeypatch, broken) + with pytest.raises(SystemExit): + cli.cmd_season(self._args(tmp_path)) + assert "rate limited" in capsys.readouterr().err + assert not (tmp_path / "drivers_championship.csv").exists() + diff --git a/tests/test_season_backtest.py b/tests/test_season_backtest.py new file mode 100644 index 0000000..bd9c05f --- /dev/null +++ b/tests/test_season_backtest.py @@ -0,0 +1,251 @@ +"""Season-level backtest harness (spec sec 7 season-level metrics, sec 8 +Phase 3; autopilot slice-9). One end-to-end run on the fixture with a real +ADVI fit at tiny step counts; everything else is pure or stubbed.""" +import json +import os + +import numpy as np +import pandas as pd +import pytest + +import backtest +import cli +import model.pace +from backtest import ( + backtest_schedule, legacy_constructor_spearman, run_season_backtest, score_season_cutoff, + season_level_summary, +) +from sim.season import constructors_championship, drivers_championship, remaining_calendar +from train import FEATURE_COLS + +FIXTURE = os.path.join(os.path.dirname(__file__), "..", "data", "driver_rounds_fixture.csv") + + +def _fixture(): + return pd.read_csv(FIXTURE) + + +def _legacy_features(teams=("A", "B", "C", "D", "E"), years=(2022, 2023, 2024), seed=0): + """Synthetic `features.build_features` output: the target is a monotone + function of `early_points_share`, so a fitted Ridge ranks it perfectly.""" + rng = np.random.default_rng(seed) + rows = [] + for y in years: + share = rng.dirichlet(np.ones(len(teams))) + for t, s in zip(teams, share): + row = {c: 0.0 for c in FEATURE_COLS} + row.update(year=y, constructor=t, early_points_share=s, season_points_share=s) + rows.append(row) + return pd.DataFrame(rows) + + +# --------------------------------------------------------------------------- +# End to end on the fixture: real ADVI fit, real chain, real scoring +# --------------------------------------------------------------------------- + +def test_end_to_end_on_the_fixture_writes_a_season_level_block(tmp_path): + # a race-level summary already in the out dir is kept, not overwritten + (tmp_path / "backtest_summary.json").write_text(json.dumps({"n_rounds_scored": 7})) + rows = run_season_backtest([2024], cutoffs=[1], n_trials=200, advi_steps=200, advi_draws=20, + out_dir=str(tmp_path), driver_rounds=_fixture(), + legacy_features=lambda c: _legacy_features()) + + assert rows["status"].tolist() == ["ok"] + summary = json.loads((tmp_path / "backtest_summary.json").read_text()) + assert summary["n_rounds_scored"] == 7 + block = summary["season_level"] + assert block["gate_valid"] is True and block["gate_invalid_reasons"] == [] + assert isinstance(block["v2_beats_legacy_spearman"], bool) + assert block["per_cutoff"]["1"]["n_seasons"] == 1 + assert block["per_cutoff"]["1"]["mean_constructor_spearman_legacy"] == pytest.approx(1.0) + r = rows.iloc[0] + assert -1.0 <= r["constructor_spearman_v2"] <= 1.0 + assert 0.0 <= r["brier_drivers_title"] <= 2.0 and 0.0 <= r["brier_constructors_title"] <= 2.0 + assert 0.0 <= r["coverage80_drivers"] <= 1.0 + for f in ("season_backtest_metrics.csv", "season_2024_r1_drivers_championship.csv", + "season_2024_r1_constructors_championship.csv"): + assert (tmp_path / f).exists() + + +# --------------------------------------------------------------------------- +# The loop with the fit and chain stubbed: leak safety and failure statuses +# --------------------------------------------------------------------------- + +def _tables(teams=("Red Bull Racing", "Ferrari", "McLaren", "Mercedes")): + drivers = pd.DataFrame({"driver": ["VER"], "p_title": [1.0], "p10_final_points": [0.0], + "p90_final_points": [100.0]}) + constructors = pd.DataFrame({"team": list(teams), "p_title": [1.0] + [0.0] * (len(teams) - 1), + "exp_share": np.linspace(0.4, 0.1, len(teams)), + "p10_final_points": 0.0, "p90_final_points": 100.0}) + return drivers, constructors + + +class TestLoop: + def _run(self, monkeypatch, tmp_path, chain=None, fit=None, cutoffs=(1,)): + seen = {} + + def fake_chain(idata, data, reliability, driver_rounds, circuits, **kw): + seen["rounds"] = driver_rounds + seen["calendar"] = remaining_calendar(2024, 1, loader=kw["schedule_loader"]) + return (chain or _tables)() + + monkeypatch.setattr(model.pace, "fit_advi", fit or (lambda *a, **kw: "IDATA")) + monkeypatch.setattr(cli, "forecast_season", fake_chain) + rows = run_season_backtest([2024], cutoffs=list(cutoffs), n_trials=10, out_dir=str(tmp_path), + driver_rounds=_fixture(), + legacy_features=lambda c: _legacy_features()) + return rows, seen + + def test_the_chain_sees_nothing_after_the_cutoff_but_the_real_calendar(self, monkeypatch, tmp_path): + rows, seen = self._run(monkeypatch, tmp_path) + assert rows["status"].tolist() == ["ok"] + assert seen["rounds"]["round"].max() == 1 + assert seen["calendar"]["event_name"].tolist() == ["Saudi Arabian Grand Prix"] + + def test_a_failed_fit_is_recorded_and_voids_the_verdict(self, monkeypatch, tmp_path): + def boom(*a, **kw): + raise RuntimeError("diverged") + rows, _ = self._run(monkeypatch, tmp_path, fit=boom) + assert rows["status"].tolist() == ["fit_failed"] + block = json.loads((tmp_path / "backtest_summary.json").read_text())["season_level"] + assert block["failed_cutoffs"] == ["2024:1 (fit_failed)"] + assert block["v2_beats_legacy_spearman"] is None + + def test_a_failed_chain_is_recorded(self, monkeypatch, tmp_path): + def boom(): + raise RuntimeError("no schedule") + rows, _ = self._run(monkeypatch, tmp_path, chain=boom) + assert rows["status"].tolist() == ["forecast_failed"] + + def test_a_cutoff_with_nothing_left_to_chain_is_a_failure_not_a_skip(self, monkeypatch, tmp_path): + rows, _ = self._run(monkeypatch, tmp_path, cutoffs=(1, 2, 6)) + assert rows["status"].tolist() == ["ok", "no_rounds_after_cutoff", "no_rounds_after_cutoff"] + block = json.loads((tmp_path / "backtest_summary.json").read_text())["season_level"] + assert block["n_cutoffs_expected"] == 3 and block["n_cutoffs_ok"] == 1 + assert block["gate_valid"] is False and block["v2_beats_legacy_spearman"] is None + + +# --------------------------------------------------------------------------- +# Summary: gate_valid / null-verdict discipline +# --------------------------------------------------------------------------- + +def _rows(v2, legacy, statuses=None, cutoffs=None): + n = len(v2) + return pd.DataFrame({ + "season": [2022 + i // 3 for i in range(n)], "cutoff": cutoffs or [(2, 6, 12)[i % 3] for i in range(n)], + "status": statuses or ["ok"] * n, "constructor_spearman_v2": v2, + "constructor_spearman_legacy": legacy, "brier_drivers_title": [0.5] * n, + "brier_constructors_title": [0.4] * n, "coverage80_drivers": [0.8] * n, + "coverage80_constructors": [0.7] * n, + }) + + +def test_complete_run_reports_a_real_verdict_per_cutoff(): + out = season_level_summary(_rows([0.9, 0.8, 0.95], [0.85, 0.85, 0.85]), [2022], [2, 6, 12]) + assert out["gate_valid"] is True + assert out["v2_beats_legacy_spearman"] is True + assert out["per_cutoff"]["6"]["mean_constructor_spearman_v2"] == pytest.approx(0.8) + assert out["per_cutoff"]["6"]["mean_constructor_spearman_legacy"] == pytest.approx(0.85) + assert out["legacy_reference_spearman"] == 0.88 + + +def test_v2_losing_to_legacy_is_false_not_none(): + out = season_level_summary(_rows([0.5, 0.5, 0.5], [0.9, 0.9, 0.9]), [2022], [2, 6, 12]) + assert out["v2_beats_legacy_spearman"] is False + + +def test_a_partial_run_cannot_report_a_verdict(): + # season 2023 was requested but never attempted + out = season_level_summary(_rows([0.9, 0.9, 0.9], [0.1, 0.1, 0.1]), [2022, 2023], [2, 6, 12]) + assert out["v2_beats_legacy_spearman"] is None + assert "2023:2 (not attempted)" in out["failed_cutoffs"] + assert out["n_cutoffs_expected"] == 6 and out["n_cutoffs_ok"] == 3 + + +def test_a_failed_cutoff_cannot_report_a_verdict_and_is_named(): + rows = _rows([0.9, np.nan, 0.9], [0.1, np.nan, 0.1], statuses=["ok", "fit_failed", "ok"]) + out = season_level_summary(rows, [2022], [2, 6, 12]) + assert out["gate_valid"] is False and out["v2_beats_legacy_spearman"] is None + assert out["failed_cutoffs"] == ["2022:6 (fit_failed)"] + + +def test_a_missing_legacy_number_cannot_report_a_verdict(): + out = season_level_summary(_rows([0.9, 0.9, 0.9], [0.1, np.nan, 0.1]), [2022], [2, 6, 12]) + assert out["v2_beats_legacy_spearman"] is None + assert any("legacy" in r for r in out["gate_invalid_reasons"]) + + +# --------------------------------------------------------------------------- +# Scoring against the real final standings +# --------------------------------------------------------------------------- + +def test_scores_against_the_whole_season_not_the_cutoff(): + dr = _fixture() + final = dr[dr["session_type"] == "R"].groupby("abbreviation")["points"].sum() + leader = final.idxmax() # VER, both rounds + drivers = pd.DataFrame({"driver": [leader, "PER"], "p_title": [0.75, 0.25], + "p10_final_points": [final[leader], 0.0], + "p90_final_points": [final[leader], 1.0]}) + _, constructors = _tables() + out = score_season_cutoff(drivers, constructors, dr, 2024, lineup=[leader, "PER"]) + assert out["brier_drivers_title"] == pytest.approx(0.25 ** 2 + 0.25 ** 2) + # leader's interval is exactly the true total (covered); PER's misses + assert out["coverage80_drivers"] == 0.5 + assert out["brier_constructors_title"] == 0.0 + + +def test_coverage_ignores_drivers_the_chain_did_not_simulate(): + dr = _fixture() + drivers = pd.DataFrame({"driver": ["VER", "XXX"], "p_title": [1.0, 0.0], + "p10_final_points": [0.0, 5.0], "p90_final_points": [100.0, 10.0]}) + out = score_season_cutoff(drivers, _tables()[1], dr, 2024, lineup=["VER"]) + assert out["coverage80_drivers"] == 1.0 + + +def test_championship_tables_carry_the_80pct_points_interval(): + pts = np.tile(np.arange(100.0), (2, 1)).T # 100 trials, two entries + d = drivers_championship(pts, np.zeros_like(pts), ["A", "B"], ["T", "T"], [0.0, 0.0]) + c = constructors_championship(pts, np.zeros_like(pts), ["T", "U"], [0.0, 0.0]) + for table in (d, c): + assert table["p10_final_points"].tolist() == pytest.approx([9.9, 9.9]) + assert table["p90_final_points"].tolist() == pytest.approx([89.1, 89.1]) + + +def test_schedule_is_read_off_the_rounds_that_ran_with_sprints_flagged(): + dr = _fixture() + sprint = dr[(dr["round"] == 2) & (dr["session_type"] == "R")].assign(session_type="S") + cal = remaining_calendar(2024, 0, loader=backtest_schedule(pd.concat([dr, sprint]))) + assert cal["round"].tolist() == [1, 2] + assert cal["has_sprint"].tolist() == [False, True] + + +# --------------------------------------------------------------------------- +# Legacy at the matching cutoff +# --------------------------------------------------------------------------- + +def test_legacy_trains_only_on_earlier_seasons(): + feats = _legacy_features(years=(2021, 2022, 2023, 2024)) + assert legacy_constructor_spearman(feats, 2023) == pytest.approx(1.0) + # poisoning a later season's target cannot move an earlier season's number + poisoned = feats.copy() + later = poisoned["year"] == 2024 + poisoned.loc[later, "season_points_share"] = -poisoned.loc[later, "early_points_share"] * 50 + assert legacy_constructor_spearman(poisoned, 2023) == legacy_constructor_spearman(feats, 2023) + + +def test_legacy_without_earlier_seasons_has_no_number(): + assert np.isnan(legacy_constructor_spearman(_legacy_features(years=(2024,)), 2024)) + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + +def test_cli_routes_season_level_to_the_season_harness(monkeypatch): + calls = {} + monkeypatch.setattr(backtest, "run_season_backtest", lambda **kw: calls.update(kw)) + monkeypatch.setattr(backtest, "run_backtest", lambda **kw: pytest.fail("race-level ran")) + args = cli.build_parser().parse_args(["backtest", "--season-level", "--seasons", "2022-2023"]) + args.func(args) + assert calls["seasons"] == [2022, 2023] and calls["cutoffs"] == [2, 6, 12] + assert cli.build_parser().parse_args(["backtest"]).season_level is False