Compare model runs, and say what a comparison could have detected

[chross22]

Two functions for comparing model runs, per the scope agreed: the test and the power curve, pairing on shared stations.

compare_runs() — is the gap real?

Refits nothing. Every run already stores its held-out predictions with the station index and the fold, so the per-fold metrics are there to be paired; this adds only the comparison.

compare_runs(list(rf = rf, gam = gam, weak = weak))
#>   comparison reference_score comparison_score difference   lower   upper
#> 1        gam           0.854            0.892     0.0384  0.0141  0.0627
#> 2       weak           0.854            0.654    -0.1997 -0.2964 -0.1030
#>   p_value detectable n_stations
#> 1 0.01177     0.0325        684
#> 2 0.00458     0.1294        684

The test is corrected_paired_test() — the same Nadeau–Bengio correction the jackknife uses, for the same reason — but two-sided, since asking which of two models is better has no direction the way leaving a covariate out does.

detectable is the column that earns its place. It is the smallest true difference the comparison would have found at 80% power, given the fold-to-fold variance it actually saw. "Not significant" on its own conflates these models perform alike with this survey could not tell them apart. It is deliberately not observed power, which is a deterministic function of the p-value and carries no new information (Hoenig & Heisey 2001).

power_curve() — what would it take?

power_curve(dat, list(full = config, starved = starved),
            fractions = c(0.25, 0.5, 1), replicates = 3)
#>   fraction n_stations replicates difference std_err df power detectable
#> 1     0.25        171          3     -0.316  0.1169  4 0.428      0.434
#> 2     0.50        342          3     -0.267  0.0893  4 0.632      0.332
#> 3     1.00        684          3     -0.272  0.0339  4 1.000      0.126

Everything is refitted at every size — a model trained on half the stations is a different model, not the same one evaluated on fewer, and building the curve the cheap way would measure something else. Both runs see the same subsample and the same folds at every point, which keeps the comparison paired the whole way down. It reuses the jackknife's fold_scores(), so it pays for fitting and scoring and nothing else.

Pairing

On the station index, not row order — tune returns folds in its own order and two runs need not agree on it. Runs covering different stations (what different covariate sets produce, through different missingness) are compared on what they share and warn with the count, rather than intersecting silently. There is a test that shuffling one run's rows changes nothing.

Verification

• 1184 tests passing, 0 failures, run under _R_CHECK_LIMIT_CORES_=TRUE — the variable that broke CI on the last PR.
• 15 new tests. The behavioural ones are asserted, not eyeballed: a covariate-starved run is found 0.20 of AUC below the full model at p = 0.005; the curve climbs 0.43 → 1.00 as stations grow while detectable falls 0.43 → 0.13; minimum_detectable() and achieved_power() are checked to be inverses.
• Citation check run locally through the shared distsamp engine, not just assumed: the new Hoenig & Heisey DOI is registered in tools/citations.csv and all five sections pass.

Documented limits

Stated in the docs and the README rather than left for a reader to find: the curve's target difference defaults to the observed one, so a gap that is itself mostly noise yields a curve about a size that may not be real; and these are differences in a bounded metric, so the normal-theory interval behind detectable degrades as AUC approaches 1.

corrected_paired_test() gains an alternative argument, defaulting to the one-sided behaviour the jackknife already relied on.

🤖 Generated with Claude Code