Test covariates by jackknife, and fit ensembles of model types

[chross22]

Two features, both off by default and both driven from the config, plus a YAML bug found on the way.

Testing covariates by jackknife

covariates.jackknife refits the model without each covariate in turn, over the same cross-validation folds as the main fit, and reports how much worse it ranks stations. Alongside it goes the other half of the classical jackknife — the model on that covariate alone — because the two answer different questions:

	says
score_without	what only that covariate carries (its unique contribution)
score_only	what it carries at all, duplicated or not

A covariate can score high on one and nothing on the other. That combination — information that is real and duplicated — is a very different situation from information that is not there, and ranking on contribution alone treats them identically.

The significance test

Two, reported side by side:

• p_value — one-sided on the per-fold differences, with the variance correction of Nadeau & Bengio (2003). The correction is the load-bearing part: a plain paired t-test treats the folds as independent when any two training sets share most of their rows, so it finds significance almost everywhere (Dietterich 1998). There is no unbiased estimator of the variance of k-fold CV (Bengio & Grandvalet 2004), so this inflates the naive variance by 1/k + 1/(k-1) instead. p_adjusted is Holm across covariates.
• parametric_p — drop-in-deviance LRT for glm, mgcv's approximate term p-value for gam, NA for rf/brt. Which is why the fold test is the default criterion: it means the same thing for all four model types.

Dropping is opt-in

drop defaults to false. The run reports the table and says what dropping would have removed. A covariate that fails this test is one the other covariates already account for on these stations, which is a statement about collinearity in this sample at least as much as about ecology — depth and surface temperature carry much of the same information on a shelf, and the test will call either one redundant depending on which the model reached for first. keep protects a covariate that is in the model because the study is about it; min_predictors is a floor.

What it does remove goes through the existing covariates.exclude, so a dropped covariate is still fetched for anything that needs it as an ingredient.

Ensembles of model types

model.type: ensemble fits several algorithms on the same data and combines them — BIOMOD_EnsembleModeling() from the pipeline this package replaces. Four rules (mean, weighted_mean, median, committee), all computed and written; the config picks which becomes the suitability layer. The result is a drop-in for a fit_patch_model() object, so project_patch_model() projects it unchanged.

The evaluation is the ensemble's own, not the average of its members'. Every member is fitted on the same folds from the same seed, so held-out predictions line up by .row, and the combined out-of-fold predictions go through the same evaluation_table() / optimal_threshold() / bootstrap_evaluation() a single model does. Averaging member scores would report the ensemble as the average of its parts, which is not what an ensemble does — combining members that make different mistakes beats all of them, combining members that make the same mistakes does not, and only a cross-validated ensemble prediction tells those apart. On the vignette's mock run the ensemble reaches 0.903 out of fold against members at 0.892 and 0.901.

Two different things are called an ensemble now, and they are independent:

	combines over	spread column
model.ensemble	algorithms	algorithm_sd
projection.uncertainty	resamples of the data, within one algorithm	suitability_sd

Both can be on; each member carries its own resample interval, those replicates are pooled by member weight, and algorithm disagreement is reported on top. uncertainty_layers() is renamed projection_layers() for the same reason (internal, never exported).

The YAML bug

Found while testing. YAML 1.1 — what yaml::read_yaml() parses — reads a bare n as the boolean false, along with y, yes, no, on, off. Right for a value, wrong for a key, and silent: a derivoce step written

- type: lag_covariate
  vars: [CHL]
  n: 2

parses to a list whose key is named FALSE, so spec$n is NULL and R/derivoce.R falls back to a one-month lag. The config asked for two, the run used one, nothing said so. cfin_gom.yaml hid it by asking for the value the fallback already gives.

yaml's handlers receive the scalar as it was written, so the source text is recoverable — but a handler cannot see whether it is being called for a key or a value, so this marks each boolean scalar and then walks the finished structure: text in a name position is the key the file wrote, text in a value position becomes the logical it meant. The marker is a string prefix rather than an attribute, because it has to survive yaml collapsing a sequence of scalars into an atomic vector.

Only the read side was affected — yaml::as.yaml() already quotes these keys, so a config save_config() wrote has always been read correctly by anything.

Parallelism

Both features spread their model fits over one shared helper, which forks — so Windows runs sequentially and says so rather than pretending. It is one fit per covariate per fold either way.

Verification

• R CMD check --no-manual: Status: OK — 0 errors, 0 warnings, 0 notes, with the vignette built and the suite run inside it.
• Test suite: 1118 passing, 0 failures, 0 errors. 57 new tests across test-jackknife.R, test-ensemble.R, and the YAML cases in test-config.R.
• The intermediate commit was checked out into a separate worktree and tested on its own (971 passing) so the history has no broken step.
• Ran both features together end to end and confirmed a projection carries algorithm_sd, suitability_sd, and novelty as distinct layers.
• Every DOI and URL added to the README resolves.

Version bumped to 0.2.0. No exported function changed meaning.

Retiring a stale test file

tests/testthat/test-citations.R (7 tests) tested inst/tools/check_citations.R, which bdb47bf retired in favour of the shared engine in chross22/distsamp. All 7 have skipped silently on every run since — and a skipped test reads as a passing test in most summaries. Deleted rather than moved: every one exercises the retired script's internals through sys.source(), so there is nothing taupatch-specific to port.

distsamp's test-citation-engine.R already covers 2 of the 7 (the DOI styles and the trailing punctuation). The other 5 have no visible counterpart there — doi.org de-duplication, the two year-extraction cases, live-vs-dead handle, and which files get scanned. Whether those still describe how the shared engine behaves is a question about that engine rather than this repo, so I have not touched distsamp. Worth a look, and the other migrated repos likely carry the same stale file.

test-citation.R (singular) is untouched — it tests inst/CITATION, is offline, and has been passing throughout.

The suite now runs with 1122 passing and zero skips.

🤖 Generated with Claude Code