Take predictions as well as models, and MESS a data frame

[chross22]

Two generalisations, both prompted by trying to use fancyfx from a cross-validated pipeline (taupatch) and finding the door shut. Both are things fancyfx wants on its own terms, and neither changes behaviour for any existing caller.

1. held_out() — evaluate predictions you already have

Every evaluation function here re-predicts. That is the right default — it keeps the scored predictions and the model provably in step — but it assumes the caller holds a model that can reproduce them, and a cross-validated workflow does not.

Under k-fold CV each observation is predicted by the one fold model that did not see it. The honest predictions are spread across k models, none of which is the final fit. By the time a pipeline has a single model to hand it has either discarded them or kept them with nothing to pass them to, and evaluation_pairs() offered no way in. Re-predicting from the final model on the same rows answers a different and more flattering question.

pairs <- held_out(cv$observed, cv$predicted)
plotROC(pairs, folds = cv$fold)
plotThreshold(pairs, folds = cv$fold)
plotCalibration(pairs)

evaluation_pairs() short-circuits on it, so threshold_metrics(), calibration_estimates() and the three plots over them work unchanged, folds included. newdata stops being required when the predictions arrive with the object, and the error when it is still missing now names held_out() as the alternative.

Two things it deliberately does not do. It cannot verify the predictions are out of sample — nothing in two numeric vectors records what made them — so in.sample is taken on trust, defaults to FALSE, and can be set TRUE to get the same annotation the model path applies when it recognises its own training data. And it cannot serve plotImportance() or permutation_importance(), which shuffle a predictor and re-predict; that needs a model by construction rather than a record of what one once said.

2. mess() on a data frame, and the limiting covariate

x may now be a data frame of covariate columns as well as a SpatRaster, and the return follows the input. Pipelines that hold a projection as a table of cells rather than as a raster had to round-trip through terra to score one, which is a tax rather than a service.

limiting = TRUE reports which covariate produced each cell's score. The surface says a cell is novel; this says what made it so, which is usually the half worth acting on — "this shelf is extrapolated" is a shrug, "extrapolated because its chlorophyll is higher than anything in training" is a decision about whether to widen the training window or clip the map. A mess_variable column on the data frame side; a categorical mess_variable layer on the raster side, since a raster cannot hold a character and the names have to live in the levels table.

Off by default, so every existing caller's return shape is unchanged.

Verification

622 tests passing, 0 failures, including 20 new ones.

The checks that carry the weight are the agreement checks, because both changes add a second way to reach existing arithmetic and the risk in that is drift:

• Scored through a model and through held_out(), the metric table, AUC, prevalence and n come back identical — with folds and without. Same for calibration_estimates(), including the calibration intercept/slope and the Brier score.
• The same cells scored as a raster and as a data frame give identical MESS values, and the raster's limiting layer agrees with the frame's limiting column.
• mess() was also scored against an independent implementation of the same method — the one in taupatch, written from Elith, Kearney & Phillips (2010) rather than from this code — and the surfaces and the culprit variables agree exactly. That is the more convincing of the two, since it shares no line of code with this package.

One existing test changed

test-spatial.R:173 asserted mess(data.frame(...), training) errors with "must be a SpatRaster" — the exact restriction this lifts. It now asserts what replaced it: a data frame is read, and a list is still refused. Flagged because changing an existing assertion deserves more scrutiny than adding one.

🤖 Generated with Claude Code