Most model repositories show someone building a model. A model risk function does the opposite job: it takes a model it did not build, assumes nothing, and tries to find the reason it should not be approved.
This is that harness. It takes anything exposing predict_proba — a
scorecard, a scikit-learn estimator, a vendor API, a file of stored scores —
and returns findings with severities, organised along the three elements
of SR 11-7 and the equivalent sections of the ECB's TRIM guide.
make demo # validate a real scorecard end to end
make test # 60 testsRunning it against the scorecard from hamiid01/credit-scorecard, 9,000-account holdout:
OPINION: APPROVED WITH CONDITIONS
Overall rating: WEAKNESS
PASS: 6 OBSERVATION: 3 WEAKNESS: 1 SEVERE: 0
1. [WEAKNESS] Outcomes analysis — Grade-level backtest (Jeffreys)
1 of 7 grades significantly underestimate their PD.
-> Reassign the PDs of the affected grades.
The finding is real and specific. Grade G5 predicted 18.0% and realised 21.2% across 1,215 accounts, Jeffreys p = 0.0018 — while the portfolio as a whole calibrated almost perfectly at a 1.004 observed/predicted ratio. Aggregate calibration hid a grade-level failure, which is exactly why supervisors test grade by grade rather than in total.
The deliverable is findings, not a metrics table, because the audience is a committee deciding approve / approve-with-conditions / reject. The overall rating is the worst finding, deliberately not an average: a model with one severe finding and twenty passes is not 95% approvable, it is blocked.
Two models scored on the same accounts have correlated AUCs. Comparing their confidence intervals for overlap is the wrong test and is badly conservative — overlapping intervals can still be a highly significant difference.
In the demo the champion scores AUC 0.7603 and a random forest 0.7693. The two are correlated at 0.90, and DeLong puts that 0.0089 gap at p = 0.0015. A naive independent comparison would have called it noise.
The harness then separates two questions the usual write-up conflates:
- Is the gap real? Statistics. Here: yes.
- Is it worth having? Judgement. Here: 0.0089 AUC is below the 0.03 materiality threshold, so it is reported as an observation, not a finding. An unconstrained booster that beats a monotonic scorecard by 0.009 has not earned the loss of explainability and adverse-action reasons.
Also informative: plain logistic regression on the same inputs scores 0.0458 worse (p = 2.5e-18), so the scorecard's WOE binning is doing real work rather than ceremony.
Implementation is the midrank form of Sun & Xu (2014), O(n log n) instead of
DeLong's original O(n²), and tested to reproduce roc_auc_score to 1e-12
including on heavily tied scores — a grade scale is nothing but ties, and
without midranks the AUC comes out wrong.
| test | what it catches | weakness |
|---|---|---|
| Central tendency | portfolio-level level bias | blind to offsetting grade errors |
| Hosmer–Lemeshow | deciles diverging | power scales with n; grouping-dependent |
| Spiegelhalter z | miscalibration, no grouping | less familiar to committees |
| Jeffreys (ECB) | grade-level underestimation | needs a grade assignment |
On the demo the tests disagree: Hosmer–Lemeshow rejects calibration (p = 0.00089) while Spiegelhalter accepts it (p = 0.10). The harness says so explicitly rather than quoting whichever supports its conclusion, and rests the opinion on Spiegelhalter plus the effect size — reliability error 0.0004, immaterial — because Spiegelhalter needs no arbitrary decile count.
test_hosmer_lemeshow_gains_power_with_sample_size pins the underlying
problem: the same 8% relative bias is accepted at n=500 and rejected at
n=80,000.
Jeffreys over the binomial test for grade backtests, because a grade with zero defaults is routine at the safe end of a portfolio and the binomial test degenerates to p = 1 there — it cannot distinguish "safe" from "no data". The Beta(D+½, N−D+½) posterior stays defined. All grade tests are one-sided against underestimation: overshooting a PD is expensive, undershooting causes a capital shortfall.
Every sensitivity test treats the model as opaque, which is the only approach that works uniformly across a scorecard, a booster and a vendor API:
- Permutation ablation — shuffle an input, measure the AUC lost. On the
demo
PAY_0carries 41.2% of the model's discriminatory lift (below the 50% threshold, so a pass, but the number is on the record). - Missing-input handling — blank each input and see whether the model still answers. Feeds break; a model that raises in production is an operational finding whatever its Gini. The scorecard handles all 11.
- Perturbation and named stress scenarios, expressed in standard deviations so a downturn can be stated the way a committee discusses it.
The usual starting position for a vendor or legacy model is a CSV of scores and realised outcomes, with no executable model. Backtesting still works:
python -m model_validation scores.csv --target default --pd-column pd --grade-column gradeExit code is non-zero when the opinion is NOT APPROVED, so a monitoring job can gate on it.
What cannot be run — sensitivity, missing-input handling, benchmarking — is reported as a coverage finding rather than quietly skipped. A gap in validation coverage is itself something the committee needs to see.
Every severity in this harness comes from a number in
thresholds.py — minimum Gini 0.30,
central tendency band 0.80–1.25, materiality gap 0.03 AUC, and so on.
They live in one dataclass because the thresholds are the validation
policy: two institutions running identical tests can reach different
opinions, and the difference should be inspectable and arguable rather than
buried in branches. None of them are laws; they are the common supervisory
conventions, and a model risk function should override them.
test_thresholds_are_what_drive_severity checks that changing them changes
the opinion on identical evidence.
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"from model_validation import validate, render
report = validate(model, X_dev, y_dev, X_val, y_val,
development_gini=0.55, grades=grades)
report.opinion # 'APPROVED WITH CONDITIONS'
print(render(report))src/model_validation/
findings.py Finding / Severity / ValidationReport
protocol.py the minimal model contract; ArrayModel for stored scores
discrimination.py Gini/KS/AUC, stratified bootstrap, DeLong
calibration.py central tendency, Hosmer-Lemeshow, Spiegelhalter, Jeffreys
stability.py PSI, CSI, migration matrices, Herfindahl concentration
sensitivity.py ablation, perturbation, missing inputs, stress scenarios
benchmark.py champion vs challenger with DeLong
thresholds.py the judgement policy, in one place
validator.py the run; statistics -> findings
report.py rendering, opinion first
experiments/ validating the credit-scorecard model
tests/ 60 tests
The bootstrap is stratified on the outcome. At low default rates an unstratified resample will occasionally contain no defaults at all, where the statistic is undefined; discarding those resamples biases the interval.
Migration matrices alongside PSI. Accounts can swap grades wholesale while
the distribution is unchanged — PSI sees nothing, the migration matrix sees
everything. test_churn_is_invisible_to_psi_but_not_to_migration pins it.
Herfindahl on the grade distribution. A scale whose accounts pile into two grades is not discriminating at the granularity it claims, whatever its Gini. The demo portfolio has 7.0 effective grades out of 7.
Severity is not averaged. ValidationReport.rating is max() over
findings.
- Federal Reserve SR 11-7 / OCC 2011-12, Supervisory Guidance on Model Risk Management
- ECB, Guide for the Targeted Review of Internal Models (TRIM)
- ECB, Instructions for reporting the validation results of internal models (the Jeffreys test for PD backtesting)
- DeLong, DeLong & Clarke-Pearson (1988), Biometrics 44(3)
- Sun & Xu (2014), IEEE Signal Processing Letters 21(11) — the fast midrank algorithm
- Spiegelhalter (1986), Statistics in Medicine 5(5)
MIT