I built a machine-learning trading model, then independently validated it to a bank-style model-risk standard (Federal Reserve / OCC SR 11-7). The verdict: do not deploy. This repository is that validation — the report, the reproducible checks, and the evidence behind the decision.
Most trading-model repositories exist to show a model that "works." This one shows the opposite discipline: taking a model I built and had every reason to believe in, running it through a strict out-of-sample harness, and documenting the honest finding that its edge does not survive — which is exactly the job of a model validator.
The model is a LightGBM (gradient-boosted) regression over 18 macro/regime features that predicts a daily-swing trade's return and picks the best-scoring trades (463,736 observations, 8-fold walk-forward). Under a strict, leakage-controlled harness:
| Metric (baseline, flat sizing) | Value |
|---|---|
| Trades | 103 |
| Expected value / trade | +$0.09 |
| Win rate | 40.8% |
| Sharpe | +0.019 |
| 95% CI on P&L | [−$81.93, +$106.01] |
| Profitable folds | 4 of 8 |
| Verdict | MARGINAL — edge not distinguishable from zero |
The confidence interval straddles zero and the Sharpe is a rounding error above flat. A clean in-sample result did not survive honest out-of-sample testing. That is the result, and reporting it plainly is the point.
The independent checker (deepchecks) reached the same conclusion from a different
angle: the model beats a naïve baseline by 0.0038%, the training and test
periods are so different a classifier separates them perfectly (drift AUC 1.00),
and out-of-sample R² degrades 138.9% — while the data and the harness both check
out clean. So the model fails because there's no signal to find, not because of a
bug or a bad test. A second tool, giskard, scanned the model a different way and
independently flagged its fragility: small noise on one feature flips a quarter of
its predictions.
Tuning the exit bracket on the training folds produced a strong in-sample number that flipped negative out-of-sample:
| Exit rule | In-sample (train) | Out-of-sample (test) |
|---|---|---|
Best-on-train (-5% / +10% / 10d) |
+$49.32 | −$67.43 |
Baseline (-3% / +6% / 5d) |
+$13.57 | −$4.18 |
The best configuration on training data was the worst out-of-sample — a clear case of why in-sample optimization isn't evidence, and why validation is done on held-out data. Catching this before capital is committed is the whole point of the discipline.
One positive result was robust: confidence-weighted position sizing improved risk-adjusted return (Sharpe +0.058 vs. +0.019 flat), while volatility- and combined-sizing degraded it. A real, if modest, finding — reported alongside the negatives, not in place of them.
I am an accounting student who builds his own AI/ML systems. This artifact sits at the intersection those two skills define — the independent, evidence-based validation of a statistical model that feeds a financial decision — which is the core of bank Model Risk Management (SR 11-7) and the emerging AI-assurance function inside audit.
It demonstrates, on real work:
- Leakage-controlled evaluation — walk-forward folds with separate fit / validation / test windows and a 10-day embargo; no look-ahead.
- Out-of-sample discipline — every "winner" is judged on held-out data; in-sample results are labeled as such and discounted.
- Reconciliation / benchmarking control — the trade simulator was independently re-computed against stored results and matched 103 / 103 trades to 0.0000% mean difference, confirming the P&L engine is sound before its outputs are trusted.
- Honest reporting — a marginal/negative verdict, stated plainly, with the confidence interval and failure modes shown rather than hidden.
The report is organized around the three elements of the Federal Reserve / OCC SR 11-7 supervisory guidance on model risk:
- Conceptual soundness — are the model's design and features defensible?
- Ongoing monitoring & outcomes analysis — does it hold up out-of-sample and across regimes?
- Benchmarking & process verification — do independent re-computations reproduce the model's own reported results, and does it beat a naïve baseline?
It closes with a short hypothesis on why the edge wasn't there — what a model's failure at this scale says about where retail-accessible edge has gone.
Full report: report/model_validation_report.md.
ml-model-validation/
├── README.md # this file
├── requirements.txt # pinned environment (Python 3.11)
├── report/
│ └── model_validation_report.md # the full SR 11-7-style report
├── validation/
│ ├── run_validation.py # rebuilds the model, runs the deepchecks suites
│ ├── run_giskard.py # second-tool cross-check (giskard scan)
│ ├── make_figures.py # generates the report figures
│ ├── prepare_data.py # exports the portable data slice
│ ├── data/validation_slice.parquet # the data the report is built on (~17 MB)
│ ├── data_integrity.html # interactive deepchecks reports
│ ├── train_test_validation.html
│ ├── model_evaluation.html
│ ├── giskard_scan.html # giskard second-tool scan
│ └── summary.txt / summary.json / giskard_summary.txt # pass/fail registers
└── figures/
├── fig1_feature_drift.png
├── fig2_generalization.png
└── fig3_learning_curve.png
The data slice ships with the repo, so the validation runs out of the box:
uv venv --python 3.11
uv pip install -r requirements.txt
python validation/run_validation.py # deepchecks reports + summary
python validation/run_giskard.py # giskard second-tool cross-check
python validation/make_figures.py # the report figures(deepchecks 0.19 requires NumPy 1.x / scikit-learn < 1.6, so the environment is
pinned to run on a clean stack rather than a patched one. validation/prepare_data.py
regenerates the data slice from the original dataset and is only needed if you have
that source; the committed Parquet is otherwise self-sufficient.)
- This validates a research/backtest model that was never traded live — it was built, tested, and shelved on the strength of exactly the findings here.
- The strategy targets small-capital retail conditions; results should not be generalized to other regimes or scales.
- The committed data is a small, portable Parquet slice (date + 18 features + label). No brokerage credentials, account data, or live keys are included — it is a clean, self-contained slice of a larger private project.
Author: Jesse Laguna — accounting + AI. Built and validated end-to-end.