This is risk-model validation, not a strategy backtest. Three
one-day-ahead VaR/ES forecasting methods run over a fixed multi-asset
portfolio, and the deliverable is whether each model's stated coverage
holds on realised data: Kupiec unconditional coverage, Christoffersen
independence and conditional coverage, and the Basel traffic light. Every
forecast for day t is computed from returns up to t−1 only, enforced by
construction and verified by a corruption test
(tests/test_backtest.py::test_no_lookahead_all_methods).
Part of a three-repo series. Pairs-trading proposes the strategy, the overfitting-audit framework convicts or clears it, and this repo (var-backtesting) risk-measures what survives. This repo is the risk stage: the plan was to measure the residual exposure of whatever survived the audit. Nothing did, so the validation machinery is exercised on a fixed benchmark portfolio instead.
Portfolio: SPY, TLT, GLD, IWM, EFA, equal 20% weights, daily log returns, 2010-01-05 to 2026-08-27 (4,187 days; 3,687 forecast days after the 500-day warm-up).
| method | level | breaches | expected | Kupiec uc | Christoffersen ind | Christoffersen cc | verdict |
|---|---|---|---|---|---|---|---|
| historical | 99% | 43 | 36.9 | pass (p=0.32) | REJECT (p=1e-4) | REJECT (p=3e-4) | FAIL |
| historical | 95% | 183 | 184.4 | pass (p=0.92) | REJECT (p=8e-7) | REJECT (p=5e-6) | FAIL |
| ewma | 99% | 69 | 36.9 | REJECT (p=2e-6) | REJECT (p=3e-4) | REJECT (p=2e-8) | FAIL |
| ewma | 95% | 190 | 184.4 | pass (p=0.67) | pass (p=0.30) | pass (p=0.54) | PASS |
| garch-filtered | 99% | 41 | 36.9 | pass (p=0.50) | REJECT (p=0.011) | REJECT (p=0.030) | FAIL |
| garch-filtered | 95% | 182 | 184.4 | pass (p=0.86) | pass (p=0.18) | pass (p=0.41) | PASS |
n = 3,687 forecast days per row; LR tests at 5% significance. Full detail:
reports/coverage_table.{csv,md}, reports/coverage_summary.json.
Realised daily returns against the −VaR/−ES lines; breaches in crimson. The ES line sits beyond VaR by construction (ES ≥ VaR always, unit-tested).
99% VaR, 2020-02-01 to 2020-06-30 (104 trading days, 1.0 expected breaches):
| method | breaches in window | max consecutive |
|---|---|---|
| historical | 10 | 2 |
| ewma | 8 | 2 |
| garch-filtered | 5 | 2 |
Longest full-sample 99% breach streaks land on real events: historical, 3 days 2025-04-03 to 07 (tariff selloff); EWMA, 3 days 2015-08-20 to 24 (CNY devaluation); GARCH-filtered, 2 days 2020-02-24 to 25 (COVID onset). The clustering evidence that drives the Christoffersen rejections is these episodes, not diffuse noise.
At 99%, the level that matters, no method passes everything, and each failure is informative:
- Gaussian EWMA-parametric VaR fails everything: 69 breaches against 36.9 expected. A normal tail is too thin at 99%; the classic under-capitalisation failure mode of parametric VaR, exactly what Kupiec is built to catch.
- Historical simulation gets the frequency right (43 vs 36.9) but clusters breaches (Christoffersen ind p = 1e-4): the 500-day window reacts to volatility only after it has already arrived.
- The GARCH(1,1)-filtered bootstrap is the only method passing Kupiec at 99% (41 vs 36.9) and has the best-behaved breach process, but still fails independence marginally (LR_ind = 6.54, p = 0.011): conditional σ_t cannot fully adapt within multi-day correlation blowouts. Reported as-is; a marginal reject at the best method is more informative than a manufactured pass.
At 95% all three methods pass every LR test: the thinner the tail claimed, the easier the claim is to meet. The Basel traffic light (99% convention) ends green for all three methods on the trailing 250 days.
- Historical simulation (500-day rolling window):
VaR_p = −empirical p-quantile,ES_p = −mean(window ≤ VaR). - Parametric EWMA (RiskMetrics, λ = 0.94, μ = 0):
σ²_t = λσ²_{t−1} + (1−λ)r²_{t−1};VaR_p = −Φ⁻¹(p)·σ_t,ES_p = σ_t·φ(Φ⁻¹(p))/p. - GARCH(1,1)-filtered bootstrap (FHS): fit ConstantMean+GARCH(1,1) by
MLE (
arch), refit every 21 days on an expanding window; one-day σ_t from the recursion;r* = μ̂ + σ_t·ê*with ê* resampled from standardised residuals (10,000 paths, seeded), VaR/ES read off the simulated distribution.
Coverage tests: Kupiec POF LR_uc ~ χ²(1); Christoffersen LR_ind ~ χ²(1)
on the first-order Markov transition of the breach indicator,
LR_cc = LR_uc + LR_ind ~ χ²(2); Basel traffic light on trailing 250-day
breach counts (green 0-4, yellow 5-9 with multiplier step-ups 3.40 to 3.85,
red 10+, multiplier 4.00).
uv sync # pinned deps, Python 3.12
uv run pytest # 28 synthetic unit tests, no network
uv run python experiments/01_run_all.py # full coverage table + charts (~1 min)
uv run python experiments/02_crisis_window.py
uv run pytest -m integration # optional: needs data/cache or networkFirst experiment run fetches prices from yfinance (Stooq fallback) into
data/cache/; afterwards everything is offline and deterministic (seeded
bootstrap).
- Fixed equal weights on log returns imply daily rebalancing; no transaction costs (this is a forecast-validation exercise, not a tradable strategy).
- EWMA assumes μ = 0 (RiskMetrics convention); the GARCH filter uses normal innovations, and non-normality re-enters only through the residual bootstrap.
- The Basel 250-day zones are applied to a 5-asset ETF book, not a bank trading book; reported at 99% only (the convention the thresholds assume).
- What a pass does and does not certify: one-day-ahead coverage frequency and first-order independence of the breach process. It does not certify tail shape beyond the ES estimate, parameter stability across regimes, or behaviour under structural breaks; the marginal Christoffersen reject of the (best) GARCH-filtered model is a live reminder of exactly that.
MIT; see LICENSE.