Forecasting on the M4 competition benchmark — 414 Hourly and 359 Weekly series, scored on M4's own holdout so the numbers are comparable to published work rather than to a split I invented. Built by a third-year Applied Computer Science (AI) student.
Nobody staffs a warehouse against the mean. They staff against the upper bound. So this project treats the prediction interval as the deliverable and point accuracy as the easy part — because a nominal 95% interval that covers 85% of the time is not slightly imperfect, it is wrong, and no sMAPE will ever say so.
1. Nothing achieves its nominal coverage on Hourly. The best is 86.6%.
Every method here advertises a 95% interval. None delivers one:
| method | interval | coverage (nominal 95%) | width | MSIS |
|---|---|---|---|---|
| seasonal_naive | empirical | 86.6% | 4.31 | 8.82 |
| seasonal_naive | analytic | 85.2% | 6.06 | 11.06 |
| naive | empirical | 82.8% | 4.87 | 12.10 |
| theta | analytic | 91.3% | 23.52 | 33.30 |
| naive2 | analytic | 90.2% | 23.52 | 33.19 |
Theta and naive2 look closest to nominal — at nearly four times the width. That is the trap in reading coverage alone: any target is reachable by widening until the interval is useless. MSIS charges for width, and on that they are the two worst methods on the board.
2. Measuring your errors beats trusting your model. Same point forecast,
two ways of putting an interval around it. For seasonal_naive, empirical
residual quantiles give better coverage (86.6% vs 85.2%) at 29% narrower
width — and MSIS drops from 11.06 to 8.82. The model's analytic band assumes
Gaussian, correctly-specified residuals; both are false, in the same direction.
3. The baseline nobody reports wins outright. On Hourly, seasonal_naive
takes an OWA of 0.843 — best point accuracy and best intervals. Theta, the
method that won M3, scores 1.013: worse than the naive2 baseline it is
measured against. On Weekly it is worse still, at 1.288.
| Hourly | sMAPE | MASE | OWA |
|---|---|---|---|
| seasonal_naive | 13.91 | 1.19 | 0.843 |
| theta | 15.41 | 2.11 | 1.013 |
| naive2 (baseline) | 17.36 | 2.27 | 1.000 |
| naive | 43.00 | 11.61 | 5.118 |
My first empirical intervals covered 44–49% on a nominal 95%. The cause was not subtle once seen: I estimated the 2.5% and 97.5% quantiles from three backtest folds. A tail quantile of three numbers cannot reach past their minimum and maximum, so the "95% interval" was really about a 50% one.
What makes it worth writing down is that it passed every test I had, because
every test I had asked whether the maths ran, not whether an interval covered
anything. The fix is a fold count derived from available history (up to 24) with
a floor of 12 before tail quantiles are trusted at all — below that it falls
back to a Gaussian scaling of the same residuals, which uses them without
pretending to resolve their tails. tests/test_fc.py now asserts end-to-end
coverage, which is the test that would have caught it.
- Weekly has no seasonality in M4's setup (
m=1), sonaive,naive2andseasonal_naiveare the same forecast there and report identical numbers. That is correct behaviour, not a bug, and it is why the Weekly table looks degenerate. - I have not compared these against the published M4 leaderboard. OWA here is computed against my own Naive2 implementation, which is the competition's definition but not necessarily identical to their code to the decimal. Ranking claims are internal to this repo.
- Two frequencies, not six. Hourly and Weekly are 773 of M4's 100,000 series. Hourly is strongly seasonal and Weekly is not, which is a deliberate contrast, but Monthly and Quarterly dominate the real benchmark and are absent.
- No ML models. Only statistical baselines. The M4 finding that pure ML underperformed statistical methods is well known; testing it myself is the obvious next step and is not done here.
Notably, the Weekly analytic intervals are well calibrated (94.9% against 95%) while the Hourly ones are not (85.2%). The same construction, honest on one frequency and not the other — which is the argument for measuring coverage per dataset rather than trusting a method's reputation.
make setup && make testmake data && make backtestData is fetched from the public M4-methods repository. No credentials, no API keys.
MIT. M4 data © the M4 competition organisers.