M5-Inspired Retail Forecasting across 3 models

I built this project and its evaluation harness to compare three models: LightGBM, seasonal naive, and AutoARIMA for retail demand forecasting, then use the winning model to recommend inventory orders. LightGBM won the three-model backtest on the synthetic demo, with the lowest order-cycle WAPE.

The experiment was inspired by the approach that made LightGBM successful in the M5 forecasting competition: learning across related retail sales series using sales history, calendar effects, and prices. The M5 results paper reports that all top 50 methods used machine learning, with LightGBM prominent among them. Their forecast error was more than 14% lower than the strongest statistical benchmark, with the top five improving by more than 20%. That motivated the experiment here: compare LightGBM with statistical baselines, then apply the forecasts to a replenishment policy. The M5 findings are specific to that competition. M5 walkthrough was a resource used afterwards for validation.

LINK TO SLIDES

Approach

I'm a backend Python engineer. I built this in under seven days to:

1. Dust off data science skills

   • learn retail forecasting
   • turn data into a working Python pipeline with
     • validated inputs
     • model comparisons
     • inventory decisions
     • monitoring

2. Demo my Python harness

   I used loopgate_harness to structure development with coding agents and automated quality checks. The harness configures linting, formatting, complexity checks, type checking, security scanning, tests with a 100% coverage threshold, etc. The harness provides checks around iteration; the model choices and limitations are documented below. Loopgate enabled me to quickly build a full end-to-end 3-model pipeline in a domain that is not my area of expertise (data science) with production-engineering practices automated.

The engineering work connects the forecasting experiment to a repeatable workflow with explicit assumptions and testable behavior. A shared LightGBM model learns from multiple products and locations. Features include prior sales, product and location identifiers, prices, calendar effects, and forecast lead time. It produces lower, median, and upper quantile forecasts over a four-week horizon. Seasonal naive, LightGBM, and optional AutoARIMA are compared on the same series and dates across 18 weekly backtest origins. Model selection uses WAPE over the replenishment period. M5 used hierarchical WRMSSE, so these scores are not directly comparable with its leaderboard.

Evaluation harness

I built a shared evaluation harness for all three forecasting models. Each model runs on the same series, 18 weekly forecast origins, and 28-day horizons, using training history before each origin. The harness:

• Validates forecast dates, series keys, and quantile bounds before scoring.
• Measures point-forecast error, bias, pinball loss, and prediction-interval coverage.
• Selects the winner by pooled WAPE over the replenishment period and saves each forecast for inspection.
• Feeds the winner into the ordering policy and monitors all evaluated models for drift.

The completed three-model comparison selected LightGBM. The notebook defaults to two models for speed; enabling INCLUDE_ARIMA reproduces the full comparison.

Example run

The figures below show the notebook's default synthetic demo, not M5 competition results. Running the notebook regenerates both charts from the current backtest.

Run the notebook

m5_forecasting.ipynb contains the complete workflow: data, forecasting, replenishment, and visuals.

uv sync --all-groups
uv run jupyter lab m5_forecasting.ipynb

Run all cells for a repeatable synthetic demo. For M5, download sales_train_evaluation.csv, calendar.csv, and sell_prices.csv from the M5 Accuracy competition into data/, then set DATA_SOURCE = "m5" in the notebook.

SERIES_COUNT defaults to 20 and HISTORY_DAYS to 420 for a manageable M5 sample. Set INCLUDE_ARIMA = True to include AutoARIMA. The notebook replaces local pipeline inputs, outputs, and figures on each run; downloaded M5 files remain intact.

Replenishment experiment

Each store is treated as an independent planning location. Inventory and ordering parameters are modeling assumptions: three or thirty days of stock cover, five delivery days, a twenty-unit minimum order, and two buffer days. The inventory snapshot is placed at the final observed sales date.

The selected forecast drives an order recommendation, compared with a trailing-average policy. Results are written to data/output/:

• backtest_forecasts_*.csv: predictions and actual sales for each forecast origin.
• orders_next_cycle.csv: recommended quantities and comparison with the baseline policy.
• monitoring_signals.csv: changes in error, bias, interval coverage, and active-series counts.

Validation and limits

OMP_NUM_THREADS=1 uv run pytest -q

Tests cover data validation, time boundaries, model behavior, ordering, and monitoring. Runtime CSV files are excluded from Git.

Backtests assume future prices were known at each forecast origin. M5 promotion status is not modeled; forward orders assume the latest regular price. Selection and reporting share overlapping windows, without an independent final evaluation. The default sample is small and selected by file order.

Replenishment results describe a simulation. Service estimates measure historical coverage of a policy target; retail-value comparisons are not inventory costs. Real deployment would also require interval calibration, in-transit stock, capacity constraints, and planner approval.