Football Forecasting Lab

An experimental football forecasting and sports-analytics project. It produces Home/Draw/Away probabilities, keeps live forecasts, retrospective backtests and synthetic demo records separate, and evaluates a frozen chronological holdout with proper probabilistic scores.

Probabilities are uncertain estimates, not certainties or financial recommendations. Retrospective performance does not establish future accuracy or suitability for real-world decisions.

What is included

• Temporal feature engineering using only information available before kickoff
• Expanding-window out-of-fold goal-model stacking
• Fixed 50/50 blend of calibrated XGBoost and independent-Poisson score probabilities
• Separate chronological training, calibration and test windows
• Accuracy, log loss, unscaled multiclass Brier score and calibration error
• Training-derived majority-class and prior-probability baselines
• FastAPI reads and guarded prediction generation
• Streamlit views that keep live, backtest and synthetic demo contexts separate
• Deterministic, credential-free synthetic data for interface testing

CI is configured for Python 3.11 through 3.14. The locked environment is managed with uv.

Installation

Install a supported Python version, Git, Make and uv, then clone the repository and install the locked development environment:

git clone https://github.com/dk3yyyy/football_predictor.git
cd football_predictor
make install

make install runs uv sync --locked --extra dev and creates .venv. Do not commit .env, downloaded provider data, generated SQLite databases or model artifacts.

Architecture and data boundaries

flowchart LR
    S[External football sources] --> M[(SQLite matches)]
    M --> F[Pre-kickoff feature pipeline]
    F --> T[Chronological train / calibrate / test]
    T --> C[Calibrated XGBoost classifier]
    T --> G[Poisson goal models]
    C --> E[Fixed probability ensemble]
    G --> E
    E --> B[Versioned model bundle]
    B --> P[Prediction service]
    P --> L[(Live prediction rows)]
    R[(Optional stored retrospective rows)] --> API
    D[Synthetic fixture generator] --> X[(Isolated demo database)]
    L --> API[FastAPI]
    R --> UI
    X --> API
    B --> UI[Streamlit evaluation]
    L --> UI
    X --> UI

Loading

run_kind is the storage and reporting provenance boundary: live, backtest, demo and migration-only legacy rows are never combined in dashboard metrics. The current trainer writes only versioned model artifacts; it does not create stored backtest rows. The schema retains backtest for separately generated retrospective records, but the bundle and its holdout contract—not those rows or prediction-log naming conventions—are the authoritative frozen evaluation artifacts.

Offline interface demo

This path needs no credentials or network access. It creates four fictional fixtures with hand-authored probabilities in db/football_predictor_demo.db and does not train a model or create performance evidence.

make demo-data
make run-dashboard

Open the loopback-only Streamlit URL printed in the terminal. When the primary database has no prediction rows, the dashboard selects the isolated demo database and displays a persistent synthetic-data notice. Expect two scheduled fictional fixtures. Accuracy and calibration claims are suppressed in demo context.

To choose a different demo-only file:

DEMO_DATABASE_URL=sqlite:////tmp/football_predictor_demo.db make demo-data
DATABASE_URL=sqlite:////tmp/football_predictor_demo.db make run-dashboard

Repeated demo loads are idempotent. They do not call football providers or write to the default primary database.

Real-data prerequisites and constraints

The repository includes neither a third-party dataset nor a pretrained models/model_bundle.joblib. Real-data commands are therefore not turnkey: they require network access, provider access and enough valid historical matches. They write to the configured database and training creates local model artifacts.

• A Football-data.org API key is required. Account tier and competition coverage determine which historical seasons are available; a free key does not imply access to every configured season.
• backfill.py also attempts FBref collection. That source can throttle or reject automated requests, and each failed league/season is logged and skipped, so the command can finish with partial data.
• Provider coverage, terms, rate limits and response formats can change. Review provider terms before collecting data.
• Competitions and seasons come from config/settings.py and optional FOOTBALL_CURRENT_SEASON / FOOTBALL_SEASONS overrides.
• The model uses results-derived, pre-kickoff temporal features. Stored xG, possession, defensive, odds, injury and lineup fields are not training features because they do not yet have a complete as-of snapshot contract.

Configure a local environment without committing secrets:

cp .env.example .env
# Set FOOTBALL_DATA_API_KEY in .env, then review season and database settings.

The default database is db/football_predictor.db. Set DATABASE_URL to an absolute SQLite URL if isolation is required. Never put a real API key in a tracked file.

Real retrospective evaluation

Apply the schema, load history, train a frozen chronological holdout, and print its report:

make init-db
uv run --locked python backfill.py
uv run --locked python -m models.train
uv run --locked python seed_accuracy.py

Inspect the backfill logs and database coverage before training; backfill.py catches per-source failures and partial collection is possible. At least 80 finished matches are required; training otherwise exits unsuccessfully without creating a bundle. The chronological calibration window must contain all three outcome classes. Other data-shape or split failures also stop training rather than producing a misleading artifact.

models.train freezes the first evaluation split in models/model_bundle.holdout.json, then writes models/model_bundle.joblib and models/model_bundle.manifest.json after successful training. The holdout contract records explicit train, calibration and test row-key manifests, chronological boundaries, and content hashes. Later rows after the frozen test window do not silently move the evaluation set; missing, changed or newly inserted historical rows fail closed. Creating a different holdout requires an intentional new evaluation version and allow_new_evaluation=True through the training API. That evaluation version becomes the persisted model version, so a rotated evaluation cannot share live prediction identity with the previous artifact; the trainer software version is recorded separately. seed_accuracy.py requires the bundle, prints its frozen holdout report, and does not insert retrospective rows into live logs. If no bundle exists, it exits with an instruction to train first.

The training artifact records model/schema/evaluation versions, split boundaries and counts, holdout row-key and content hashes, label order, scores, baselines, calibration method and limitations. The current trainer binds source, competition and season coverage into bundle metadata. The dashboard renders that verified coverage and reports Coverage metadata unavailable only for older or incomplete artifacts rather than inferring it.

Backtesting is not live forecasting

Backtesting evaluates a model on a historical window after fitting and calibration on earlier windows. It does not prove how a future live forecast will perform. If separate retrospective tooling stores run_kind=backtest rows, they remain isolated for provenance; the training command itself does not create them, and the frozen bundle report is the authoritative holdout result.

The repository does not ship downloaded provider data or generated model bundles. A checked-in six-season Premier League rolling-origin report records source URLs and hashes, exact temporal windows, machine-readable fold results and limitations. Across 1,140 untouched test matches, v2 reached 53.77% weighted accuracy versus 45.09% for the majority baseline and 56.70% for normalized Bet365 closing probabilities. It beat naive accuracy, log-loss and Brier baselines in every fold, but beat the bookmaker benchmark in none; this remains an experiment, not evidence of competitive forecasting or betting profitability.

Evaluation method

The dataset is sorted by kickoff time and split into disjoint training, calibration and test windows. Goal-model features for the classifier are generated out of fold with expanding chronological training windows. The fitted classifier is calibrated on the separate calibration window. Its probabilities are blended at a fixed 50/50 weight with Home/Draw/Away probabilities derived from the fitted home and away goal rates under an independent-Poisson assumption. The untouched test window is evaluated once and stored in the versioned bundle.

Metrics shown by the dashboard:

• Selective accuracy: share of correct predictions among rows with one unique highest-probability outcome. Probability ties are abstentions; the dashboard reports the evaluated denominator, abstention count and coverage.
• Log loss: penalizes confident wrong probabilities. Lower is better.
• Multiclass Brier: mean squared error across Home/Draw/Away probabilities, using the unscaled 0–2 range. Lower is better.
• Classwise and maximum ECE: summary calibration error. Lower is better, but estimates from small samples are unstable and bin-dependent.
• Majority baseline: always selects the most common training-window outcome and is compared on the same non-tied holdout rows used by selective accuracy.
• Prior-probability baseline: assigns the training-window class frequencies to every holdout match.

Interpret model metrics beside their baselines, sample size, class support and exact holdout dates. A score above one baseline on one window is not evidence of general superiority. Live accuracy is descriptive operational monitoring and is displayed separately from the frozen holdout. The dashboard calculates rolling live results oldest to newest and discloses the sample and evaluation window.

Live forecasting

Live generation is a separate workflow. It requires a successfully trained model bundle plus scheduled fixtures and earlier finished matches in the configured database. It always writes or previews run_kind=live; it does not recreate the frozen backtest. Persisted live rows require valid UTC creation and kickoff timestamps with creation strictly before kickoff. Storage derives correctness from recorded probabilities and the actual outcome, while API and dashboard reporting exclude older invalid live rows that bypassed this contract.

Set ENABLE_MUTATION_ENDPOINTS=true and a long random MUTATION_API_KEY in the untracked .env file. Start the API in one terminal, then use authenticated preview mode from another terminal; preview runs inference without writing prediction rows:

make run-api

read -rsp 'Mutation key: ' MUTATION_KEY; echo
curl -X POST \
  -H "Authorization: Bearer ${MUTATION_KEY}" \
  'http://127.0.0.1:8000/predictions/generate?preview=true&max_matches=10'

This preview can legitimately return generated_count: 0 when the bundle or upcoming fixtures are absent. Persistent generation uses preview=false and should only be enabled after checking the database, model identity and destination. Generation is synchronous, single-instance and capped at 100 matches.

API and dashboard usage

Start the dashboard:

make run-dashboard

Start the API:

make run-api

The API listens on http://127.0.0.1:8000; interactive documentation is at http://127.0.0.1:8000/docs while the server is running. Useful read-only endpoints are:

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/ready
curl 'http://127.0.0.1:8000/predictions/upcoming?run_kind=live&limit=20'
curl 'http://127.0.0.1:8000/metrics?run_kind=live&model_name=temporal-stacked-xgboost&model_version=3.1.0&artifact_id=<bundle-build-id>'

CORS and mutation endpoints are disabled by default. CORS requires an explicit CORS_ORIGINS allowlist; wildcard origins are rejected. Enabling prediction generation requires both ENABLE_MUTATION_ENDPOINTS=true and a long random MUTATION_API_KEY; callers must send it as a Bearer token.

Upcoming reads expose live or demo. The /metrics endpoint additionally accepts backtest, requires an explicit model_version, and requires an explicit artifact_id, so it reports evaluated count, correct count and accuracy for one immutable model artifact and run kind at a time. Only non-demo live/backtest rows with complete outcomes are evaluated; demo requests return provenance and a non-performance notice without synthetic accuracy. Proper probabilistic metrics come from the frozen bundle and dashboard Evaluation page. legacy remains migration-only. API validation rejects unknown fields, unsupported run kinds and oversized requests.

Scheduler

ENABLE_SCHEDULER=true SCHEDULER_RUN_ON_STARTUP=false \
  uv run --locked python -m scheduler

The explicit enable flag starts scheduled jobs without an immediate fetch. Set SCHEDULER_RUN_ON_STARTUP=true only when an immediate live fixture fetch is intentional and provider credentials are configured. Jobs coalesce delayed runs, reject overlap and log partial failures. The in-memory scheduler is suitable for one preview process, not high availability.

Tests and quality gates

The pytest suite is offline and needs no provider credentials:

make test

Run individual and aggregate quality gates with:

make lint
make format-check
make type
make coverage
make security
make check

make check verifies the lockfile, Ruff lint/format and mypy over every tracked Python file, coverage across the API, dashboard, scheduler, migrations, model training/serving and supporting packages with a 69% minimum, installed-package consistency, dependency advisories and tracked-file secret scanning. make security and therefore make check may need network access for current advisory data. GitHub Actions runs the quality suite on Python 3.11, 3.12, 3.13 and 3.14.

Focused dashboard and documentation contracts:

uv run --locked --extra dev pytest tests/test_dashboard.py tests/test_documentation.py -q

Focused scheduler, database and synthetic-data contracts:

uv run --locked --extra dev pytest \
  tests/test_scheduler.py tests/test_settings.py tests/test_database.py \
  tests/test_demo_safety_contracts.py -q

Data access and provenance

Football-data.org is the main authenticated fixture/result source. Other scraper modules are experimental and may be affected by site changes, robots policies, throttling or incomplete historical coverage. The project does not bundle third-party datasets.

Schema changes use explicit Alembic migrations. Before an SQLite database advances to the current 0005_match_detail_source revision, SQLite's online backup API writes *.pre-0005_match_detail_source.bak with owner-only permissions and verifies its integrity, including committed WAL state. The suffix follows the current migration head as revisions advance. Destructive downgrade is disabled; stop application writers and restore that backup to roll back. Importing the API, database or scheduler does not create tables.

Synthetic rows use fictional team names, source=synthetic-demo, model_name=synthetic-demo-model, model_version=fixture-v1 and run_kind=demo. They are interface fixtures, not observations or model output.

Known limitations and responsible use

• Football outcomes are noisy and affected by information absent from this feature set.
• Provider availability, league coverage and historical completeness vary.
• Calibration and classwise metrics are unstable on small holdouts.
• A historical holdout can still become stale under distribution shift.
• Live accuracy alone does not measure probability quality.
• The API /metrics route does not expose log loss, Brier score, calibration error or evaluation dates.
• Current model bundles record source, competition and season coverage; older or incomplete artifacts remain explicitly unavailable.
• Expected goals are model estimates, not exact-score forecasts.
• The scheduler is disabled unless ENABLE_SCHEDULER=true; enabled instances coordinate jobs through expiring database-backed owner leases.
• Live feature reconstruction currently fails closed above 5,000 finished matches until a persisted temporal-state design is added.

Use the project for learning, reproducible forecasting experiments and sports analytics. Do not treat its output as a guarantee, financial recommendation, safety-critical signal or substitute for independent judgment. Verify data provenance, model version, sample size and evaluation window before communicating any result.

Project layout

api/            FastAPI application and request validation
dashboard/      Streamlit UI and testable query/presentation helpers
config/         Runtime settings and fixed safety limits
db/             SQLAlchemy schema, migrations and synthetic demo loader
features/       Temporal feature engineering
models/         Training, bundle persistence and inference
scrapers/       External data adapters
tests/          Offline unit and integration tests
backfill.py     Historical loading entry point
seed_accuracy.py  Read-only frozen holdout report

License

MIT. See LICENSE.