✈️ Flight Delay Prediction System

Predicting whether a US domestic flight will arrive 15+ minutes late — from raw DOT data to a served, containerized model.

🌍 Try the Project

Service	Link
🖥️ Live Dashboard	https://flight-delay-prediction-rbggjwfpafrechjrgbedoy.streamlit.app
⚡ FastAPI Backend	https://flight-delay-prediction-1.onrender.com
📚 Swagger API Docs	https://flight-delay-prediction-1.onrender.com/docs

Prediction form (left) takes raw flight details — airline, route, distance, and a scheduled date/time — the API derives every model feature internally. The live-traffic panel filters OpenSky's worldwide feed to aircraft inside the LAX–JFK route's bounding box (1,561 of 8,157 tracked globally) — a separate live layer, not part of the delay prediction itself.

---

Table of Contents

• Overview
• Results at a Glance
• Architecture
• Dataset
• Exploratory Data Analysis
• Feature Engineering
• Modeling
• Key Findings
• Model Explainability (SHAP)
• Error Analysis
• Threshold and Business Framing
• Engineering Decisions Worth Noting
• Serving the Model
• Interactive Dashboard
• Project Structure
• Getting Started
• Testing
• Deploying This
• Limitations and Future Work
• Tech Stack
• License

---

📌 Overview

Every year, roughly 1 in 5 US domestic flights arrives more than 15 minutes late. This project builds a full, reproducible pipeline that:

1. Cleans and engineers features from 5.2M+ real flight records (US DOT/BTS 2015 on-time performance data),
2. Trains and compares Logistic Regression → Random Forest → XGBoost, validated with stratified k-fold CV and tuned via grid search,
3. Explains predictions with SHAP, not just feature importances,
4. Ships the winning model behind a FastAPI endpoint that takes raw flight details and derives every feature internally, and
5. Wraps it in a Streamlit dashboard for interactive, human-facing predictions with a live air-traffic overlay.

The goal wasn't just "get a good AUC" — it was to treat this like a production ML problem: guard against label leakage, validate that results generalize, understand where the model fails, and package it as something a downstream service could actually call.

Scope note: this model predicts delay for completed flights only. Cancelled and diverted flights (1.81% of the raw data) were excluded, since "delay" has no meaningful definition for a flight that never completed its scheduled arrival — predicting cancellations is a different problem, out of scope here.

---

📊 Results at a Glance

Model	Accuracy	Precision	Recall	F1	AUC-ROC
Logistic Regression (baseline)	57.0%	24.7%	64.6%	0.357	0.633
Random Forest	61.8%	27.4%	64.8%	0.385	0.680
XGBoost (default)	62.8%	28.1%	65.1%	0.393	0.690
XGBoost (tuned)	63.2%	28.4%	65.5%	0.397	0.696

Best model: XGBoost, max_depth=8, learning_rate=0.1, n_estimators=150 (found via 3-fold GridSearchCV over 12 candidates, CV AUC-ROC 0.693) → held-out test AUC-ROC 0.696.

Cross-validated on 5 stratified folds independent of the final test set: mean AUC-ROC 0.677 ± 0.0008 — tight enough to confirm the single train/test split wasn't a lucky draw.

Being upfront: 0.696 AUC-ROC is solid but not exceptional for this problem — and that's expected. Flight delays are driven heavily by weather and air-traffic-control events that aren't in this dataset. See Error Analysis for exactly where the model breaks down and why.

---

🏗️ Architecture

flowchart LR
    A[("Raw DOT/BTS CSV\n5.8M flights")] --> B["Cleaning\nnotebooks 01-02"]
    B --> C["EDA\nnotebook 03"]
    B --> D["Feature Engineering\nnotebook 04"]
    D --> E["Model Training\nnotebooks 05-09\nLogReg -> RF -> XGBoost -> Tuned"]
    E --> F["SHAP + Error Analysis\nnotebooks 10-11"]
    E --> G[("Production Pipeline\nnotebook 12\nColumnTransformer + XGBoost")]
    G --> H["FastAPI\n/predict, /health"]
    H --> I["Streamlit Dashboard"]
    H -.dockerized.-> J[("Docker container")]

Loading

The production pipeline is a single sklearn.pipeline.Pipeline (ColumnTransformer + XGBClassifier) serialized with joblib — one artifact, no separate preprocessing script to keep in sync with the API.

---

🗂️ Dataset

• Source: US DOT/BTS 2015 Flight Delay dataset (Kaggle)
• Raw size: 5,819,079 flights × 31 columns
• Target: IS_DELAY — 1 if ARRIVAL_DELAY > 15 minutes (the DOT's own official on-time performance definition), else 0
• Class balance: 82.1% on-time / 17.9% delayed (~4.4:1 imbalance) — handled via class_weight='balanced' (Logistic Regression / RF) and scale_pos_weight (XGBoost), not resampling
• Cleaning removed:
  • 105,071 rows (1.81%) — cancelled/diverted flights, where ARRIVAL_DELAY is undefined
  • 482,878 rows (8.45%) — legacy numeric airport ID codes, a known 2015 DOT encoding inconsistency with no reliable IATA lookup
  • Final training set: 5,231,130 flights.

Raw and processed CSVs are gitignored (the raw file is ~1GB). To reproduce: download the dataset from Kaggle and place it at data/raw/flights.csv — see Getting Started.

---

🔍 Exploratory Data Analysis

Cascading delay effect: delay rate climbs from ~7% at 5 AM to a ~26% peak around 8–9 PM. Aircraft fly multiple legs per day, so an early delay compounds through the schedule.	Airline matters a lot: Spirit (NK) delays 29.6% of flights vs. Hawaiian (HA) at 10.7% — nearly 3x apart. Budget/ULCC carriers show markedly higher delay rates than legacy/regional carriers.
Weekly pattern: Monday (19.8%) and Thursday (19.9%) run highest; Saturday is a clear outlier at 15.7%, consistent with lighter weekend commercial traffic.	Distance ≠ strong linear signal: most flights are short/medium-haul (0–1,500 mi) with a much wider delay spread; long-haul (3,000+ mi) flights cluster near on-time, likely due to larger built-in schedule buffers.

Month is a stronger predictor than expected: February and June both peak (~22–23%), likely winter weather and summer travel volume respectively, while September dips to ~12.5%. Random Forest ranked MONTH as its #2 feature (12.6% importance) — see Model Explainability.

---

🛠️ Feature Engineering

Feature	Why
SCHED_DEP_HOUR, SCHED_ARR_HOUR	Derived server-side from the caller's scheduled departure/arrival datetime — the API takes raw timestamps, not pre-computed hour integers.
DEP_HOUR_SIN/COS, ARR_HOUR_SIN/COS	Cyclical encoding so 23:00 and 00:00 are recognized as 1 hour apart, not 23. Verified against math.sin/cos manually before trusting the pipeline.
DAY_OF_WEEK, IS_WEEKEND	Derived from the scheduled departure date; directly encodes the Saturday-effect found in EDA.
ROUTE_POPULARITY	Historical flight count per ORIGIN_DESTINATION pair, looked up from src/route_popularity.json (built via src/build_route_popularity.py), falling back to the historical mean for unseen routes.
AIRLINE_DELAY_RATE (baseline/RF/XGBoost notebooks only)	Historical per-airline delay rate, computed train-fold-only and mapped onto test to avoid label leakage — see Engineering Decisions.

The final production pipeline drops the custom AIRLINE_DELAY_RATE feature in favor of a plain OrdinalEncoder on AIRLINE — see why below.

---

🤖 Modeling

1. Baseline — Logistic Regression (class_weight='balanced', scaled features) → AUC-ROC 0.633.
2. Random Forest (n_estimators=50, max_depth=15, class_weight='balanced') → AUC-ROC 0.680, validated with 5-fold stratified CV (0.677 ± 0.0008).
3. XGBoost (default) (scale_pos_weight for imbalance) → AUC-ROC 0.690.
4. XGBoost (tuned) via GridSearchCV over max_depth ∈ {4,6,8}, learning_rate ∈ {0.05, 0.1}, n_estimators ∈ {100,150} (36 fits, 3-fold) → AUC-ROC 0.696, chosen as the production model.
5. Consolidated into a single sklearn.Pipeline (ColumnTransformer + XGBClassifier) for one-artifact deployment.

Notebooks 05–09 walk through each stage with metrics and reasoning inline.

---

🔑 Key Findings

• Airline is the strongest signal. Budget carriers delay far more often than legacy carriers — consistent with tighter turnaround schedules and less operational buffer.
• Cascading delay is real and visible in the data. An early delay compounds through the day as aircraft fly multiple legs.
• Month is a stronger predictor than expected, capturing both winter weather and summer travel-volume effects a simpler model would miss.
• The model's blind spot is "surprise" delays — false negatives cluster on historically-reliable airlines having an unexpected bad day, exactly where AIRLINE_DELAY_RATE gives the least signal. This is the clearest direction for future improvement (see Limitations).

---

🧠 Model Explainability (SHAP)

• AIRLINE_DELAY_RATE dominates: high historical delay rate (red) consistently pushes predictions toward "delayed" — the model learned an intuitive, business-sensible pattern rather than noise.
• SCHED_DEP_HOUR: late-day departures (red) push toward delay, confirming the cascading-delay pattern from EDA.
• MONTH: red and blue points are spread across both sides — the effect is non-linear (February and June both show peaks, for different underlying reasons), which SHAP captures but a linear model couldn't.
• IS_WEEKEND: confirmed as the weakest feature, consistent across both Random Forest and XGBoost importance rankings.

Random Forest and XGBoost disagreed on top feature rank (RF favored AIRLINE_DELAY_RATE, XGBoost favored SCHED_DEP_HOUR) — explained by how the two algorithms attribute importance (RF averages over independent trees; XGBoost's sequential boosting concentrates importance on whichever feature reduces error fastest early on, especially with correlated features like the four cyclical time features).

---

🔬 Error Analysis

On the held-out test set: 66,729 false negatives (missed delays) vs. 318,222 false positives (false alarms) — expected given the precision/recall tradeoff of a recall-leaning threshold on imbalanced data.

• Average predicted probability: false negatives sat at 0.393 vs. 0.636 for true positives — most misses were borderline, not confident wrong calls.
• False negatives cluster on historically-reliable airlines (AIRLINE_DELAY_RATE 0.179 for FN vs. 0.197 for TP) — i.e. the model struggles most on unexpected delays where its strongest feature gives no signal. This points directly at the biggest lever for improvement: real-time signals (weather, ATC ground stops) that this 2015 static dataset doesn't contain.

---

⚖️ Threshold and Business Framing

False negatives (missed delays) averaged a 0.393 predicted probability, versus 0.636 for true positives — most missed delays were borderline calls near the 0.5 decision boundary, not confident model failures.

In a real deployment, the right threshold depends on the cost of each error type:

• Lowering the threshold (e.g., to 0.4) catches more true delays at the cost of more false alarms — worth it if under-predicting delays (a passenger caught off guard) is costlier than a false alarm.
• Raising the threshold reduces false alarms but misses more real delays — better if the alerting channel is expensive or attention-limited (e.g., paid SMS alerts).

This model ships with the standard 0.5 threshold; delay_probability is returned uncapped so a deployment can choose its own operating point.

---

💡 Engineering Decisions Worth Noting

1. Leakage, caught twice. During feature engineering, AIRLINE_DELAY_RATE was first computed on the entire dataset — which leaks test-set delay information into training. It was dropped once identified, then reintroduced correctly in the modeling notebooks: computed on the train fold only, then mapped onto both train and test.

2. The "obvious" feature turned out to be redundant — and that's fine. The final production pipeline swaps the custom, leakage-prone AIRLINE_DELAY_RATE for a plain OrdinalEncoder on AIRLINE, and gets virtually identical AUC-ROC (0.696 vs 0.696). XGBoost's tree splits were already implicitly learning the airline-delay relationship on their own. Keeping the simpler encoding removes a leakage risk vector at zero accuracy cost — a deliberate simplicity-over-cleverness call for production.

3. Single train/test splits aren't trusted on their own. Every headline number here is backed by 5-fold stratified cross-validation (σ = 0.0008) specifically to rule out "got a lucky split."

4. The API takes raw input, not pre-computed features. Earlier versions required the caller to supply hour sin/cos, day-of-week, and route popularity directly — which meant every consumer (Streamlit, tests, anyone else) had to duplicate the model's internal feature logic. api/main.py now takes a plain airline/route/datetime/distance and derives everything server-side via build_features(), so there's exactly one place that knows how to go from "a flight" to "a model input."

---

🚀 Serving the Model

FastAPI app (api/main.py) wraps the pipeline with pydantic request validation — airline and airport codes are checked against src/valid_categories.json (extracted from the fitted pipeline), so an unrecognized code returns a clear error instead of a silent bad prediction.

GET /health
# → {"status": "ok"}

POST /predict
Content-Type: application/json

{
  "AIRLINE": "AA",
  "ORIGIN_AIRPORT": "LAX",
  "DESTINATION_AIRPORT": "JFK",
  "SCHEDULED_DEPARTURE": "2026-12-15T18:00:00",
  "SCHEDULED_ARRIVAL": "2026-12-15T23:00:00",
  "DISTANCE": 2475
}

# → {"is_delayed": true, "delay_probability": 0.5921}

Invalid input (e.g. an unknown airline code, or a malformed datetime) returns a 422 with a clear validation message.

Containerized: Dockerfile builds a slim python:3.12-slim image, installs pinned dependencies, and serves via uvicorn on port 8000.

docker build -t flight-delay-api .
docker run -p 8000:8000 flight-delay-api

---

🖥️ Interactive Dashboard

streamlit_app.py provides a point-and-click UI over the API: pick an airline, route, distance, and a scheduled date/time, get a delay-risk verdict with the underlying probability. API_URL is read from an environment variable (defaulting to localhost), so the dashboard keeps working once the API and dashboard are deployed as two separate services.

It also pulls live worldwide ADS-B aircraft positions from the OpenSky Network API, filtered to a bounding box around the selected route, and plots them on a map — a live-tracking layer that ties the historical model back to what's flying right now. This is a visual extra only: OpenSky provides no schedule or delay data, so it plays no role in the prediction itself. See the screenshot at the top of this README.

uvicorn api.main:app --reload      # terminal 1
streamlit run streamlit_app.py     # terminal 2

---

📁 Project Structure

flight-delay-prediction/
├── api/
│   └── main.py                     # FastAPI app: /health, /predict
├── docs/
│   └── screenshot.png              # dashboard screenshot for this README
├── notebooks/
│   ├── 01_data_loading.ipynb
│   ├── 02_cleaning.ipynb
│   ├── 03_eda.ipynb
│   ├── 04_feature_engineering.ipynb
│   ├── 05_baseline_model.ipynb
│   ├── 06_random_forest.ipynb
│   ├── 07_xgboost.ipynb
│   ├── 08_cross_validation.ipynb
│   ├── 09_hyperparameter_tuning.ipynb
│   ├── 10_shap_explainability.ipynb
│   ├── 11_model_comparison.ipynb
│   └── 12_pipeline.ipynb           # final production pipeline
├── reports/figures/                # saved EDA & model plots
├── src/
│   ├── production_pipeline.pkl     # serialized sklearn Pipeline (XGBoost)
│   ├── valid_categories.json       # valid airline/airport codes, from the fitted pipeline
│   ├── build_route_popularity.py   # regenerates src/route_popularity.json locally
│   └── airport_coordinates.py      # lat/lon lookup for the live-traffic map panel
├── tests/
│   └── test_api.py                 # 8 tests: health check + validation cases
├── streamlit_app.py
├── Dockerfile
├── conftest.py
├── requirements.txt                # full dev/notebook environment
├── requirements-api.txt            # lean -- what Docker/Render actually installs
└── requirements-streamlit.txt      # lean -- what Streamlit Cloud actually installs

(.github/workflows/tests.yml runs pytest on every push — see Testing.)

---

⚡ Getting Started

# 1. Clone
git clone https://github.com/Prashant-4527/flight-delay-prediction.git
cd flight-delay-prediction

# 2. Set up environment
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt

# 3a. Just run the pre-trained model
uvicorn api.main:app --reload
# → API live at http://127.0.0.1:8000/docs

streamlit run streamlit_app.py
# → Dashboard live at http://localhost:8501

# 3b. Reproduce the pipeline from scratch
#     Download flights.csv (Kaggle: US DOT 2015 flight delays) into data/raw/
#     Then run notebooks 01 -> 12 in order.
#     Optionally run `python src/build_route_popularity.py` afterwards to
#     regenerate the exact per-route popularity lookup (the API falls back
#     to the historical mean for any route without it).

# 4. Run with Docker
docker build -t flight-delay-api .
docker run -p 8000:8000 flight-delay-api

---

✅ Testing

pytest tests/ -v

tests/test_api.py::test_health_check                        PASSED
tests/test_api.py::test_predict_valid_input                  PASSED
tests/test_api.py::test_predict_invalid_airline               PASSED
tests/test_api.py::test_predict_invalid_airport               PASSED
tests/test_api.py::test_predict_missing_field                 PASSED
tests/test_api.py::test_predict_negative_distance             PASSED
tests/test_api.py::test_predict_malformed_datetime             PASSED
tests/test_api.py::test_predict_lowercase_codes_normalized     PASSED

======================== 8 passed in ~2s ========================

Covers the health check, a valid end-to-end prediction, unknown airline/airport codes, a missing field, negative distance, a malformed datetime, and case-insensitive code normalization.

---

🌐 Deploying This (Two Free Services)

The API and dashboard are deployed separately, connected via API_URL. Two lean, service-specific requirement files keep each deployment fast and small — neither service needs the full ML/notebook stack in requirements.txt:

• requirements-api.txt — what the Dockerfile actually installs (fastapi, pandas, scikit-learn, xgboost, joblib — no matplotlib/jupyter/shap at runtime)
• requirements-streamlit.txt — even leaner (streamlit, requests, pandas only — the dashboard has zero ML dependencies, it just calls the API)

1. Backend (FastAPI) → Render:

1. New → Web Service → connect this GitHub repo
2. Environment: Docker (Render detects the Dockerfile automatically)
3. Deploy — Render assigns its own PORT; the Dockerfile's CMD already reads $PORT at runtime, so no config needed there
4. Copy the resulting URL (e.g. https://flight-delay-api.onrender.com)

2. Frontend (Streamlit) → Streamlit Community Cloud:

1. New app → connect this repo → main file streamlit_app.py
2. Advanced settings → set the requirements file to requirements-streamlit.txt
3. Advanced settings → Secrets → add API_URL = "https://flight-delay-api.onrender.com" (the Render URL from step 1)
4. Deploy — streamlit_app.py already reads API_URL from the environment, so no code changes needed

Free-tier caveat: Render's free web services spin down after ~15 minutes of inactivity — the first prediction after idle time takes 30-60s to wake the container back up. This is a hosting-tier limitation, not a bug.

---

⚠️ Limitations and Future Work

• Single-year dataset (2015). Airline operations, fleet composition, and route networks have all changed since; a production version would need continuous retraining on recent data.
• No weather or air-traffic-control data. The clearest lever identified in error analysis — a same-day weather feature or an "is the inbound aircraft already late" cascading-delay feature would likely move the AUC meaningfully, since these are exactly the "surprise" cases the current model misses.
• Route popularity is a static lookup, not live. In a real production system this would be served from a feature store updated continuously, rather than a JSON file regenerated by hand.
• No airport-level features (e.g., known congestion hubs, runway count) — only airline and route-level signals are used.

---

🧰 Tech Stack

Layer	Tools
Data & ML	pandas, numpy, scikit-learn, XGBoost, SHAP
API	FastAPI, Pydantic, Uvicorn
Dashboard	Streamlit
Testing	pytest, httpx (FastAPI TestClient)
Packaging	Docker
Notebooks	Jupyter

Full pinned versions in requirements.txt.

---

📄 License

MIT — see LICENSE.

🙋 Author

Prashant Sulaniya — GitHub @Prashant-4527 · LinkedIn