A local-first, open-source data engineering and analytics system for real-time air quality monitoring, anomaly detection, and near-term PM2.5 forecasting.
Current air-quality dashboards fail to distinguish normal seasonal and hourly variation from genuinely unusual pollution behavior, leaving operators unable to identify when an event requires attention. Raw sensor data from public sources contains duplicates, stale values, flatlines, and malformed records, yet existing systems often present this unreliable data directly to end users without meaningful validation or context. Air quality engineers need a trustworthy foundation that ingests, validates, and contextualizes observations so that alerts and forecasts reflect actual risk, not sensor noise.
The Air Quality Intelligence Platform solves this by implementing a rigorous data pipeline that continuously collects observations from OpenAQ and weather data from Open-Meteo, deduplicates and validates every record, computes historical baselines at the location and hour level, detects anomalies using robust statistical methods, identifies persistent pollution events, trains CPU-friendly time-series forecasting models, and exposes the results through an API and interactive dashboardβall on modest hardware (AMD Ryzen 3 / 16 GB RAM) using open-source tools and free public data.
EXTERNAL SOURCES
|
+--β OpenAQ (air quality)
+--β Open-Meteo (weather)
|
v
βββββββββββββββββββββββββββββββββββ
β INGESTION SERVICES β
β (fetch, parse, validate, dedupe)β
ββββββββββββ¬βββββββββββββββββββββββ
|
v
ββββββββββββββββ
β RAW LAYER β
β (Parquet) β
ββββββββ¬ββββββββ
|
βββββββvβββββββββββββββββββββ
β DATA QUALITY & VALIDATION β
β (Structural, semantic, β
β temporal, referential) β
ββββββββ¬βββββββββββ¬βββββββββββ
| |
v v
ββββββββββββ ββββββββββββ
β CLEAN β βQUARANTINEβ
β(Parquet) β β β
ββββββββ¬ββββ ββββββββββββ
|
βββββββvβββββββββββββββββββββββββββ
β TRANSFORMATION LAYER β
β (Polars / dbt) β
ββββββββ¬ββββββββββββ¬ββββββββββββββββ
| |
v v
βββββββββββββββ ββββββββββββ
β ANALYTICAL β β FEATURES β
β MARTS β β (for β
β (hourly/ β β ML) β
β daily) β ββββ¬ββββββββ
βββββββ¬ββββββββ |
| v
| ββββββββββββββββ
| β ML TRAINING β
| β & REGISTRY β
| ββββββββ¬ββββββββ
| |
| v
| ββββββββββββββββββ
| β PREDICTIONS β
| β & INTERVALS β
| ββββββββββ¬ββββββββ
| |
+βββββββ¬βββββββ
v
ββββββββββββββββββββββββββ
β CONTROL PLANE β
β (PostgreSQL metadata, β
β lineage, model registry)
ββββββββββββββββββββββββββ
|
βββββββββββββvβββββββββββββββ
β SERVING LAYER β
β FastAPI (REST) β
β Streamlit (Dashboard) β
ββββββββββββββββββββββββββββββ
- Data Plane: Ingestion, raw storage, deduplication, quality validation and cleaning.
- Intelligence Plane: Analytics, anomaly detection, pollution events, baselines and forecasting.
- Control Plane: PostgreSQL metadata, pipeline state, lineage and model registry.
- Docker & Docker Compose (v20.10+)
- Python 3.12+
- 8 GB RAM, 50 GB disk
- Git
git clone https://github.com/AseemPrasad/Air-Quality-Intelligence.git
cd Air-Quality-Intelligence
cat > .env << 'EOF' DATABASE_URL=postgresql://aq_user:aq_password@postgres:5432/air_quality POSTGRES_USER=aq_user POSTGRES_PASSWORD=aq_password POSTGRES_DB=air_quality API_HOST=0.0.0.0 API_PORT=8000 API_LOG_LEVEL=INFO DATA_DIR=/data PARQUET_PATH=/data/parquet OPENAQ_API_KEY=your-key-here AIRFLOW__WEBSERVER__SECRET_KEY=change-me-in-production EOF
### 3. Launch Services
```bash
docker-compose up -d
sleep 30 # Wait for services to start
# Verify
docker-compose ps
Services:
- PostgreSQL (port 5432): Metadata, control plane, audit log
- API (port 8000): FastAPI with Pydantic validation
- Airflow (port 8080): Orchestration & DAGs
# Initialize database schema
docker-compose exec api python -m src.aq_engine.init_db
# Test API health
curl http://localhost:8000/api/system/health
# View Airflow DAGs
open http://localhost:8080 # admin / admin# Option A: Via Airflow UI (http://localhost:8080)
# Trigger "aq_hourly_ingest" DAG
# Option B: Via CLI
docker-compose exec api python -c "
from src.aq_engine.connectors import OpenAQConnector
from src.aq_engine.storage import ParquetStorage
storage = ParquetStorage('/data/parquet')
connector = OpenAQConnector()
records = connector.fetch()
storage.write(records)
"
# Verify data flow
curl http://localhost:8000/api/locationsResult: Hourly ingestion from OpenAQ & Open-Meteo, 1,200 + 450 records/hour
curl http://localhost:8000/api/locations/kolkata_001/currentResponse:
{
"location_id": "kolkata_001",
"pollutants": [{
"pollutant": "PM2.5",
"value": 65.5,
"anomaly_severity": "HIGH",
"baseline_median": 55.0
}],
"weather": {
"temperature_c": 32.5,
"humidity_pct": 75,
"wind_speed_kmh": 12.0
}
}curl "http://localhost:8000/api/locations/kolkata_001/forecast?horizon=1h,3h,6h"Response:
{
"forecasts": [
{
"horizon_minutes": 60,
"predicted_pm25": 62.0,
"lower_bound": 55.0,
"upper_bound": 70.0,
"confidence": 0.85
}
]
}curl "http://localhost:8000/api/locations/kolkata_001/events?start_date=2026-08-14T00:00:00Z&end_date=2026-08-15T23:59:59Z"curl http://localhost:8000/api/system/healthSee API Specification for full endpoint documentation.
.
βββ src/aq_engine/ # Main package
β βββ __init__.py
β βββ cli.py # Typer CLI
β βββ config.py # Configuration
β βββ connectors/ # Data source connectors
β βββ quality/ # Validation & quality
β βββ analytics/ # Analytical layer
β βββ ml/ # Forecasting & ML
β βββ api/ # FastAPI application
β βββ dashboard/ # Streamlit app
βββ dags/ # Airflow DAGs
βββ dbt/ # dbt models & tests
βββ sql/ # SQL migrations & schemas
βββ tests/ # Test suite
βββ configs/ # Configuration files
βββ data/ # Data storage (local)
βββ docker/ # Docker & Compose configs
βββ docs/ # Documentation & ADRs
βββ pyproject.toml # Project metadata & deps
βββ README.md
| Component | Technology | Purpose |
|---|---|---|
| Language | Python 3.12+ | Core implementation |
| Database | PostgreSQL | Metadata, control plane |
| Analytics | DuckDB | SQL over Parquet |
| Storage | Parquet | Immutable raw/clean/mart data |
| Processing | Polars | Memory-efficient transforms |
| Transforms | dbt Core | Reproducible SQL models |
| Orchestration | Airflow | DAGs, retries, backfills |
| ML | scikit-learn | Time-series forecasting |
| API | FastAPI | REST endpoints |
| Dashboard | Streamlit | Interactive analytics UI |
| Testing | pytest | Unit & integration tests |
- Air Quality: OpenAQ β real-time observations from global monitoring networks
- Weather: Open-Meteo β temperature, humidity, wind, pressure, precipitation
pytest tests/ -v --cov=src/aq_engineruff check src/ tests/
mypy src/aq_engine --strictCode is formatted with black and linted with ruff. All public APIs require type hints and docstrings.
Comprehensive documentation is in docs/:
- 01-system-architecture.md β Three-plane architecture, technology stack, data flow, deployment model
- 02-data-contracts.md β Canonical schemas (air quality, weather, hourly facts, baselines, quality classifications)
- 03-anomaly-detection-logic.md β MAD-based z-score, severity thresholds, fallback logic
- 04-ml-specification.md β 46 features, time-series splits, baselines, 4 ML candidates, 5% promotion criteria
- 05-api-specification.md β 8 REST endpoints with JSON examples, error codes, rate limiting, client examples
- 06-database-schema.md β DDL for 12 PostgreSQL tables, relationships, ER diagram, indexes, partitioning strategy
- 07-deployment-guide.md β Docker Compose quick-start, env vars, K8s deployment, monitoring, troubleshooting
- 08-runbook.md β Manual backfill, model retraining, data recovery, disaster recovery procedures
- 09-architecture-decisions/ β ADRs:
001-postgresql-parquet-choice.mdβ Why dual storage (transactional DB + columnar files)004-anomaly-detection-mad.mdβ Why Median Absolute Deviation (robust to outliers)
To maintain a clean root directory, all phase summaries, implementation reports, and deliverable checklists have been migrated to the project tracking directory:
- M0: Foundation (repo, environment, config, logging)
- M1: Ingestion (connectors, raw storage, idempotency)
- M2: Quality (validation, quarantine, deduplication)
- M3: Analytics (hourly facts, baselines, station health)
- M4: Intelligence (anomaly detection, pollution events)
- M5: ML (models, training, registry, predictions)
- M6: Serving (API, dashboard)
- M7: Orchestration (Airflow, monitoring)
- M8: Hardening (tests, documentation, performance tuning)
MIT
For questions or contributions, open an issue or contact the Air Quality Engineering Team. "feature change"