मार्गदृष्टि · MargaDrishti

A unified spatio-temporal ML system that predicts where and when Bengaluru loses road capacity — to illegal parking and to live events — and ranks where scarce enforcement should go.

Live Dashboard · Models on HF Hub · Implementation Document · Report Bug · Request Feature

machine-learning · spatiotemporal-forecasting · urban-mobility · geospatial · h3 · xgboost · graph-neural-networks · bayesian-statistics · fastapi · react · bengaluru

Project status — all 12 phases complete. The pipeline runs end to end from raw CSV through a built feature store (2,399,088 rows × 51 columns) to a trained leaderboard of 17 model configurations across all 11 TRD families across seven targets in both modules, plus spatial statistics, a constructed impact index, a fused deployment planner, a published enforcement-bias audit, a static dashboard and generated model cards, with 280+ tests and 14 temporal-leakage guards. Deep models run on Apple Metal (MPS). The Results below are real, from a single held-out test period.

Overview

Problem. Bengaluru loses road capacity two ways. Chronically, illegal on-street parking near commercial areas, metro stations, and markets narrows carriageways — but enforcement is patrol-based and reactive, with no heatmap of violations against congestion impact and no way to rank enforcement zones. Acutely, planned and unplanned events — rallies, festivals, breakdowns, water-logging, construction — cause localised breakdowns whose impact is never quantified in advance, so manpower deployment is experience-driven with no post-event learning loop.

Solution. These are not two problems. They are the same question about two causes: at a given place and hour, how much road capacity is about to be lost, and is that loss worth an officer? MargaDrishti models both on one shared substrate — H3 hexagons × hourly bins — with two prediction modules and a fusion layer that emits a single ranked deployment plan.

Why the fusion is legitimate. It was verified, not assumed. The two source datasets share 54 of 54 police stations exactly, over time windows that agree to 13 minutes across a five-month export. They are two extracts from one platform.

Data. 298,450 parking-violation records and 8,173 traffic-event records from Bengaluru Traffic Police's ASTraM platform, covering 2023-11-09 → 2024-04-08.

Key Findings from the Data

These come from analysis of the raw files and are reproducible via the notebooks.

#	Finding	Why it matters
F1	Timestamps are mislabelled. Both files tag times `+00` (UTC), but the true local clock is the raw value +12h ± 1h. Reading them as-is puts more incidents at 03:00 than at noon (events night/midday ratio 5.71); the IST reading is barely better (0.90); the corrected clock gives 0.017 on events and 0.003 on violations. Under it, events go bimodal at 09:00 and 17:00 (rush-hour breakdowns) and enforcement runs 07:00–20:00 with a visible lunch dip. The exact hour is not resolvable — the two datasets' optima sit ~1.25 h apart.	Two independent files agreeing the shift is ~half a day is strong evidence of anonymisation-time shifting; trusting the `+00` label destroys every hour-of-day feature. The residual ±1 h does not affect model performance — a constant shift only relabels hour categories — but it does constrain joins against real-world-time data like weather.
F2	100% of the 298,450 violation records are parking violations (10 distinct parking offences; 17 secondary offences only ever co-occur).	Reframes the whole module. The task is not "find parking violations" — it is intensity forecasting, severity weighting, and impact ranking.
F3	`validation_status` is a real, unused label on 165,154 adjudicated records — 115,400 approved (69.9%), 49,754 rejected (30.1%). A further 8,042 sit in non-terminal states (`created1`, `processing`, `duplicate`) and are excluded.	A genuinely supervised problem hiding in the data and absent from the brief: predict whether a field capture survives officer review. ~29% rejection is wasted field time.
F4	Only 2,720 of 8,173 events (33.3%) have a usable clearance duration, and it is extremely right-skewed (median 52 min, mean 525 min, P95 3,916 min).	Forces `log1p` targets, quantile regression for P50/P90, MAE over RMSE, and confidence intervals on every metric.
F5	Neither dataset contains any traffic-flow measurement. No speeds, volumes, occupancy, or delay.	"Quantify congestion impact" therefore cannot be supervised regression. It is built as an explicitly-labelled constructed index with four independent external validation tests — never presented as a measured quantity.
F6	`corr(log patrol_hours, log total_captures) = 0.96` across all 2,534 res-9 cells. About 93% of the variance in per-cell violation counts is explained by how much patrolling happened there — not by how much illegal parking happened there. On the 662 active cells the models train on it is 0.89 (79%), lower only because restricting to busy cells truncates the range.	Enforcement bias isn't theoretical, it's the dominant structure in the data. A naive model would largely reconstruct the existing patrol roster, and deploying it as a recommender would close the feedback loop. Hence exposure as both a feature and a Poisson offset.
F7	Parking and events have different geographies. Only 43.6% of events fall in parking-active cells; the rest cluster on arterial corridors (Bellary Road, Mysore Road, ORR).	People don't illegally park on the Outer Ring Road, but they do break down there. The fusion layer needs a union geography of 1,802 cells, not the 662 parking-active ones, or it goes blind to corridor incidents.
F8	The rainfall validation check's premise is false here. Water-logging is ~5× less likely in rainy hours (lift 0.16), flat across every offset from −24h to +24h. 349 of 458 such events fall in March–April 2024 — Bengaluru's dry season, during a documented water crisis when 6,900 of 13,900 borewells ran dry.	Water on the road then was tanker spillage and pipe leaks, not rain. The flat curve is the tell: a real clock misalignment peaks at some lag. The pre-committed "drop weather" rule did not fire — a failed test with an invalid premise is evidence of nothing.
F9	The holiday alignment instrument is underpowered. Only 6 public holidays fall in the window; effect is 0.114 at zero shift vs 0.175 at −3 days, on 5–6 observations.	Neither external anchor resolves the clock, so the +12h evidence stays internal (diurnal shape) — strong but self-referential. The question moves downstream to a real-vs-shifted-weather ablation.
F10	The A2 goal was unreachable by construction. PR-AUC ≥ 0.45 assumed ~10% prevalence, set before density was measured. Only 2.94% of cell-hours are non-zero, so the label lands at 0.291%.	The best achieved is 0.1365 (random forest) — a 46.9× lift over base rate with ROC-AUC 0.938, not a 3× shortfall. Goals set before measuring the data can be wrong in ways the model can't fix.
F11	The review process changed regime mid-window. February 2024 rejection jumps to 0.790 from ~0.28, while reviewed volume collapses ~6×.	The chronological split trains on one regime and validates inside the anomaly, so every A3 model is miscalibrated (Brier worse than baseline). A3 is reported as not-yet-answerable, not as a weak result.
F12	Enrichment is redundant, not uninformative. OSM/weather/calendar give −0.09% with full features but +2.05% once history and exposure are stripped — positive for all four families.	The signal is real but entirely subsumed by enforcement exposure (F6). These features are redundant with a variable that shouldn't be trusted causally — a more uncomfortable finding than a plain null.
F13	`priority` is an operational rule, not a latent quantity. Corridor status predicts High priority at 99.84% (13 exceptions in 8,173). Every model family returned PR-AUC 0.9999.	Seven families agreeing that closely is the signature of a rule, not a hard problem. B3 is reported as a recovered rule — presenting 0.9999 as modelling performance would be misleading.
F14	*The first Gi run reported zero hotspots — an artifact. With 999 permutations the p-floor is 1e−3, but the BH threshold for the smallest of 662 tests is 7.6e−5, so no cell could pass regardless of signal**.	Fixed by computing the required permutation count (26,480) and refusing to run below it. Two further defects fixed: an undefined Gi* self-weight, and esda's one-sided default inflating hotspot counts.
F15	CIS's absolute magnitude is a floor, not a measurement. The worst cell accumulates 12.2 carriageway-seconds of obstruction per day.	It counts ticketed violations, and enforcement detects a small unknown fraction of actual illegal parking. Valid for ranking cells; never quotable as "this junction loses X% of capacity". Choosing physical units exposed this — a 0–100 score would have hidden it.
F16	CIS passes its construction checks and fails its corroboration checks (2/4). Sensitivity τ=0.852 and the width ablation pass; congestion (n=1, p=0.73) and closure (p=0.085) co-occurrence fail.	Internally robust, not externally validated — a materially weaker claim, stated that way everywhere. The congestion FAIL is underpowered, not disproven; the two aren't collapsed.
F17	The system's own fairness audit returns HIGH concern. 0 of 8 recommendations reach an under-observed cell though 25.8% are flagged; station Gini 0.796 vs 0.587.	Not a bug to fix before shipping — the honest consequence of F6. It constrains what the system may claim: predicted enforcement demand, never where illegal parking occurs.
F19	All three late-added families lost to the GBDTs. Stacking 0.5071 (does not beat its own best base), Bayesian 0.6207, GraphSAGE 0.7767 — against xgboost's 0.4999.	Stacking won on a 100-cell subsample (0.4612) and lost on the full panel: an over-fitted meta-learner over correlated bases. GraphSAGE suggests the k-ring columns already capture the spatial signal. The Bayesian model is 12th but is the only one returning posterior intervals.
F18	ONNX export verification caught a silent conversion bug. onnxmltools misroutes ~1% of LightGBM rows at split boundaries — median divergence 4.3e−07 (float32 noise) but 5 of 500 rows off by up to 0.09. skl2onnx converts RandomForest exactly (1.0e−06).	A silently wrong conversion would have the browser returning confident nonsense. The browser ships RandomForest — defensible only because the leaderboard already showed the two statistically indistinguishable (0.5034 vs 0.4990, inside ±0.13 sd).

Cause is also strongly predictive, and the two event targets rank causes very differently — which is exactly why they are modelled separately:

Event cause	n	Road-closure rate	Median clearance
`vip_movement`	20	80.0%	fast
`public_event`	84	46.4%	—
`tree_fall`	284	39.4%	218 min
`construction`	480	26.5%	428 min
`pot_holes`	537	2.4%	1,488 min
`vehicle_breakdown`	4,896	4.3%	41 min

Key Features

Feature	Description	Module
Parking hotspot forecasting	Violation intensity per H3 cell per hour, as Poisson/Tweedie count regression	A
Statistical hotspot discovery	ST-DBSCAN, HDBSCAN, and Getis-Ord Gi* with FDR correction — significant clusters, not eyeballed ones	A
Congestion Impact Score (CIS)	Constructed index in fraction of carriageway-hours obstructed, normalised by OSM road width, with 4-way external validation	A
Enforcement triage	Calibrated rejection risk for field captures, so review effort goes to likely-valid cases	A
Clearance-time prediction	P50 and P90 event duration via quantile regression — "usually 50 min, plan for 22 h"	B
Road-closure prediction	Calibrated probability an event will require closure	B
Multilingual cause classification	Event cause from free-text `description` containing mixed Kannada and English	B
What-if event simulator	Forecast a planned event's impact and get a manpower/barricading plan before it happens	B
Road Capacity Risk Index	Fuses chronic parking pressure and acute event burden into one map	Fusion
Budget-constrained allocation	"I have 8 officers" → top-8 cells with a spatial-diversity constraint and a rule trace	Fusion
Exposure-bias audit	Explicitly separates "no violations" from "no patrolling" — surfaced in the UI, not buried	Cross
Full model leaderboard	10 families × 7 targets — 18 learned models and 5 baselines, 66 fits, all retained with model cards including the losers	Cross
Per-model metric cards	All 44 metrics the harness computes, each with what it measures, which direction is good, and where it misleads	Cross
Pan/zoom map, 3 basemaps	Streets, satellite imagery, or minimal — hand-built Web Mercator, no mapping library	Cross
In-app field guide	~3,500 words behind a `?` on every screen: what the panel shows, why it is built that way, and what the number does not mean	Cross

Tech Stack

Layer	Technology
Language	Python 3.13, TypeScript
Data	pandas 2.3, PyArrow, Parquet, Pandera (schema contracts)
Classical ML	scikit-learn 1.7 — Ridge, Poisson/Tweedie GLM, Logistic, RandomForest, ExtraTrees, HistGradientBoosting
Gradient boosting	XGBoost 3.2, LightGBM 4.6, CatBoost
Deep learning	PyTorch — LSTM/TCN sequence models, GraphSAGE GNN over the junction network
Bayesian	PyMC — hierarchical spatiotemporal model with partial pooling across 54 stations
Geospatial	H3 (res 8/9/10), OSMnx / Overpass, libpysal + esda
Spatial statistics	Getis-Ord Gi*, Moran's I, ST-DBSCAN, HDBSCAN
NLP	sentence-transformers (multilingual, for Kannada + English)
Explainability	SHAP
Experiment tracking	MLflow (local)
API	FastAPI, Uvicorn, Pydantic, OpenAPI
Frontend	React 19, Vite, TypeScript, `h3-js` — map, charts and pan/zoom hand-built (see Phases 10–11)
Basemaps	OpenStreetMap (streets), Esri World Imagery (satellite + labels), Esri Light Gray Canvas (minimal) — all keyless
Quality	ruff, mypy, pytest, GitHub Actions

System Architecture

flowchart TB
    subgraph SRC["Raw Sources"]
        V["violations_raw.csv<br/>298,450 rows · 104.5 MB"]
        E["events_raw.csv<br/>8,173 rows · 4.3 MB"]
    end

    subgraph ING["Ingestion"]
        SCH["Pandera schema validation"]
        TS["Timestamp correction +12h<br/>(Finding F1)"]
        CLN["Clean · drop null cols<br/>clamp impossible dates"]
    end

    subgraph SUB["Shared Substrate"]
        H3["H3 grid res 8/9/10<br/>× hourly bins"]
        EXP["Enforcement exposure<br/>device_id · officer_id"]
        ENR["Enrichment<br/>OSM · Open-Meteo · Calendar"]
        FS[("feature_store.parquet")]
    end

    subgraph MA["Module A — Parking (chronic)"]
        A1["A1 intensity<br/>count regression"]
        A2["A2 hotspot<br/>binary"]
        A3["A3 rejection triage<br/>binary · n=165,154"]
        A4["A4 ST-DBSCAN · Gi*"]
        A5["A5 Congestion Impact Score<br/>constructed index"]
    end

    subgraph MB["Module B — Events (acute)"]
        B1["B1 duration<br/>regression + quantile · n=2,720"]
        B2["B2 road closure<br/>binary · 8.3% pos"]
        B3["B3 priority<br/>binary"]
        B4["B4 cause<br/>multilingual NLP"]
    end

    ZOO["Model Harness · 11 families<br/>unified temporal CV → MLflow"]
    FUS["Fusion<br/>Road Capacity Risk Index"]
    ALO["Budget-constrained allocation"]
    API["FastAPI · REST + OpenAPI + SHAP"]
    UI["React Dashboard · pan/zoom H3 choropleth"]

    V --> SCH
    E --> SCH
    SCH --> TS --> CLN --> H3
    H3 --> EXP --> ENR --> FS
    FS --> MA
    FS --> MB
    MA --> ZOO
    MB --> ZOO
    ZOO --> FUS --> ALO --> API --> UI

In plain language. Two raw CSVs are validated against a schema contract, then pass through the timestamp correction that Finding F1 makes mandatory. Both are projected onto one shared grid of H3 hexagons and hourly time bins — this is what lets two different problems share a model harness. That grid is enriched with road geometry, weather, and calendar data, plus enforcement exposure features that record how much patrolling actually happened in each cell, so the models can tell "clean street" apart from "never visited." Module A predicts the chronic parking load; Module B predicts acute event load. Both run through the same harness of 11 model families under the same temporal cross-validation, so results are directly comparable. The fusion layer combines them into one risk index, converts it into a ranked officer-deployment plan, and serves it through an API to a map-first dashboard.

Application Flow

sequenceDiagram
    participant I as Traffic Inspector
    participant UI as React Dashboard
    participant API as FastAPI
    participant M as Model Registry
    participant F as Fusion Layer

    I->>UI: Select station + hour
    UI->>API: GET /api/v1/hotspots?station=Upparpet&hour=17
    API->>M: Load promoted A1 + A5 models
    M-->>API: Predicted intensity + CIS per H3 cell
    API-->>UI: Cells + "under-observed" exposure flags
    UI-->>I: Choropleth (colour=intensity) + CIS on cell select

    I->>UI: Click a cell
    UI->>API: GET /api/v1/hotspots/{h3}/explain
    API-->>UI: SHAP waterfall
    UI-->>I: Why this cell ranks high

    I->>UI: "I have 8 officers" → Allocate
    UI->>API: POST /api/v1/allocate {station, budget:8}
    API->>F: RCRI + spatial-diversity constraint
    F-->>API: Ranked 8-cell plan + rule trace
    API-->>UI: Deployment plan
    UI-->>I: Exportable patrol assignment

Data & ML Pipeline

1 · Data sources & collection

Source	Records	Window	Access
BTP parking violations (anonymised)	298,450 × 24	2023-11-09 → 2024-04-08	`scripts/download_data.py`
ASTraM traffic events (anonymised)	8,173 × 46	2023-11-09 → 2024-04-08	`scripts/download_data.py`
OpenStreetMap	Bengaluru road + POI	static	OSMnx / Overpass, cached
Open-Meteo Historical	hourly weather	matching window	free API, cached
Calendar	holidays, festivals, fixtures	matching window	`holidays` + curated

Raw CSVs are not version-controlled — the violations file is 104.5 MB, over GitHub's 100 MB per-file hard limit. scripts/download_data.py restores data/raw/ exactly, verifying row counts and reporting SHA-256.

2 · Cleaning

Three columns are 100% null and dropped (description, closed_datetime, action_taken_timestamp). One impossible future date (end_datetime max 2027-11-09) is clamped. Negative validation lags (min −14.7 h) are dropped as clock skew. junction_name = "No Junction" (49.5% of rows) is treated as an explicit mid-block category, not missingness. The full 12-item data-quality register is in the implementation document.

3 · Feature engineering

Temporal — hour, DOW, month, cyclical sin/cos encodings, holiday and festival flags, match days. All derived from the corrected clock.
Spatial — H3 at three resolutions, distance to centre and to metro, target-encoded station, and k-ring neighbour aggregates (k=1, k=2) as the primary spatial-autocorrelation feature.
Lag / history — strictly causal: t−1h, t−24h, t−168h; rolling 7d/28d means; days since last violation.
Built environment — OSM highway class, lanes, width, junction degree, edge betweenness, POI density by category within 200 m.
Weather — precipitation, temperature, humidity, wind; 3h/24h rolling rainfall.
Enforcement exposure — distinct devices and officers active per cell, used both as features and as a Poisson offset.

Leakage controls. Post-outcome columns excluded from forecasting targets; all lags computed with a strict shift on a time-sorted frame and unit-tested against a synthetic case; target encoding fitted inside each CV fold only.

4 · Training approach

Eleven model families, trained for every applicable target through one interface and one CV protocol so results are directly comparable:

Poisson/Tweedie GLM · Ridge/Logistic · RandomForest · ExtraTrees · XGBoost · LightGBM (+quantile) · CatBoost · HistGradientBoosting · LSTM/TCN · GraphSAGE GNN · Bayesian hierarchical · stacked ensemble — against mandatory seasonal-naive and median/majority baselines.

Splits are temporal, never random:

|--------- TRAIN ---------|--- VAL ---|--- TEST ---|
 2023-11-09  →  2024-02-15   → 03-10     → 04-08

Expanding-window CV for selection; the test period is touched once. Because Module B has only 2,720 labelled durations, B1 additionally reports repeated nested CV with confidence intervals. A separate leave-one-station-out evaluation measures spatial generalisation.

5 · Evaluation metrics

Regression — MAE, RMSE, median AE, Poisson deviance, pinball loss (P50/P90). Classification — PR-AUC (primary; prevalences are 8.3% and 10%), ROC-AUC, Brier, calibration curves. Ranking — Precision@10/50, NDCG@50, Kendall τ, because the real operational question is "are the top-N cells the right ones?"

Results & Model Performance

Phase 4 complete — 14 model families trained on the real 2.4M-row panel. All figures below are from a single held-out test period (2024-03-10 → 2024-04-09) touched once, on a chronological split. Full leaderboard: reports/leaderboard.csv.

A1 · Parking intensity — goal met

Ranked by Poisson deviance, the principled loss for a count target. (MAE is shown but is not the ranking metric: on a target that is 97% zero, MAE is minimised by the conditional median — zero — so ranking by it rewards under-prediction.)

Model	Family	Poisson dev ↓	vs baseline	Fit
xgboost	gbdt	0.4999	+43.0%	21 s
hist_gbm_poisson	gbdt	0.5042	+42.5%	18 s
lightgbm	gbdt	0.5055	+42.4%	15 s
stacked	ensemble	0.5071	+42.2%	288 s
extra_trees	trees	0.5086	+42.0%	191 s
catboost	gbdt	0.5162	+41.2%	33 s
hist_gbm	gbdt	0.5189	+40.9%	6 s
poisson_glm	linear	0.5668	+35.4%	8 s
torch_mlp	deep	0.5787	+34.0%	26 s
random_forest	trees	0.5793	+34.0%	696 s
ridge	linear	0.6026	+31.3%	6 s
bayesian_hierarchical	bayesian	0.6207	+29.3%	27 s
torch_lstm	deep	0.6279	+28.4%	154 s
graphsage	gnn	0.7767	+11.5%	38 s
`mean` (baseline)	—	0.8775	—	—
`cell_hour_mean` (baseline)	—	1.3910	−58.5%	—
`seasonal_naive` (baseline)	—	3.5434	−303.8%	—

XGBoost wins at +43.0% over the strongest baseline — beating the ≥25% goal.

All three families added last lost to the gradient-boosted trees, and that is reported rather than buried. Stacking (0.5071) does not beat its own best base: it won on a 100-cell subsample (0.4612) and lost on the full panel, which is what an over-fitted meta-learner over highly-correlated bases looks like. The Bayesian model is 12th, but it is the only family returning posterior intervals — station counts span three orders of magnitude, and it is the one model that says which estimates it trusts. GraphSAGE is the weakest learned model: the k-ring columns the tabular families already read apparently capture the spatial signal, and learning the aggregation added nothing. random_forest remains the clearest negative-value result at 696 s for 10th place.

The result that matters operationally isn't the deviance, though: it's Precision@10 of 0.90–1.00 and NDCG@50 near 0.99. Point predictions are poor (R² ≈ 0.15 — this target is genuinely hard), but the models put the right cells at the top of the list, which is the actual deployment question.

A2 · Hotspot classification — goal was unreachable by construction (F10)

Model	PR-AUC ↑	ROC-AUC	Brier	Prevalence
random_forest	0.1365	0.9380	0.0027	0.291%
catboost	0.1305	0.9341	0.0027	0.291%
xgboost	0.1281	0.9368	0.0027	0.291%
extra_trees	0.1218	0.9362	0.0027	0.291%
stacked	0.1200	0.9363	0.0028	0.291%
logistic	0.1127	0.9204	0.0027	0.291%
hist_gbm	0.0881	0.9128	0.0031	0.291%
lightgbm	0.0357	0.9229	0.0054	0.291%
`prior` (baseline)	0.0029	0.5000	0.0029	0.291%

The design set PR-AUC ≥ 0.45 assuming ~10% prevalence — a number chosen before panel density was measured. Only 2.94% of cell-hours are non-zero at all, so a top-decile-of-active label can never reach 10%; it lands at 0.291%. So 0.1365 is not a 3× shortfall, it's a 46.9× lift over base rate with ROC-AUC 0.938. The goal was wrong, not the model; it is recalibrated to ≥0.10 in models/targets.py.

A3 · Capture rejection — goal missed, and the reason is in the data (F11)

Model	PR-AUC ↑	ROC-AUC	Brier	Prevalence
xgboost	0.3088	0.5750	0.712	24.6%
catboost	0.2987	0.5630	0.682	24.6%
`prior` (baseline)	0.2459	0.5000	0.188	24.6%

ROC-AUC 0.575 against a ≥0.75 goal, and every model's Brier score is worse than the baseline's — a signal of miscalibration, not just weak discrimination.

The cause is a documented regime shift in the review process:

Month	Reviewed	Rejection rate
2023-11	39,063	0.278
2023-12	57,536	0.288
2024-01	50,949	0.266
2024-02	8,242	0.790
2024-03	9,364	0.248

Reviewed volume collapses ~6× and the rejection rate triples in February. The chronological split trains on the stable ~28% regime, validates almost entirely inside the February anomaly, and tests back in the ~25% regime. Every family behaves identically, so this is a property of the label, not the models. A3 is reported as not-yet-answerable rather than as a weak result: the question needs regime-aware splitting or restriction to one regime first.

Phase 7 · Congestion Impact Score — a construction, not validated

CIS answers "quantify impact on traffic flow" the only way the data honestly allows. Neither dataset holds a single flow observation (F5), so it is built, not fitted:

CIS = Σ [ offence weight × vehicle footprint × assumed dwell ] / carriageway area

Output carries physical units — carriageway-hours obstructed per day — rather than a dimensionless score. That choice immediately surfaced F15: the worst cell in Bengaluru accumulates 12.2 carriageway-seconds of obstruction per day. Absurd as a congestion claim, and that is the finding — CIS counts ticketed violations, so it is a floor, not a measurement.

Check	Result
Weight sensitivity (±30%)	✅ τ = 0.852, 92.8% top-50 overlap
Width-normalisation ablation	✅ 76% overlap, τ = 0.727
Congestion co-occurrence	❌ 1 event in top-50, 0.55×, p = 0.73
Closure co-occurrence	❌ 1.41×, p = 0.085

2 of 4 — and which two matters. The construction checks pass; the corroboration checks don't. The index is robust to its own assumptions but cells it ranks highest don't carry more independently-reported congestion. The congestion check is also underpowered past usefulness (n=1), so that FAIL means cannot tell, not disproven. CIS is a coherent, robust construction that has not been externally validated.

Phase 8 · Fusion — RCRI and deployment allocation

Spans the union geography (1,802 cells), not Module A's 662 — per F7, 1,140 fused cells are event-only. A top-8 plan covers 10.43% of total city risk.

The first normalisation was wrong and the output said so: percentile clipping saturated four different cells at an identical 1.57624, manufacturing ties exactly where a deployment plan ranks. Replaced with log1p + min-max; coverage more than doubled (4.36% → 10.43%).

Phases 10–11 · Serving and dashboard

Deployed as a Hugging Face Static Space: precomputed surfaces plus client-side ONNX, no server. The whole surface bundle is 0.14 MB Parquet / 1.75 MB JSON (columnar, rounded, dictionary-encoded strings), and the dashboard is 457 KB JS (143 KB gzipped) — around 40 KB of that is the in-app field guide's prose.

Two heavyweight dependencies were removed, and removing them fixed the bugs: MapLibre's GeoJSON worker never loaded (raster rendered, every vector layer stayed invisible), and Recharts 3 rendered zero paths under React 19. Eleven bars is a <div> with a width, and the map is ~150 lines of Web Mercator: a camera (lat/lng/zoom), OSM raster tiles fetched at round(zoom) and scaled by the fractional remainder so a continuous zoom never requests a level that does not exist, and zoom anchored on the lat/lng under the cursor. Bundle fell 73% against the MapLibre build, with drag-to-pan and wheel-zoom that the library version never delivered.

Every screen carries a field guide behind a ? control — roughly 3,500 words covering what each panel shows, why the modelling choice behind it was made, and what the number does not mean, with a project-wide tab for the shared caveats (the clock correction, the 0.96 exposure correlation, measured vs constructed). It opens on click and closes when the pointer leaves it.

The dashboard opens on an Overview, not on the map. A choropleth read cold says "parking is worst here" when what it shows is "enforcement has been productive here", so the three findings that change how every other number should be read arrive before the first chart does.

The bundle originally exported 4 of the 44 metrics the harness computes, and the Model Lab ranked every target by one of two columns. That dropped the ranking metrics — precision@10 and NDCG@10 are closer to this system's actual job than deviance is, since nobody deploys to 662 cells, they deploy to the top of a list — every fold standard deviation, and three models entirely: both quantile models (scored on pinball loss, and the source of the P50/P90 the Event Console displays) and the B4 text classifier (macro F1 only), whose target rendered as an empty screen. Each target is now ranked on its own primary metric, unrankable rows are listed rather than filtered, and every model has a card where each number carries what it measures and where it misleads. Six tests pin it, because an export silently losing a column is invisible from the screen — which is how it went unnoticed.

ONNX export verifies every model numerically before shipping it — which caught that onnxmltools misroutes ~1% of LightGBM rows at split boundaries (F18). The browser ships RandomForest instead, defensible only because the leaderboard already showed the two statistically indistinguishable.

Phase 9 · Enforcement-bias audit — HIGH concern, on our own system

Test	Statistic	Concern
Feedback loop	ρ = +0.613	MODERATE
Under-observation coverage	0 of 8 recommendations, vs 25.8% flagged	HIGH
Station disparity	Gini 0.796 vs 0.587 (excess +0.209)	HIGH
Counterfactual patrol	30% of top-50 survive equal effort	MODERATE

A quarter of cells are barely patrolled and the recommender sends officers to none of them — the textbook feedback loop, measured rather than asserted away. This isn't a bug to fix before shipping; it's the honest consequence of F6, and it fixes what the system may claim: it ranks predicted enforcement demand conditioned on patrol effort, and does not identify where illegal parking occurs. The audit has no pass/fail field by design — a test asserts that.

Full report: reports/fairness_audit.md.

What the deep models showed

Both PyTorch families ran on Apple Metal (MPS), not CUDA. Both lost to every gradient-boosted tree on the ranking metric — torch_mlp 0.566 and torch_lstm 0.584 against XGBoost's 0.501.

This is reported as a result, not hidden. Two things are worth noting: torch_mlp had the best MAE of any model (0.131), and torch_lstm had the best Kendall τ (0.878) — the sequence model ranks cells best while scoring worst on deviance. The LSTM is the only family that sees temporal order rather than unordered lag columns, and on this data that structural advantage did not pay off against well-tuned GBDTs.

random_forest is the clearest negative-value result: 432 s to finish 8th.

Enrichment ablation — enrichment is redundant, not uninformative

Feature set	Enrichment gain
Full (history + exposure)	−0.09%
Minus history	−0.09%
Minus history and exposure	+2.05%

Stopping at the first row would say OSM, weather, and calendar are useless. The third row says otherwise: with nothing else encoding cell intensity, enrichment helps every family (+0.84% to +3.18%). The signal is real but entirely subsumed by enforcement exposure — F6's r = 0.96 again. That's a more uncomfortable reading than a plain null: these features are redundant with a variable that shouldn't be trusted as a causal driver.

Real weather beat week-shifted weather by 0.34% — inside noise — so this third instrument is inconclusive too. F8 and F9 stand: the +12h correction rests on internal diurnal evidence alone.

Phase 5 · Spatial statistics — no hotspot survives exposure adjustment

Basis	Moran's I	Hot cells (FDR q<0.05)
Raw violation counts	+0.385	6
Per patrol-hour	+0.211	0
		overlap: 0

Six locations are statistically significant hotspots by raw count. Adjust for enforcement effort and every one disappears, with nothing replacing them. The honest reading isn't "there are no hotspots" — exposure adjustment may over-correct, since police plausibly patrol where violations are — but that this data cannot separate a genuine hotspot from a patrol-allocation pattern. The raw-count heatmap the problem statement asks for would substantially be a map of where officers went.

Three methodological defects were found and fixed here — see F14.

HDBSCAN and ST-DBSCAN answer different questions on the same data: HDBSCAN finds persistent places (clusters spanning the full 3,600-hour window), ST-DBSCAN finds transient episodes (~9-hour bursts).

Phase 6 · Module B — event congestion

B1 · clearance duration (n = 2,720; mean over 5 chronological folds ± sd)

Model	Poisson dev ↓
xgboost	0.4990 ± 0.128
random_forest	0.5034 ± 0.134
extra_trees	0.5084 ± 0.122
`mean` (baseline)	0.6904 ± 0.213
torch_mlp	4.4045 ± 3.908

Quantile models supply what a control room actually commits to — LightGBM P50 (pinball 0.566) and P90 (0.234). With a P95 of 3,906 min against a median of 52, an expected value is close to useless operationally. torch_mlp is unusable at this sample size, reported rather than dropped.

B2 · road closure — CatBoost 0.3615 ± 0.071 PR-AUC vs 0.087 prevalence: a ~4× lift on an 8.3% positive class.

B3 · priority — not a prediction problem (F13). Reported as a recovered rule, not a model result.

B4 · cause from free text — macro-F1 0.478 vs 0.044 majority baseline (10.8×), short of the 0.60 goal. The notable result is equity, not accuracy: English 0.483 vs Kannada 0.472 — essentially no language gap. That is exactly what character n-grams were chosen for over word tokenisation, on a field that mixes both scripts inside single entries.

Deployment & Infrastructure

Total cost: ₹0. Every dependency, data source, and hosting tier in this project is free. Full breakdown in the implementation document.

Why static-first

MargaDrishti is a batch system over a frozen 5-month extract — real-time streaming is explicitly out of scope. Nothing needs on-demand server compute:

Hotspot surfaces are precomputed. A typical-week surface (672 active cells × 168 hours-of-week = 112,896 rows) is 1–3 MB as Parquet, and is the more useful artifact anyway — an inspector wants "the Tuesday 17:00 pattern," not one specific historical hour.
The what-if simulator runs client-side. GBDT models export to ONNX and execute in-browser via onnxruntime-web (~2–10 MB, lazy-loaded).
SHAP is precomputed for the cells and scenarios the UI surfaces.

No server, no cold start, no scaling limit, no bill.

Where it runs

Piece	Host	Cost
Model artifacts + cards	HF Hub — `adarshcod30/margadrishti-models`	₹0
Dashboard + surfaces	HF Static Space	₹0
(optional) live SHAP + multilingual NLP API	Google Cloud Run or AWS Lambda always-free	₹0 within quota

On HF Spaces. As of 2026, only the Static SDK is free. Gradio and Docker Spaces run on compute and require PRO ($9/mo). This project therefore targets a Static Space — which, given its batch nature, is the better fit regardless of price.

The FastAPI app is still built and still the contract the frontend codes against — it is the local dev server (make serve), just not required in production. Every endpoint in the API reference resolves against a precomputed artifact or a client-side ONNX session.

Training compute

Also ₹0, because the data is sparse. The full H3 res-9 cross-product is 9.56 M rows (~2.3 GB), but only 0.87% of cell-hours contain a violation. Restricting to the 672 cells with ≥50 violations gives 2.43 M rows ≈ 0.58 GB while retaining 92.7% of all violations — comfortably in laptop RAM, keeping the <90 min full-retrain target realistic.

Concern	Approach
Local development	`make serve` — FastAPI on `:8000`, Vite on `:5173`
Experiment tracking	MLflow local file store; promoted models pushed to HF Hub
CI	GitHub Actions — ruff, mypy, pytest, smoke-train on a sample
Reproducibility	Fixed seeds, pinned deps, config-driven; `make reproduce`
Monitoring	Prediction-vs-actual drift in the Analytics screen; fairness audit re-run each retrain

Project Structure

MargaDrishti/
├── docs/
│   ├── IMPLEMENTATION.md        # PRD · TRD · app flow · UI/UX · risk register
│   ├── MODEL_CARDS/             # one card per trained model, winners and losers
│   └── DATA_DICTIONARY.md
├── config/
│   ├── features.yaml
│   ├── models.yaml              # hyperparameter grids
│   └── impact_weights.yaml      # CIS weights, with sources
├── data/                        # gitignored — restored by the download script
│   ├── raw/ · interim/ · processed/ · external/
├── scripts/
│   └── download_data.py         # restores both raw CSVs, verifies row counts
├── src/margadrishti/
│   ├── data/                    # ingest · validate · clean
│   ├── features/                # grid · temporal · spatial · osm · weather · exposure
│   ├── models/                  # zoo · train · tune · registry · deep · bayesian
│   ├── spatial/                 # clustering · getis_ord · stdbscan
│   ├── impact/                  # cis · validate_cis
│   ├── fusion/                  # rcri · allocate
│   ├── evaluation/              # metrics · backtest · fairness · shap_utils
│   └── api/                     # main · routers · schemas
├── notebooks/
│   ├── 00_data_overview.ipynb
│   ├── 01_timestamp_forensics.ipynb    # reproduces Finding F1
│   ├── 02_eda_parking.ipynb
│   ├── 03_eda_events.ipynb
│   ├── 04_spatial_analysis.ipynb
│   ├── 05_model_comparison.ipynb
│   └── 06_cis_validation.ipynb
├── app/                         # React + Vite + TypeScript dashboard
├── tests/
└── .github/workflows/ci.yml

Getting Started

Prerequisites

Python 3.11+ (developed on 3.13)
Node 18+ (only for the dashboard)
~150 MB free disk for the raw datasets

Installation

git clone https://github.com/adarshcod30/MargaDrishti.git
cd MargaDrishti

python -m venv .venv && source .venv/bin/activate

pip install -e ".[dev]"

Fetch the data

The raw CSVs are not in the repo (the violations file exceeds GitHub's 100 MB limit). Restore them with:

python scripts/download_data.py

This downloads both files into data/raw/, verifies row counts against the expected 298,450 and 8,173, and prints a SHA-256 for each.

Run the pipeline

make features

make train

make evaluate

Run the app

make serve

API on http://localhost:8000 (docs at /docs), dashboard on http://localhost:5173.

Usage / API Reference

Method	Endpoint	Purpose
`GET`	`/health`	Liveness probe
`GET`	`/api/v1/hotspots?station=&hour=&res=`	Predicted parking intensity + CIS per H3 cell
`GET`	`/api/v1/hotspots/{h3}/explain`	SHAP breakdown for one cell
`GET`	`/api/v1/clusters?method=stdbscan\|hdbscan\|gistar`	Clusters / Gi* significance
`POST`	`/api/v1/events/predict`	Duration P50/P90, closure probability, priority
`POST`	`/api/v1/events/simulate`	What-if planned-event scenario
`POST`	`/api/v1/captures/triage`	Rejection risk for a field capture
`GET`	`/api/v1/rcri?station=&hour=`	Fused Road Capacity Risk Index
`POST`	`/api/v1/allocate`	Budget-constrained deployment plan
`GET`	`/api/v1/models`	Full leaderboard
`GET`	`/api/v1/models/{id}/card`	Model card
`GET`	`/api/v1/audit/fairness`	Exposure-bias audit results

Example:

curl -X POST http://localhost:8000/api/v1/events/predict -H "Content-Type: application/json" -d '{"cause":"tree_fall","latitude":12.9255,"longitude":77.6186,"start_local":"2024-03-07T17:00:00","veh_type":null,"corridor":"Non-corridor"}'

Testing

pytest -q --cov=src/margadrishti

Coverage target ≥ 80%. The suite covers schema contracts, the timestamp correction, temporal-leakage guards on every lag feature, H3 assignment, CIS weight sensitivity, and API contract tests. Lint and types:

ruff check . && mypy src/

Roadmap

Phases 0–6 constitute the minimum viable project; 7–12 are incremental and independently shippable.

Ethical Position

This project predicts enforcement demand, not criminality, and never individuals. Three commitments are binding on the build:

Enforcement exposure is modelled, never ignored. A cell with no recorded violations may be clean — or simply never patrolled. Patrol history enters as both a covariate and a Poisson offset, under-observed cells are flagged in the UI, and a fairness audit explicitly tests for the feedback loop where a model just recommends patrolling where patrols already happen.
No person-level targets. vehicle_number, created_by_id, and device_id are used only in aggregate. No model ranks officers or vehicle owners.
Constructed indices are labelled as such. The Congestion Impact Score and RCRI are transparent constructions, not measurements. Because neither dataset contains any traffic-flow data (Finding F5), no claim of measured flow impact is made anywhere in this project.

Contributing

This is a personal research project, but issues and suggestions are welcome. Fork, branch descriptively, keep ruff and mypy clean, and add tests for new behaviour.

License

MIT — see LICENSE.

Source datasets are anonymised public releases from the HackerEarth challenge distribution and remain subject to their original terms.

Contact

Adarsh Dwivedi · GitHub @adarshcod30

_{Built for Bengaluru's roads.}

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
app		app
config		config
data		data
docs		docs
models		models
notebooks		notebooks
reports		reports
scripts		scripts
src/margadrishti		src/margadrishti
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation