nyc-tlc

AWS data engineering portfolio project using NYC taxi data, PySpark, Amazon S3, AWS Glue, Athena, and Terraform.

Heatmap of average pick-ups by location and time of NYC TLC 2024 records

Smart Traffic — reproducible multimodal ITS pipeline (public datasets): Kafka streams → Spark Structured Streaming + Delta Lake, MinIO frames, YOLOv8+ByteTrack + LightGBM→ONNX workers, Streamlit UI, Prometheus/Grafana. Docker-first, CPU-only, Windows-friendly.

Did NYC's congestion toll cut taxi and Uber/Lyft trips? Difference-in-differences, a forecast counterfactual and synthetic control on 1.3B trips, checked against fake toll dates.

End-to-end NYC TLC HVFHV data pipeline analyzing Tens of Millions Uber and Lyft trips with Python, Parquet, DuckDB, GeoPandas, and Streamlit.

Multi-agent RL environment for fairness-aware delivery fleet dispatch — PettingZoo + RLlib, calibrated on real NYC TLC data. Includes human-annotation, automated QA gating, checkpoint fine-tuning, synthetic scenario generation, distributed rollout benchmarking, and a versioned benchmark suite.

End-to-end serverless data engineering pipeline on AWS — ingests NYC TLC HVFHV trip data using Glue, Step Functions, Lambda, S3, and SNS with event-driven orchestration.

End-to-end NYC TLC real-time data engineering pipeline using Kafka, Python, PostgreSQL and Power BI.

Causal estimates of NYC congestion pricing on 1.3B TLC for-hire trip records. A -13% headline fails its fake-start placebo, with the design committed first.

Data pipeline implementing Medallion Architecture with PySpark and MongoDB for NYC TLC Trip Data, including comprehensive data auditing.

Production-grade ELT pipeline: NYC TLC taxi data → Snowflake → dbt → Streamlit + Grafana FinOps | Astronomer (Airflow 3) orchestration

End-to-end data analytics project on NYC taxi data including cleaning, EDA, and insights on trip patterns, fares, and peak demand.

🚕 End-to-end data analytics pipeline and 7-tab Streamlit command center auditing 3.47M NYC TLC taxi trips, congestion surcharge compliance, forensic ghost trip anomalies, OD flow corridors, and policy pricing simulations under NYC's Central Business District Tolling Program.

DuckDB and SQL warehouse over 48.7M NYC TLC trips, with dimensional modeling, tested data quality rules, BI-ready marts, and bilingual documentation.

Validated mobility ingestion with DuckDB/dbt marts, orchestration, quality checks, and an analytics API.

Leakage-free hourly demand forecasting project using NYC TLC High Volume FHV trip data, temporal baselines, and machine learning models.

Reproducible Spark lakehouse for heterogeneous NYC TLC trip records

Airflow 3 + PySpark batch pipeline on 383k real NYC TLC green-taxi trips: resumable ingest, DQ flags, reconciliation gate

Monthly ELT pipeline for NYC Uber/Lyft trips: S3 lake, PySpark, Postgres, dbt star schema, Airflow 3 in Docker, GitHub Actions CI

Explainable NYC taxi anomaly review system with data contracts, robust baselines, evaluations, evidence receipts, and human approval boundaries.