Batch lakehouse processing 4.4M+ Chicago 311 service request records (2018–present) for SLA compliance, backlog visibility, and geographic-equity reporting.

• Kimball star schema with an explicitly documented SCD strategy per dimension — Type 0 for immutable geography and dates, Type 1 for department names that can change
• Two fact tables with different grains: an accumulating-snapshot fact for ticket lifecycle, and a separate transaction fact (completed tickets only) for SLA/equity analysis
• Implemented a Write-Audit-Publish pattern via BigQuery audit-table swap, since BigQuery doesn't support native Iceberg branching
• Defined and documented explicit pipeline SLAs (freshness, ingestion completion time, failure recovery) separate from the city's own 311 response-time targets
• Two Shiny dashboards serving distinct stakeholder personas: daily operational triage vs. weekly executive/SLA reporting

Stack:  |  🔗 dezc-capstone-311-chicago-sr

A pipeline and interactive mapping tool modeling US socioeconomic patterns from Census ACS, TIGER/Line, and IRS migration data across 2012–2024.

• Bronze → Silver → Gold medallion on S3 + Athena + Iceberg, ingesting 3 heterogeneous APIs into one coherent storage layer at state/county/census-tract granularity (~84k tracts)
• Orchestrated with Dagster using dynamic partitions + a sensor instead of cron schedules — the pipeline reacts to actual data releases rather than guessing a release date
• Interactive choropleth map rendering 84k census tracts via GPU-accelerated WebGL (lonboard/deck.gl) without shipping raw geometry to the browser
• Data quality enforced with Dagster asset checks at every layer transition plus dbt tests on the mart models

Stack:  |  🔗 population-demographics-de-project

Combines a batch market-data lakehouse with a real-time operational telemetry stream, built entirely within a 4GB memory constraint.

• Batch: dlt ingests EIA market data into S3, cataloged with DuckLake, transformed with dbt-duckdb, served via Motherduck, orchestrated by Kestra
• Streaming: simulated well telemetry flows through Redpanda Serverless → Bytewax (1-minute tumbling windows) → idempotent upserts into Neon serverless Postgres → Grafana
• Implemented a dead-letter-queue pattern so malformed IoT payloads are isolated without interrupting the live pipeline
• Deliberately offloaded the heaviest components (Kafka broker) to managed cloud tiers to protect local compute for the Python transformation layer

Stack:  |  🔗 oil-de-pipeline

A feature-store-centered platform serving three ML models (churn, campaign response, CLV) from one shared, point-in-time-correct pipeline for a simulated B2B SaaS company.

• Built a Dagster asset-based pipeline (Polars transforms) feeding a Feast feature store — S3 offline store, Redis online store for sub-10ms serving lookups
• Designed the pipeline specifically to prevent training/serving skew: the same build_feature_table(as_of_date) function powers both training and serving paths
• Simulated 50k customers with persona-conditioned behavioral state machines (not random data) so churn and engagement signals are genuinely learnable
• Served predictions via FastAPI, with models tracked and versioned through the MLflow Model Registry

Stack:  |  🔗 momentumcrm-marketing-de-project

Real-time monitoring platform for solar and wind assets across Austria, from raw sensor telemetry to operational and executive dashboards.

• Bridged MQTT (HiveMQ Cloud) into Kafka-compatible Redpanda Serverless, with consumer groups, manual offset commits, and at-least-once delivery
• Built a Bytewax dataflow for windowed KPI aggregation (60-minute tumbling + session windows) with on-disk SQLite state recovery
• Modeled a 4-fact dbt star schema on Snowflake (hourly telemetry → daily → monthly rollups) with documented ADRs for every major tooling decision
• Two visualization layers on different cadences: Grafana for sub-minute ops monitoring, Evidence.dev (code-as-dashboard) for business reporting

Stack:  |  🔗 green-energy-iot-de-project

A batch pipeline + decision engine that ingests CSV exports from EHR systems and returns a per-patient readmission-risk decision.

• Built a Bronze → Silver → Gold medallion architecture on S3 + Apache Iceberg (PySpark, Glue, Athena)
• Combined three decision layers: deterministic clinical rules, an XGBoost readmission model, and an LLM-generated plain-English explanation — the rules engine is the authoritative escalation path, the ML score is a soft signal
• Served via a BentoML API with per-item batch error handling, so one malformed patient record doesn't fail an entire cohort scoring run
• Streamlit dashboard giving care coordinators a cohort view plus per-patient drill-down

Stack:  |  🔗 healthcare-csv-decision-engine

Pulls real SEC EDGAR filings for 16 large-cap issuers and serves a research API with vector search over the filing text.

• Ingested 5 filing types (10-K, 10-Q, 8-K, DEF 14A, Form 4) through a pure-Iceberg medallion architecture — S3 + Glue + Athena as a serverless warehouse, no managed platform
• Embedded 24,556 real filing chunks into Qdrant for vector search, using a local zero-cost GGUF embedding model (no per-token API charges, no data egress)
• Built 6 dbt marts on Athena, bucketed by company ID to match the API's query patterns
• Three-layer data quality: Soda row-level checks, dbt column tests, and dbt source-freshness monitoring

Stack:  |  🔗 edgar-supply-graph