Data engineer and AI systems builder focused on the intersection of streaming infrastructure and intelligent query interfaces. I design and build production-grade lakehouse platforms β from raw event ingestion through governed star schemas to AI agents that query them in natural language.
My work centers on Apache Iceberg, PySpark Structured Streaming, dbt, and Trino for the data layer, and MCP-based text-to-SQL agents grounded in dbt manifest metadata for the AI layer. I care deeply about correctness guarantees: at-least-once delivery with idempotent deduplication, execution-accuracy evaluation over SQL string matching, and guardrail-enforced read-only agent access.
I approach data platform engineering with a product mindset β every pipeline decision is justified by the analytical workload it enables, and every agent capability is measured against a golden eval dataset.
Open To: Data Engineering Β· AI/ML Engineering Β· Platform Engineering Β· Senior SWE roles at companies building on modern lakehouse or AI-native data stacks.
| Domain | Proficiency | Details |
|---|---|---|
| Text-to-SQL Agents | ββββββββββ Advanced | MCP server, manifest-grounded RAG, retry loop, guardrails |
| Vector Retrieval (RAG) | ββββββββββ Advanced | pgvector, per-model embedding chunks, top-k retrieval |
| LLM Evaluation | ββββββββββ Proficient | Execution accuracy harness, golden datasets, CI regression gate |
| Prompt Engineering | ββββββββββ Proficient | Schema context injection, adversarial guardrail testing |
| Streaming ML Features | ββββββββββ Proficient | Real-time feature extraction from Kafka β Iceberg |
| LLM API Integration | ββββββββββ Advanced | OpenAI API, structured output, tool-use patterns |
ποΈ LakehouseCopilot β Streaming Lakehouse + Text-to-SQL Agent
A production-grade data engineering platform that ingests live Wikipedia edits into Apache Iceberg, transforms them with dbt into a star schema, and exposes the warehouse to an AI agent grounded in dbt manifest metadata. The core experiment measures the execution accuracy lift of manifest-grounded retrieval over naive schema prompting.
| Attribute | Detail |
|---|---|
| Stack | PySpark Β· Apache Iceberg Β· Redpanda Β· Trino Β· dbt Β· pgvector Β· Airflow Β· Docker Β· Terraform |
| Scale | Live Wikipedia SSE stream Β· 10-service Docker stack Β· 40+ golden eval questions |
| Performance | End-to-end latency < 60s Β· 30s Spark micro-batches Β· at-least-once + dedup |
| Security | SELECT-only guardrail Β· no raw.* access Β· no multi-statement Β· auto-LIMIT enforcement |
| Impact | Quantified accuracy lift: manifest-grounded RAG vs. naive information_schema prompting |
| Repository | Adityaindurkhya/LakehouseCopilot |
The architecture separates concerns cleanly: the data half (Redpanda β Spark β Iceberg β dbt star schema) runs independently of the AI half (MCP server β pgvector β Trino). The manifest.json is the bridge β dbt writes it after every build, the indexer embeds each model's description, columns, and lineage into pgvector, and the agent retrieves the top-k most relevant models as schema context at query time.
Three MCP tools are exposed: search_schema for manifest-grounded retrieval, run_query for guardrailed Trino execution, and explain_lineage for upstream dependency traversal. The eval harness compares result sets order-insensitively across 5 question categories including adversarial SQL injection attempts.
Data Engineering & AI Systems Independent Portfolio Engineering 2024 β Present
Designing and building a full-stack streaming lakehouse platform from event ingestion to AI-powered natural language querying. Architecting the complete data path: SSE producer β Kafka β PySpark Structured Streaming β Apache Iceberg β dbt star schema β Trino β MCP text-to-SQL agent.
- Implemented at-least-once Kafka offset checkpointing with idempotent deduplication in dbt using
ROW_NUMBER()onevent_key - Built a manifest-grounded RAG pipeline: per-model pgvector embeddings, top-k retrieval, schema context injection into LLM prompts
- Designed a 4-rule SQL guardrail system (SELECT-only, no multi-statement, no raw.* access, auto-LIMIT) with 15 unit tests
- Authored a 40+ question golden eval dataset across 5 categories; built an execution-accuracy harness comparing result sets order-insensitively
- Provisioned AWS infrastructure (S3, Glue Data Catalog, Athena, IAM) via Terraform; authored dbt-athena profile for cloud migration path
- Configured daily Iceberg maintenance DAG: compaction, manifest rewrite, snapshot expiry, orphan removal via Airflow + Spark
| Recognition | Details |
|---|---|
| ποΈ Streaming Lakehouse Architecture | End-to-end platform: Wikipedia SSE β Iceberg β dbt β AI agent, < 60s latency |
| π§ Manifest-Grounded RAG | Measured execution accuracy lift over naive schema prompting with golden eval harness |
| π‘οΈ SQL Guardrail System | 4-rule guardrail with 15 adversarial unit tests; zero raw table exposure |
| π 40+ Golden Eval Questions | 5 categories including adversarial SQL injection; CI regression gate on main push |
| βοΈ AWS Deploy Path | Full Terraform IaC: S3, Glue, Athena, IAM; dbt-athena profile for cloud migration |
| π§ Production Iceberg Maintenance | Daily compaction, manifest rewrite, snapshot expiry, orphan removal via Airflow |
current_focus:
learning:
- Apache Iceberg v2 row-level deletes and merge-on-read
- Flink SQL for stateful stream processing
- LLM structured output and tool-use patterns
building:
- LakehouseCopilot: manifest-grounded text-to-SQL agent
- Execution accuracy eval harness with CI regression gate
- AWS deployment path: S3 + Glue + Athena + EMR Serverless
exploring:
- Polaris / Nessie as Iceberg REST catalog alternatives
- dbt semantic layer as agent context source
- Multi-agent orchestration over lakehouse schemas
open_to:
- Data Engineering (L4βL6)
- AI/ML Platform Engineering
- Senior Software Engineering
- Streaming & Lakehouse Architecture roles