Skip to content
View Adityaindurkhya's full-sized avatar

Block or report Adityaindurkhya

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
Adityaindurkhya/README.md

Typing SVG


MS Data Analytics Engineering BTech Computer Science Location


Portfolio LinkedIn Email GitHub


Profile Views GitHub Followers GitHub Stars


β—ˆ About

Data engineer and AI systems builder focused on the intersection of streaming infrastructure and intelligent query interfaces. I design and build production-grade lakehouse platforms β€” from raw event ingestion through governed star schemas to AI agents that query them in natural language.

My work centers on Apache Iceberg, PySpark Structured Streaming, dbt, and Trino for the data layer, and MCP-based text-to-SQL agents grounded in dbt manifest metadata for the AI layer. I care deeply about correctness guarantees: at-least-once delivery with idempotent deduplication, execution-accuracy evaluation over SQL string matching, and guardrail-enforced read-only agent access.

I approach data platform engineering with a product mindset β€” every pipeline decision is justified by the analytical workload it enables, and every agent capability is measured against a golden eval dataset.

Open To: Data Engineering Β· AI/ML Engineering Β· Platform Engineering Β· Senior SWE roles at companies building on modern lakehouse or AI-native data stacks.


β—ˆ Tech Stack

Languages

Python SQL Bash Scala

Frontend & Visualization

React TypeScript

Backend & Databases

FastAPI PostgreSQL Redis Kafka

Cloud, DevOps & Tooling

AWS Docker Terraform GitHub Actions Airflow


β—ˆ AI / ML Expertise

Domain Proficiency Details
Text-to-SQL Agents β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘ Advanced MCP server, manifest-grounded RAG, retry loop, guardrails
Vector Retrieval (RAG) β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘ Advanced pgvector, per-model embedding chunks, top-k retrieval
LLM Evaluation β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘ Proficient Execution accuracy harness, golden datasets, CI regression gate
Prompt Engineering β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘ Proficient Schema context injection, adversarial guardrail testing
Streaming ML Features β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘ Proficient Real-time feature extraction from Kafka β†’ Iceberg
LLM API Integration β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘ Advanced OpenAI API, structured output, tool-use patterns

β—ˆ Featured Projects

πŸ”οΈ LakehouseCopilot β€” Streaming Lakehouse + Text-to-SQL Agent

A production-grade data engineering platform that ingests live Wikipedia edits into Apache Iceberg, transforms them with dbt into a star schema, and exposes the warehouse to an AI agent grounded in dbt manifest metadata. The core experiment measures the execution accuracy lift of manifest-grounded retrieval over naive schema prompting.

Attribute Detail
Stack PySpark Β· Apache Iceberg Β· Redpanda Β· Trino Β· dbt Β· pgvector Β· Airflow Β· Docker Β· Terraform
Scale Live Wikipedia SSE stream Β· 10-service Docker stack Β· 40+ golden eval questions
Performance End-to-end latency < 60s Β· 30s Spark micro-batches Β· at-least-once + dedup
Security SELECT-only guardrail Β· no raw.* access Β· no multi-statement Β· auto-LIMIT enforcement
Impact Quantified accuracy lift: manifest-grounded RAG vs. naive information_schema prompting
Repository Adityaindurkhya/LakehouseCopilot

The architecture separates concerns cleanly: the data half (Redpanda β†’ Spark β†’ Iceberg β†’ dbt star schema) runs independently of the AI half (MCP server β†’ pgvector β†’ Trino). The manifest.json is the bridge β€” dbt writes it after every build, the indexer embeds each model's description, columns, and lineage into pgvector, and the agent retrieves the top-k most relevant models as schema context at query time.

Three MCP tools are exposed: search_schema for manifest-grounded retrieval, run_query for guardrailed Trino execution, and explain_lineage for upstream dependency traversal. The eval harness compares result sets order-insensitively across 5 question categories including adversarial SQL injection attempts.



β—ˆ Experience

Data Engineering & AI Systems Independent Portfolio Engineering 2024 β€” Present

Designing and building a full-stack streaming lakehouse platform from event ingestion to AI-powered natural language querying. Architecting the complete data path: SSE producer β†’ Kafka β†’ PySpark Structured Streaming β†’ Apache Iceberg β†’ dbt star schema β†’ Trino β†’ MCP text-to-SQL agent.

  • Implemented at-least-once Kafka offset checkpointing with idempotent deduplication in dbt using ROW_NUMBER() on event_key
  • Built a manifest-grounded RAG pipeline: per-model pgvector embeddings, top-k retrieval, schema context injection into LLM prompts
  • Designed a 4-rule SQL guardrail system (SELECT-only, no multi-statement, no raw.* access, auto-LIMIT) with 15 unit tests
  • Authored a 40+ question golden eval dataset across 5 categories; built an execution-accuracy harness comparing result sets order-insensitively
  • Provisioned AWS infrastructure (S3, Glue Data Catalog, Athena, IAM) via Terraform; authored dbt-athena profile for cloud migration path
  • Configured daily Iceberg maintenance DAG: compaction, manifest rewrite, snapshot expiry, orphan removal via Airflow + Spark

Python Apache Spark Apache Iceberg dbt Trino Kafka pgvector Terraform Airflow


β—ˆ Achievements

Recognition Details
πŸ”οΈ Streaming Lakehouse Architecture End-to-end platform: Wikipedia SSE β†’ Iceberg β†’ dbt β†’ AI agent, < 60s latency
🧠 Manifest-Grounded RAG Measured execution accuracy lift over naive schema prompting with golden eval harness
πŸ›‘οΈ SQL Guardrail System 4-rule guardrail with 15 adversarial unit tests; zero raw table exposure
πŸ“Š 40+ Golden Eval Questions 5 categories including adversarial SQL injection; CI regression gate on main push
☁️ AWS Deploy Path Full Terraform IaC: S3, Glue, Athena, IAM; dbt-athena profile for cloud migration
πŸ”§ Production Iceberg Maintenance Daily compaction, manifest rewrite, snapshot expiry, orphan removal via Airflow

β—ˆ Certifications

Amazon Web Services

AWS AWS

NPTEL

NPTEL NPTEL

Cisco

Cisco


β—ˆ Coding Profiles

LeetCode GeeksforGeeks HackerRank CodeChef


β—ˆ GitHub Analytics


β—ˆ GitHub Trophies

trophy


β—ˆ Contribution Activity

Aditya's github activity graph


β—ˆ Contribution Snake

github contribution grid snake animation

β—ˆ Current Focus

current_focus:
  learning:
    - Apache Iceberg v2 row-level deletes and merge-on-read
    - Flink SQL for stateful stream processing
    - LLM structured output and tool-use patterns
  building:
    - LakehouseCopilot: manifest-grounded text-to-SQL agent
    - Execution accuracy eval harness with CI regression gate
    - AWS deployment path: S3 + Glue + Athena + EMR Serverless
  exploring:
    - Polaris / Nessie as Iceberg REST catalog alternatives
    - dbt semantic layer as agent context source
    - Multi-agent orchestration over lakehouse schemas
  open_to:
    - Data Engineering (L4–L6)
    - AI/ML Platform Engineering
    - Senior Software Engineering
    - Streaming & Lakehouse Architecture roles

β—ˆ Connect

Gmail LinkedIn GitHub Portfolio


"The pipeline is only as trustworthy as the guarantees you can prove about it."

Popular repositories Loading

  1. AdityaIndurkhya.github.io AdityaIndurkhya.github.io Public

    HTML

  2. DataAgentPro DataAgentPro Public

    Stateful multi-agent AI system for natural language data analysis β€” built with LangGraph, FastAPI, DuckDB, and React. Features SQL/Python execution, validation, reflection, Plotly visualizations, a…

    Python

  3. ClimateGPT ClimateGPT Public

    AI-powered climate intelligence system using MCP tools, RAG pipelines, and LLMs to answer climate risk queries. Benchmarked on 20 gold cases with 70% accuracy across BWS, RFR, and DRR indicators.

    Python

  4. DataPulseAI DataPulseAI Public

    End-to-end data quality & observability platform monitoring a retail pipeline for nulls, duplicates, schema drift, volume anomalies, and downstream lineage impact β€” built with Python, Pandas, SQLit…

    Python

  5. LakehouseCopilot LakehouseCopilot Public

    Streaming lakehouse ingesting live Wikipedia edits into Apache Iceberg via PySpark, transformed with dbt into a star schema, and queried by a manifest-grounded text-to-SQL agent measuring accuracy …

    Python

  6. Adityaindurkhya Adityaindurkhya Public