- Knowledge Sharing - Data Engineering skilling
- Table of Content (ToC)
- Overview
- References
- Articles and posts
- Idempotency with Spark
- Complete SQL and PySpark guide
- End-to-end project in the AI era
- Full open source lakehouse
- Data Engineering - How to transition from one role to the next
- The Essential Skill Stack: Foundational Knowledge Every Data Engineer Must Hav
- Data Engineering is not one job anymore
- Claude Skills in Data Engineering
- Spark on Luminousmen on Substack
- Awesome open source Data Engineering - Resources by Gunnar Morling
- Data ingestion methods
- Data Talks Club - Zoomcamp
- Reliable Data Engineering
- Data Expert - Resources by Zach Wilson
- Awesome Data Engineering - Resources by Igor Barinov
- Data Engineering resources by Amjล
- Data Engineering resources by Ahmed Alsaket
- Data Engineering articles by Saurav Singh
- Data Engineering articles by Mayurkumar Surani
- Data Engineering Q&A by Sachin Chandrashekhar
- Data Engineering on DataBricks by Jakub Lasak
- Data Engineering illustrations by Riya Khandelwal
- Books
- Curricula
- Specific topics
- DataBricks - Data Engineering
- Web sites
Created by gh-md-toc
This project aims at collecting in a single place training resources for the up skilling of data engineers.
Even though the members of the GitHub organization may be employed by some companies, they speak on their personal behalf and do not represent these companies.
- Data Engineering Helpers - Knowledge Sharing - Cheat sheets
- Material for the Data platform - Architecture principles
- Material for the Data platform - Data contracts
- Material for the Data platform - Data products
- Material for the Data platform - Data quality
- Material for the Data platform - Semantic layer
- Material for the Data platform - Data lakehouse
- Material for the Data platform - Data management
- Material for the Data platform - Modern Data Stack (MDS) in a box
- Material for the Data platform - Data life cycle
- Material for the Data platform - Metadata
- Link to the article on Medium
- Title: Mastering PySpark & SQL: A Complete Hands-On Guide for Data Engineers
- Link to the article on Medium
- Title: How Important It Has Become to Build a Pet Project or Product-Equivalent E2E Project on Your Own in the AI Era
- Link to the article
- Author: Viktor Gamov
- Date: April 2026
- Title: Building a Streaming Lakehouse with Open Source: Kafka to Iceberg to Trino to Superset
- GitHub - Flink Trino Superset pipeline
- Soda blog - Data Engineering fundamentals
- Date: May 2026
- Author: Fabiana Perraz
- Excerpts:
Key Takeaways:
- Data engineering is the discipline of building systems that move data reliably from source to use: ingestion, transformation, validation, storage, and consumption.
- The job has shifted from "moving data" to "ensuring data is correct, fresh, and usable" at every pipeline stage.
- Data engineering is evolving toward greater automation, observability, and governance.
- Most data failures are silent: pipelines run successfully but produce incorrect outputs. Engineers' job is to design systems that fail visibly.
Core skills of a Data Engineer:
- Strong SQL capability
- Python or a similar general-purpose language
- How data should be modeled
- Distributed systems thinking
- Cloud infrastructure knowledge
- Reason about data quality
- Communication
- Publisher: Substack - Luminousmen
- Date: Aug. 2025
- Link to the article on Substack
- Post on LinkedIn about the article
- Author: Kirill Bobrov (Kirill Bobrov on LinkedIn, Kirill Bobrov on Substack)
- Title: Data Engineering Design Patterns You Must Learn in 2026
- Date: Jan. 2026
- Article on Medium
- GitHub - Igor Barinov - Awesome Data Engineering
- Author: Igor Barinov
- Title: Data Engineering Resources
- Date: Aug. 2025
- Author: Ahmed Alsaket (Ahmed Alsaket on LinkedIn)
- Post on LinkedIn
- Scrimba - Master Python
- Scrimba - Learn SQL
- Scrimba - Learn MySQL
- Scrimba - Learn MongoDB
- Scrimba - Dominate PySpark
- Scrimba - Learn Bash, Airflow & Kafka
- Scrimba - Learn Git & GitHub
- Scrimba - Learn CICD basics
- Scrimba - Decode Data Warehousing
- Scrimba - Learn DBT
- Scrimba - Learn Data Lakes
- Scrimba - Learn DataBricks
- Scrimba - Learn Azure Databricks
- Scrimba - Learn Snowflake
- Scrimba - Learn Apache NiFi
- Scrimba - Learn Debezium
- Scrimba - Reddit ETL Pipeline
- Scrimba - Surfline Dashboard
- Scrimba - Finnhub Streaming Data Pipeline
- Scrimba - Audiophile End-To-End ELT Pipeline
- Scrimba - Streamify
- Medium - Mayur Surani's page
- Author: Mayurkumar Surani (Mayurkumar Surani on LinkedIn, Mayurkumar Surani on Medium)
- Publisher: Medium
- A few posts:
- Medium - Mayurkumar Surani - May 2025 - End-to-End ETL Pipeline with AWS, PySpark, and Databricks
- Medium - Mayurkumar Surani - May 2025 - Top 20 PySpark Functions Every Data Engineer Should Master
- Medium - Mayurkumar Surani - Aug. 2025 - Mastering Databricks and DBT: An End-to-End Production-Grade Data Engineering Project
- Medium - Mayurkumar Surani - Aug. 2025 - Bronze Layer - Dynamic Incremental Data Ingestion with Databricks Autoloader Part-02
- Author: Sachin Chandrashekhar (Sachin Chandrashekhar on LinkedIn)
- Date: Aug. 2025
- LinkedIn post
- Author: Jakub Lasak
- Date: Aug. 2025
- GitHub - Databricks apparel streaming
- LinkedIn post -
- Date: Jul. 2026
Databricks just made 5 paid products free - Lakebase, Agent Bricks, Genie Code, and more. You can build a hire-worthy portfolio for $0.
For years, the honest answer to "how do I build a modern Databricks portfolio at home" was: you basically couldn't - the free tier couldn't touch this stack, and a trial workspace expires before you finish.
Free Edition just killed that excuse. Five products that used to require a paid workspace are now free. Here's what each one is and what building with it proves to a hiring manager.
๐ญ. ๐๐ฎ๐ธ๐ฒ๐ฏ๐ฎ๐๐ฒ Managed Postgres that sits right next to your lakehouse. Sync a Delta gold table into it and serve a live app. Proves you grasp OLTP vs OLAP and can design a serving layer, not just batch ETL.
๐ฎ. ๐๐ด๐ฒ๐ป๐ ๐๐ฟ๐ถ๐ฐ๐ธ๐ Builds and auto-tunes AI agents on your own tables. Ship an agent that answers questions over a real dataset. Proves you can put production GenAI on governed data, the skill every team is hiring for right now. ๐ฏ. ๐๐ฒ๐ป๐ถ๐ฒ ๐๐ผ๐ฑ๐ฒ Ask in plain English and Genie Code writes, runs, and iterates on the code itself (the coding agent, not AI/BI Genie the analytics assistant) - analyzing a dataset, cleaning a pipeline, building a chart. Point it at a raw dataset and let it do the first pass, then steer its approach. Proves you can put an agentic coding assistant to work on real data and still own the result.
๐ฐ. ๐ฆ๐ฒ๐ฟ๐๐ฒ๐ฟ๐น๐ฒ๐๐ ๐๐ฃ๐จ๐ On-demand GPU compute with zero cluster setup. Fine-tune a small model or run GPU inference on a real dataset. Proves you can handle ML and AI workloads, not only SQL and Spark.
๐ฑ. ๐๐ฎ๐ธ๐ฒ๐ณ๐น๐ผ๐ ๐๐ฒ๐๐ถ๐ด๐ป๐ฒ๐ฟ Drag-and-drop builder for declarative pipelines. Wire up a full bronze-to-gold ingestion flow without hand-coding orchestration. Proves you know medallion architecture and can ship a pipeline end to end.
- LinkedIn post - Senior Databricks Data Engineer interview
- Date: Apr. 2026
- Substack - 7 Questions That Expose How You Debug Slow Pipelines
- Author: Riya Khandelwal (Riya Khandelwal on LinkedIn)
- Date: Aug. 2025
- GitHub - Data Engineering helpers - Skilling - Dedicated readme for material published by Riya Khandelwal
- Authors: Joe Reis and Matt Housley
- Date: July 2022
- LinkedIn media - Copy of the book
- Book on Amazon
- Book on O'Reilly
- Print length: 447 pages
- ISBN-10: 1098108302
- ISBN-13: 978-1098108304
- Author: Martin Kleppmann
- Book on Amazon
- Book on O'Reilly
- Date: Jan. 2026 (2nd version)
- Author: Andreas Kretz (Andreas Kretz on LinkedIn)
- LinkedIn service page - Data Engineering training program
- LinkedIn learning - Transition from data science to data engineering
- Level: beginners
- Duration: 47 minutes
- Author: Pooja Jain
- GitHub - Abel Tavares - Batch data pipeline
- Date: Nov. 2025
- Author: Abel Tavares
- Post on LinkedIn
- Overview:
- Abel built a batch data pipeline with observability, data quality,
and lineage tracking using open-source tools.
- Apache Airflow for orchestration
- DuckDB for fast analytical processing
- Delta Lake for ACID transactions on data lakes
- MinIO (S3-compatible) for storage
- Trino Software Foundation for distributed SQL queries
- Metabase for dashboards
- Soda core for data quality checks
- Marquez + OpenLineage for end-to-end lineage
- Prometheus Group + Grafana Labs for monitoring
- Abel built a batch data pipeline with observability, data quality,
and lineage tracking using open-source tools.
- The pipeline follows the Medallion Architecture (Bronze โ Silver โ Gold) with automated data cleaning, validation, and business aggregations.
- Everything runs with Docker, including automated dashboard generation in Metabase and Grafana.
- Author: Pooja Jain
- Linked post