Data Engineering Agent Skills

A production-grade, machine-validated collection of Agent Skills for data engineering, focused on multi-cloud and open-source tooling.

Skills are self-contained directories that teach an AI agent how to complete a specific data-engineering task: prerequisites, safe execution tiers, step-by-step workflow phases, validation steps, and a definition of done. Every skill is validated against a JSON Schema, linted, unit-tested, and link-checked in CI.

Installation

git clone git@github.com:your-org/DE_skills.git
make ci          # run the full local validation pipeline

To use a skill with an agent harness, point the harness at the skill directory (or install via the marketplace manifests — see Agent integrations).

Available Skills

• Data Governance

  • Data Contracts: Design and enforce producer-consumer data contracts across the platform.
  • DataHub Catalog: Register, document, and discover data assets in DataHub.
  • Metadata Catalog Comparison: Choose between DataHub, Unity Catalog, Glue, and knowledge-catalog tools.
  • OpenLineage: Emit and query lineage events from orchestrators and compute engines.
  • PII Classification & Masking: Detect, classify, and mask personally identifiable information in datasets.

• Data Infrastructure

  • CI/CD for dbt: Test, lint, and deploy dbt projects through pull-request pipelines.
  • Dockerized Airflow Development: Stand up a reproducible local Airflow development environment with Docker.
  • Kubernetes for Data Jobs: Run batch and streaming workloads on Kubernetes with best practices.
  • Terraform for Data Platforms: Provision warehouse, storage, and orchestrator infrastructure as code.

• Data Ingestion

  • Airbyte: Build and operate ELT connectors for batch and sync-based ingestion.
  • dlt (Python) Ingestion: Declarative Python data pipelines that load from any source into warehouses.
  • File Ingestion (GCS/S3): Ingest files from object storage into lakehouse tables with schema inference.
  • Apache Kafka Basics: Create topics, produce and consume messages, and size clusters for data pipelines.
  • Kafka Connect: Stand up connectors to move data between Kafka and storage/databases.

• Data Processing

  • Google Dataflow Basics: Build and run Apache Beam pipelines on Dataflow with streaming and batch support.
  • DuckDB Basics: Use DuckDB for local analytics, in-process OLAP, and zero-copy queries over Parquet.
  • Apache Flink Basics: Build streaming and batch jobs with Flink DataStream and Table APIs.
  • Apache Spark Basics: Write, run, and debug PySpark batch jobs against lakehouse tables.
  • Spark Optimization: Tune partitions, joins, memory, and shuffles for faster Spark jobs.
  • Spark Troubleshooting: Diagnose OOM, skew, stragglers, and job failures using driver/executor logs.

• Data Quality

  • Data Observability: Instrument pipelines with metrics, freshness, volume, and anomaly detection.
  • Data Quality Incident Runbook: Triage, remediate, and post-mortem data quality incidents end to end.
  • dbt Data Quality Tests: Enforce uniqueness, not-null, referential, and freshness guarantees in dbt.
  • Great Expectations: Define, run, and monitor data expectations against batch data.
  • Soda: Write and run data quality checks with Soda Core and Soda Checks Language.

• Data Transformation

  • dbt Core: Model, test, and document warehouse data with dbt projects.
  • dbt Tests & Macros: Write singular/generic tests, macros, and packages for reliable dbt projects.
  • SQL Transformation Best Practices: Write performant, maintainable, and reviewable transformation SQL.

• Databases

  • MongoDB Basics: Model, query, and integrate MongoDB document collections into pipelines.
  • MySQL Basics: Configure, tune, and integrate MySQL into data platforms.
  • PostgreSQL Basics: Configure, tune, and integrate PostgreSQL into data platforms.

• Getting Started

  • Data Engineering Authentication: Configure cloud and toolchain credentials using short-lived, scoped access.
  • Data Engineering Stack Setup: Bootstrap the local DE toolchain: Python, Spark, dbt, Airflow, and Docker.
  • MCP Servers for Data: Connect agents to data platforms through Model Context Protocol servers.

• Orchestration

  • Apache Airflow Basics: Install, configure, and operate Airflow as the pipeline orchestrator.
  • Airflow DAG Authoring: Author production DAGs with idempotency, retries, and correct task design.
  • Airflow Job Failure Troubleshooting: Diagnose scheduler, task, and infrastructure failures from Airflow logs.
  • Dagster Basics: Build typed, testable data pipelines with assets, ops, and jobs in Dagster.
  • Prefect Basics: Orchestrate Python data workflows with Prefect flows and deployments.

• Solutions

  • Batch ETL Pipeline: Build an end-to-end batch ETL pipeline with ingestion, transformation, and quality gates.
  • Data Platform Architecture: Design a production data platform: zones, tools, and governance blueprint.
  • Lakehouse Migration: Migrate from data warehouse to open lakehouse with staged cutover.
  • Streaming Analytics Pipeline: Build an end-to-end streaming pipeline with CDC, processing, and serving.

• Storage & Lakehouse

  • Delta Lake Basics: Use Delta tables, ACID transactions, and versioning in lakehouse pipelines.
  • Apache Hudi Basics: Manage incremental upserts and table services on Hudi tables.
  • Apache Iceberg Basics: Create tables, manage snapshots, and use time travel with Iceberg catalogs.
  • Object Storage Basics: Design buckets, prefixes, lifecycle, and security for S3 and GCS.
  • Parquet & File Formats: Choose and optimize Parquet, ORC, Avro, and compression for analytic workloads.

• Streaming

  • Flink SQL: Author streaming pipelines with Flink SQL and change-data-capture sources.
  • Kafka Streams: Build stateful stream processing applications on Kafka Streams.
  • Streaming Architecture Patterns: Choose event-driven, CDC, and lambda/kappa patterns for real-time platforms.

• Warehousing

  • BigQuery Basics: Manage datasets, tables, and jobs; query and load data in BigQuery.
  • Amazon Redshift Basics: Provision clusters, design distribution styles, and run analytics on Redshift.
  • Snowflake Basics: Configure warehouses, databases, stages, and query patterns in Snowflake.
  • Warehouse Optimization: Reduce cost and improve query performance with partitioning, clustering, and materialization.

Repository layout

skills/<domain>/<skill-name>/     one directory per skill
  SKILL.md                        entry point: frontmatter + workflow + definition of done
  references/                     per-topic deep-dive docs
  scripts/                        executable helpers + requirements.txt + tests
  assets/                         templates (DAGs, dbt projects, terraform, configs)
catalog.yaml                      canonical skill registry (source of truth for the index)
schema/skill.schema.json          JSON Schema for SKILL.md frontmatter
tools/                            repository tooling (validate, generate, check)

Development

See CONTRIBUTING.md for the authoring checklist and quality gates every skill must pass.

Support

Open an issue in the repository tracker for bugs, outdated patterns, or skill requests.

License

Apache 2.0. See LICENSE.