Data Catalog with DataHub

This project aims to provision end-to-end pipeline lineage with Airbyte, Airflow, dbt, BigQuery and DataHub as the Data Catalog/Lineage platform. Also ensuring sibling relationships are not duplicate (e.g: Airbyte destination table for a given source matches the same entity as dbt source table)

Quick Start:

1. Spin up DataHub

docker compose -f datahub/compose.yaml up -d

2. Spin up Airflow

docker compose -f airflow/compose.yaml up --build --force-recreate -d

3. Spin up Airbyte with abctl

brew tap airbytehq/tap
brew install abctl

abctl local install

4. Fetch Airbyte credentials

abctl local credentials

5. Build the dbt-bigquery Docker Image

docker build -t dbt-bigquery:latest dbt/ --no-cache

6. Build the datahub-ingest Docker Image

docker build -t datahub-ingest:latest datahub/ --no-cache

7. Terraform

Follow the instructions on terraform for guidelines on how to run/apply

Reference Docs

Refer to the specific project folder on how to start each component individually

• DataHub
• Airflow
• Airbyte
• dbt-bigQuery
• Terraform