AWS Snowflake Data Platform

This repository is the single monorepo for v1 of a greenfield data platform centered on AWS, Snowflake, dbt, Terraform, GitHub, Python, and SQL.

The v1 design is intentionally opinionated: one cloud, one warehouse platform, one infrastructure toolchain, and one CI/CD system. That keeps the first version interview-ready and practical without inventing premature integrations.

Selected v1 stack

Capability	v1 choice	Decision
Primary cloud	AWS	Use AWS as the only cloud boundary in v1.
Primary warehouse / lakehouse	Snowflake	Use Snowflake as the primary analytical storage and compute platform in v1.
Transformation layer	dbt + SQL	Use dbt for versioned transformations and SQL for warehouse-native logic.
Infrastructure as code	Terraform	Use Terraform as the primary IaC tool; defer Pulumi.
Source control and CI/CD	GitHub	Use GitHub as the primary repo host and CI/CD system; defer Azure DevOps.
Lightweight automation	Python	Use Python for small validation and automation tasks inside the repo.

Why these are the v1 choices

• AWS is the primary cloud because the platform goal is AWS-first rather than multi-cloud.
• Snowflake is the primary warehouse and lakehouse decision because v1 should center on one analytical platform instead of splitting between Snowflake and Databricks.
• Terraform is the primary IaC choice because declarative infrastructure is a better fit than introducing Pulumi and application-language abstractions in the first version.
• GitHub is the primary CI/CD choice because the repository already lives in GitHub and GitHub Actions is enough for v1 repository automation.
• dbt, SQL, and Python stay in the stack because they directly support modeling, warehouse administration, and lightweight repo automation in a single-repo workflow.

Deferred tools

• Fivetran: defer until the first source systems, ownership boundaries, and ingestion SLAs are defined.
• Pulumi: defer because Terraform is the primary IaC tool for v1.
• Azure DevOps: defer because GitHub is the primary source control and CI/CD platform for v1.
• Databricks: defer from v1 and revisit only if the platform needs a separate Spark-heavy compute layer beyond Snowflake and dbt.
• Sigma: defer until a concrete BI consumption layer is needed.
• Atlan: defer until data governance, catalog, and lineage ownership are defined.
• Slingshot: defer until program or portfolio tracking needs are clear.

Repository Layout

• AGENTS.md defines the operating contract for future work.
• docs/architecture.md records the current target architecture and delivery boundaries.
• docs/adr/ stores architecture decision records for this monorepo.
• docs/task-log.md tracks task-by-task changes, remaining work, and validation steps.
• infra/terraform/ contains modular Terraform under modules/ and environment roots under envs/dev and envs/prod.
• transform/dbt/ contains the dbt project scaffold.
• snowflake/ contains Snowflake-specific SQL boundaries and guidance.
• scripts/validate_repo.py validates the baseline repository structure.
• .github/workflows/repo-validation.yml runs repository validation in GitHub Actions.
• Makefile provides bootstrap-friendly Terraform format and validate targets.

Validation

Run the repository validation script from the repo root for the structural baseline:

python3 scripts/validate_repo.py

For Terraform changes, run the full local Terraform validation flow as well:

make terraform-fmt
make terraform-validate