de — Data Engineering coursework

Coursework repository for the Engenharia de Dados course (CESAR School, Computer Science, 2026.2). Each exercise builds the same underlying skill — declaring data-lake infrastructure explicitly, instead of letting a tool infer it — with a different piece of the stack per cycle.

Labs

Lab	Topic	IaC	Status
labs/01	Declared Glue schema, no Crawler, Athena scan-limit guardrail	CloudFormation	PR #1 — merged
labs/02	Same data lake, provisioned from scratch, local Terraform state	Terraform	PR #2 — merged
labs/03	The same stack refactored into module + remote backend + workspace, with a clean plan	Terraform	PR #4 — merged
labs/04	The trusted table as partitioned Parquet, with a declared per-query cost target proven in Athena	Terraform	PR #9 — merged
labs/05	An idempotent Glue pipeline: re-run, late data and a re-sent file, same answer before and after	Terraform	PR #10 — merged

Cycle 4 (analytics modeling and ELT with AWS Glue) has no lab here: the course replaced that cycle's individual exercise with Part 1 of the group project (AV1), which is kept in the group's own repository. labs/NN follows the course's Exercício numbering, so labs/04 is Exercício 04, from Cycle 5.

Shared infrastructure that outlives any single lab lives outside labs/: infra/tfstate-backend/ provisions the S3 bucket holding the remote Terraform state, so a lab's destroy can never take it down.

Each lab's own README.md has the exact deploy/verify/destroy commands. Each lab's DECISOES.md documents the required engineering decisions — schema types, partitioning, and Athena cost guardrails — with evidence, not assumption.

Stack

AWS (S3, Glue Data Catalog, Glue jobs in PySpark, Athena), Terraform and CloudFormation, Python for synthetic data generation.

Conventions

• One exercise per labs/NN/ directory, self-contained (its own terraform/ or infra/, docs/, verification/). Infrastructure whose lifecycle crosses labs lives in the root-level infra/ instead.
• DECISOES.md per lab: one justified paragraph per required decision, backed by a measurement or a query result, not a guess.
• docs/evidence/ per lab: the raw command output or screenshot behind each decision and each verification step.
• Only authored work is published. Course-provided material (slides, enunciados, rubricas, answer keys) stays local; each lab's README.md summarizes the assignment instead. Scripts the course ships and the solution depends on (data generator, verifica.sh) are kept so each lab stays reproducible.
• No terraform.tfstate or filled-in terraform.tfvars/backend.hcl — only their *.example templates.
• Language: code, READMEs, and commits are in English. DECISOES.md and the course-provided scripts are in Portuguese, because the rubric grades them in that language under that file name.