Data Engineering Zoomcamp 2026 — Capstone Project
A production-grade, end-to-end data platform built with Terraform, Bruin, Python, BigQuery, and Looker Studio.
🔗 View Live Dashboard on Looker / Data Studio
| Page | Title | Focus |
|---|---|---|
| 1 | Global Net Zero Tracker | World CO2 emitters, decoupling scatter, leaderboard |
| 2 | Indonesia Strategy — Regional Deep Dive | ASEAN benchmarking, efficiency trends, renewable gap |
Can countries grow their economies without growing their carbon emissions? This question — known as economic decoupling — sits at the heart of every climate pledge, Paris Agreement commitment, and ESG investment decision made today.
This platform answers it with data: combining 25 years of World Bank economic and environmental indicators across 231 countries, transforming them through a multi-layer analytical pipeline, and surfacing the results in an interactive dashboard that makes the decoupling story immediately understandable.
The core metric — the Decoupling Index — is defined as:
Decoupling Index = GDP Year-over-Year Growth % − CO2 Year-over-Year Growth %
A positive score means the economy is outgrowing its emissions. A negative score means emissions are outgrowing the economy. The dashboard tracks this across every country, every year, from 2000 to 2024.
┌─────────────────────────────────────────────────────────────────────┐
│ SOURCE │
│ World Bank WDI API · 231 countries · 2000–2024 · Free │
│ 3 indicators: GDP PPP · CO2 Emissions · Renewable Energy Share │
└──────────────────────────────┬──────────────────────────────────────┘
│ wbgapi · monthly @bruin schedule
▼
┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 1 — INFRASTRUCTURE (Terraform IaC) │
│ GCS Bucket · BQ raw_energy_data · BQ staging_energy_data │
│ BQ marts_energy_data · Service Account · IAM (least-privilege) │
└──────────────────────────────┬──────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 2 — BRONZE LAYER [raw.ingest_wb_data] Python Asset │
│ World Bank API → pivot → Parquet → GCS → BQ External Table │
│ 6K+ rows · 261 economy codes · Years 2000–2024 │
└──────────────────────────────┬──────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 3 — SILVER LAYER [staging.stg_energy_metrics] SQL Asset │
│ Type-cast · Null-guard · Exclude 30 WB aggregate codes │
│ Deduplication via QUALIFY ROW_NUMBER() · 9 quality checks │
│ 5K+ rows · 231 actual countries │
└──────────────────────────────┬──────────────────────────────────────┘
│
┌────────────────┴────────────────┐
▼ ▼
┌──────────────────────────┐ ┌──────────────────────────────────┐
│ PHASE 4a — DIMENSION │ │ PHASE 4b — GOLD FACT TABLE │
│ [dim_country] │ │ [fct_net_zero_performance] │
│ 231 countries │ │ PARTITION BY year_date (DATE) │
│ 3 regions: │ │ CLUSTER BY country_code, region │
│ ASEAN (11 countries) │ │ │
│ G20 (18 countries) │ │ Computed KPIs: │
│ Global (202 others) │ │ · gdp_yoy_growth_pct │
│ is_focus_country flag │ │ · co2_yoy_growth_pct │
│ 7 quality checks │ │ · decoupling_index │
└───────────────────────── ┘ │ · gdp_index_2015 │
│ · co2_index_2015 │
│ · rolling_3yr_gdp_per_co2 │
│ 8 quality checks │
└──────────────┬───────────────────┘
▼
┌──────────────────────────────────┐
│ PHASE 5 — DASHBOARD │
│ Data Studio (live BQ connect) │
│ 2 pages · 7 tiles │
│ Partition-pruned queries │
└──────────────────────────────────┘
| Layer | Tool | Version | Purpose |
|---|---|---|---|
| Infrastructure | Terraform | >= 1.0 |
Provision all GCP resources as code. Zero click-ops. |
| Cloud Platform | Google Cloud Platform | — | GCS (data lake) + BigQuery (warehouse) |
| Storage | Google Cloud Storage | — | Bronze layer — raw Parquet files |
| Warehouse | BigQuery | — | Silver + Gold layers, partitioned + clustered |
| Orchestration | Bruin CLI | v0.11.524 |
Replaces Airflow + dbt + Great Expectations in one tool |
| Ingestion | Python 3.13 | via Bruin | World Bank API → GCS → BQ External Table |
| Transformation | SQL (BigQuery dialect) | via Bruin | Staging + dimension + fact tables |
| Data Source | World Bank WDI API | wbgapi==1.0.14 |
Free, no approval, 1,500+ indicators |
| Visualization | Looker / Data Studio | — | Live BigQuery connector, partition-aware |
| Local Dev | uv | — | Fast Python package manager |
| IDE | Visual Studio Code | >= v1.111 |
Open source code editor |
ZC2026-net-zero-tracker/
│
├── assets/ # All Bruin pipeline assets
│ ├── requirements.txt # Python deps — Bruin reads this
│ ├── raw/
│ │ └── ingest_wb_data.py # WB API → GCS → BQ External Table
│ ├── staging/
│ │ └── stg_energy_metrics.sql # Clean, type, quality-check
│ └── core/
│ ├── dim_country.sql # Country dimension + regional groups
│ └── fct_net_zero_performance.sql # KPI fact table, partitioned
│
├── terraform/
│ ├── provider.tf # GCP provider, version-locked
│ ├── variables.tf # All input variables
│ ├── main.tf # GCS + BigQuery datasets
│ ├── outputs.tf # Post-apply resource outputs
│ └── terraform.tfvars.example # Template (real .tfvars is gitignored)
│
├── credentials/ # Gitignored — SA key lives here
├── img/ # Images for README.MD
├── .bruin.yml # Gitignored — Bruin GCP connection
├── pipeline.yml # Schedule: @monthly, start: 2000-01-01
├── .env.example # Env var template
└── README.md
All data is sourced from the World Bank World Development Indicators (WDI) — the most authoritative publicly available collection of development statistics, compiled from 50 international partners including the IEA, EDGAR (JRC/EC), IMF, and OECD.
| Indicator Code | Column Name | Unit | Source | Decision Rationale |
|---|---|---|---|---|
NY.GDP.MKTP.PP.KD |
gdp_ppp |
Constant 2021 int'l $ | World Bank / IMF | Constant prices (KD) remove inflation — essential for 25-year cross-country trend. Current prices (CD) inflate growth figures with exchange rate movements. |
EN.GHG.CO2.MT.CE.AR5 |
co2_emissions_mt |
Mt CO2e | EDGAR v8.0 (JRC/IEA) | Replaces deprecated EN.ATM.CO2E.KT (retired Dec 2024). Uses IPCC AR5 GWP values. Covers agriculture, energy, industry, transport, waste. Has 2024 data. |
EG.ELC.RNEW.ZS |
renewable_energy_share |
% | IEA | Renewable electricity output as % of total. Structural health check for energy transition. Note: IEA 2-year lag — latest data available is 2021. |
Coverage: 231 countries · 25 years (2000–2024) ·
@monthlyBruin schedule · ~5,747 staging rows
Fetches all three WDI indicators via wbgapi, pivots from long to wide format, uploads to GCS as Parquet, and registers a BigQuery External Table with autodetect schema.
Key engineering decisions:
- Entry point is
def main()— Bruin's required calling convention - Credentials passed explicitly to both
gcsfsandbigquery.Client()— never relies on ADC inside Bruin's isolated venv GOOGLE_APPLICATION_CREDENTIALSresolved to absolute path viapathlib.Path.resolve()before usedropna(how="all")preserves country rows with partial data (e.g. 2024/2025 sparse years) while removing truly empty rows
Promotes raw data from Bronze to Silver with full type safety and quality enforcement.
Key transformations:
- Explicit
CASTtoINT64,FLOAT64,STRING— no schema inference - Excludes all 30 World Bank aggregate region codes (
WLD,EAS,HIC, etc.) — 750 rows removed, leaving 231 actual countries CASE WHEN <= 0 THEN NULLnullifies impossible values while keeping the rowQUALIFY ROW_NUMBER()deduplication — selects the row with the most non-null metrics per(country_code, year)key
Quality checks (9 total): not_null, positive, composite unique, year range, row count floor, IDN full history, aggregate exclusion confirmation.
Country dimension table with peer group assignments. Separated from the fact table so regional groupings can be updated independently without touching KPI calculations.
Regional groups:
- ASEAN — 11 member states (including Indonesia)
- G20 (non-ASEAN) — 18 member economies (without Indonesia)
- Global — all other 202 economies with data
The Gold layer fact table — the analytical core of the entire project. Partitioned by year_date (DATE), clustered by country_code and region.
All computed columns and their SQL logic:
| Column | SQL Logic | Why |
|---|---|---|
year_date |
DATE(year, 1, 1) |
INT64 cannot be partitioned in BQ. DATE enables partition pruning from Looker date controls. |
gdp_yoy_growth_pct |
(gdp - LAG(gdp)) / LAG(gdp) * 100 |
Dimensionless growth rate — comparable across countries of any size |
co2_yoy_growth_pct |
(co2 - LAG(co2)) / LAG(co2) * 100 |
Same methodology as GDP for valid subtraction |
decoupling_index |
gdp_yoy_pct - co2_yoy_pct |
The core metric — positive = decoupling, negative = coupling |
gdp_index_2015 |
SAFE_DIVIDE(gdp, gdp_2015) * 100 |
Normalize to 2015=100 via MAX(CASE WHEN year=2015) OVER (PARTITION BY country) |
co2_index_2015 |
SAFE_DIVIDE(co2, co2_2015) * 100 |
Same — enables scatter plot comparison across countries |
rolling_3yr_gdp_per_co2 |
AVG(SAFE_DIVIDE(gdp, co2)) OVER (... ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) |
Smooths out single-year anomalies (COVID, financial crises) to reveal structural efficiency trend |
Quality checks (8 total): not_null on key columns, row count floor, IDN history completeness, 2015 base index validation (ABS(gdp_index_2015 - 100) < 0.01), aggregate code exclusion.
World-level view: who emits the most, who is decoupling, who leads the ranking.
| Tile | Type | Metric | Key Config |
|---|---|---|---|
| KPI Scorecards | Scorecard ×3 | GDP YoY %, CO2 YoY %, Renewables % | Average aggregation, Fixed filter: year = 2021 (IEA lag) for renewables share |
| Global CO2 Footprint | Geo Map | co2_emissions_mt |
Color scale: light to dark orange |
| Decoupling Scatter | Scatter chart | gdp_index_2015 vs co2_index_2015 |
Filter: year = 2023. Reference lines at X=100, Y=100. Bottom-right = absolute decoupling. |
| Decoupling Leaderboard | Table | decoupling_index |
Ranked descending. Conditional formatting: GDP Growth % (blue), CO2 Growth % (neutral), Decoupling Index (green) |
Indonesia-focused view with ASEAN peer benchmarking.
| Tile | Type | Metric | Key Config |
|---|---|---|---|
| KPI Scorecards | Scorecard ×3 | GDP YoY %, CO2 YoY %, Renewables % | Fixed filter: country_code = IDN, Fixed filter: year = 2021 (IEA lag) for renewables share |
| Annual Decoupling Trend | Combo chart | GDP bars (green) vs CO2 line (red) | Dual Y-axis. Reference line at Y=0. Fixed filter: country_code = IDN |
| Peer Pressure | Time series | rolling_3yr_gdp_per_co2 |
All ASEAN countries overlaid. IDN highlighted in amber. 3-year rolling smoothing. |
| Renewable Adoption Gap | Horizontal bar | renewable_energy_share |
Filter: year = 2021 (IEA lag). Reference line: ASEAN average. IDN highlighted via conditional formatting. |
Global controls (both pages): Date Range Control (year_date) · Region dropdown · Country Code dropdown
| Standard | Implementation |
|---|---|
| No credentials in source code | All secrets via env vars or .bruin.yml (both gitignored) |
| Least-privilege IAM | SA has only dataEditor + jobUser + objectAdmin — no Admin roles |
| No click-ops | Every GCP resource provisioned by Terraform. terraform apply = full environment |
| Immutable raw layer | force_destroy = false on GCS, delete_contents_on_destroy = false on all BQ datasets |
| Public access prevention | public_access_prevention = "enforced" on GCS bucket |
| Version-locked deps | All packages pinned in assets/requirements.txt — reproducible across machines |
| SAFE_DIVIDE everywhere | No division-by-zero crashes in production SQL |
| 17 quality checks total | Enforced by Bruin before any downstream asset runs |
Any reviewer can reproduce the entire platform in two commands after completing the one-time setup below.
- GCP project with billing enabled
- Terraform
>= 1.0installed - Bruin CLI
v0.11.524installed (winget install Bruin.Bruin) - Python
>= 3.13withuv
# 1. Clone the repo
git clone https://github.com/YOUR_USERNAME/ZC2026-net-zero-tracker.git
cd ZC2026-net-zero-tracker
# 2. Configure Terraform
cd terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars: set project_id, bucket_name
# 3. Configure environment
cd ..
cp .env.example .env
# Edit .env: set GCP_PROJECT_ID, GCS_BUCKET_NAME, GOOGLE_APPLICATION_CREDENTIALS# Command 1 — Provision all GCP infrastructure
cd terraform && terraform init && terraform apply && cd ..
# Generate & download SA key from GCP Console → IAM → Service Accounts → Keys
# Place at: credentials/YOUR-KEY.json
# Load env vars (PowerShell)
Get-Content .env | ForEach-Object {
if ($_ -match '^([^#=][^=]*)=(.*)$') {
[System.Environment]::SetEnvironmentVariable($matches[1].Trim(), $matches[2].Trim(), "Process")
}
}
# Command 2 — Run the full pipeline
bruin run . --start-date 2000-01-01 --end-date 2025-12-31Expected pipeline output:
[Bruin] ✓ raw.ingest_wb_data 6,497 rows → GCS + BQ
[Bruin] ✓ staging_energy_data.stg_energy_metrics 9 quality checks passed
[Bruin] ✓ marts_energy_data.dim_country 7 quality checks passed
[Bruin] ✓ marts_energy_data.fct_net_zero_performance 8 quality checks passed
Pipeline complete. 4/4 assets succeeded. 24 quality checks passed.
raw_energy_data.wb_indicators ← BQ External Table → GCS Parquet
country_code STRING
year INT64
gdp_ppp FLOAT64
co2_emissions_mt FLOAT64
renewable_energy_share FLOAT64
│
▼
staging_energy_data.stg_energy_metrics ← Materialized table, 9 quality checks
(same schema, typed + cleaned + aggregates removed)
│
├──────────────────────────────────┐
▼ ▼
marts_energy_data.dim_country marts_energy_data.fct_net_zero_performance
country_code STRING PK PARTITION BY year_date
region STRING CLUSTER BY country_code, region
is_focus_country BOOLEAN
year_date DATE
year INT64
country_code STRING
region STRING
is_focus_country BOOLEAN
gdp_ppp FLOAT64
co2_emissions_mt FLOAT64
renewable_energy_share FLOAT64
gdp_yoy_growth_pct FLOAT64
co2_yoy_growth_pct FLOAT64
decoupling_index FLOAT64
gdp_index_2015 FLOAT64
co2_index_2015 FLOAT64
rolling_3yr_gdp_per_co2 FLOAT64
| Criterion | Implementation | Location |
|---|---|---|
| ✅ Problem description | Decoupling Index — tracks economic growth vs. CO2 growth for 231 countries. Indonesia vs. ASEAN/G20 peers. 5 specific analytical questions answered in dashboard. | README + Dashboard |
| ✅ Cloud | GCP — GCS (data lake) + BigQuery (data warehouse) | Provisioned by terraform/ |
| ✅ Infrastructure as Code | Terraform provisions all GCP resources. Zero click-ops. terraform apply is the only command needed. |
terraform/ |
| ✅ Batch / Stream pipeline | Batch Pipeline: Bruin @monthly schedule. Python: API → GCS → BQ. SQL: 3 transformation assets with declared dependencies. Full lineage enforced. |
assets/ |
| ✅ Data warehouse | BigQuery. Fact table partitioned by year_date (DATE), clustered by country_code + region. Partition pruning confirmed active from Looker queries. |
assets/core/ |
| ✅ Transformations | 4-layer medallion: Raw → Staging → Dimension → Gold Fact. Bruin replaces Airflow + dbt + Great Expectations in a single tool. | assets/ |
| ✅ Dashboard | Looker Studio. 2 pages, 7 tiles, live BigQuery connector. Public link in README. | Looker / Data Studio |
| ✅ Reproducibility | Two commands from a fresh clone to full pipeline. All deps pinned. SA key pattern documented. Works on any machine with GCP credentials. | README |
| Data | Source | Indicator Code | Methodology |
|---|---|---|---|
| GDP, PPP (constant 2021 $) | World Bank WDI | NY.GDP.MKTP.PP.KD |
ICP/PPP, constant 2021 price base |
| CO2 Emissions excl. LULUCF | World Bank WDI | EN.GHG.CO2.MT.CE.AR5 |
EDGAR v8.0 (JRC/IEA), IPCC AR5 GWP |
| Renewable Electricity Share | World Bank WDI | EG.ELC.RNEW.ZS |
IEA Electricity Information (2-yr lag) |
- World Bank WDI December 2024 Update: Release Notes
- EDGAR v8.0 GHG Database: JRC European Commission
- IPCC AR5 GWP values: Fifth Assessment Report (2013)
-
AI-Assisted Content: Some materials in this repository, including code and documentation, were developed with the assistance of AI tools (Claude and Gemini). All AI-generated outputs have been manually reviewed, refined, and adjusted to ensure technical accuracy and alignment with project goals.
-
Educational Context: The structure and methodologies applied in this project were inspired by and built upon the curriculum of the DataTalksClub Data Engineering Zoomcamp.
- Course: Data Engineering Zoomcamp 2026
- Organization: DataTalksClub
- Repository Reference: Data Engineering Zoomcamp GitHub
- GCP Free Tier: This project was designed to fit within the Google Cloud Free Tier limits.
-
Copyright & Takedown: If you identify any content that infringes upon your intellectual property rights or was used without proper authorization, please contact me directly. I am committed to addressing such concerns and will remove the content promptly upon verification.
Wisnu Anugrah Pratama
Built as a capstone project for the Data Engineering Zoomcamp 2026 by DataTalks.Club.
Pipeline: Terraform · Python · SQL · Bruin · BigQuery · Looker Studio Data: World Bank API · 231 countries · 2000–2024