Cloud Data Governance Engineering Lab

A technical data governance engineering lab showing how metadata, lineage, data quality, access controls, stewardship workflows, and audit evidence can be embedded into a cloud-style analytics platform.

This project is built as a working governance engineering example. It generates synthetic regulated-sector data, runs a bronze/silver/gold pipeline, applies data quality controls, creates governance metadata artifacts, exports audit evidence, and visualizes governance risk through a Streamlit dashboard.

Executive summary

Many organizations have data governance policies. The harder problem is turning those policies into working controls inside the data platform.

This lab shows how governance can be operationalized through code, metadata, data quality rules, ownership models, access controls, lineage, and audit-ready evidence.

The project is designed for technical data governance, data platform, data quality, metadata, lineage, risk-control, and analytics engineering roles in regulated environments such as banking, insurance, pensions, utilities, public sector, and infrastructure.

What this project proves

This repo demonstrates the ability to:

• generate realistic synthetic regulated-sector data
• build a bronze, silver, and gold analytics pipeline
• detect and report data quality issues
• promote validated data into governed outputs
• simulate a data catalog and business glossary
• classify data assets by sensitivity and PII exposure
• document lineage from raw files to gold marts
• define data contracts for critical source assets
• assign data owners, stewards, custodians, and escalation paths
• apply role-based and classification-based access rules
• export audit evidence for governance review
• run a Streamlit dashboard for quality, risk, catalog, lineage, access, and stewardship monitoring
• validate the build through GitHub Actions CI

Current build outputs

Output	Location
Synthetic raw customer, account, transaction, and service request data	data/raw/
Gold customer 360 mart	data/processed/gold_customer_360.csv
Gold governance risk summary	data/processed/gold_governance_risk_summary.csv
Data quality results	reports/evidence/data_quality_results.csv
Pipeline audit evidence report	reports/evidence/pipeline_audit_evidence.md
Data catalog	metadata/data_catalog.csv
Business glossary	metadata/business_glossary.csv
Data classification policy	metadata/data_classification_policy.csv
Access control matrix	metadata/access_control_matrix.csv
Stewardship model	metadata/stewardship_model.csv
Data lineage map	metadata/data_lineage.csv
Data contracts	metadata/data_contracts.csv
Streamlit dashboard	app/streamlit_app.py
CI workflow	.github/workflows/ci.yml

Architecture

Raw synthetic data flows through a governed analytics pipeline:

1. Raw regulated-sector data is generated.
2. Bronze tables preserve source-like records.
3. Silver tables apply validation, standardization, duplicate handling, and referential-integrity controls.
4. Gold tables produce governance-ready customer and risk outputs.
5. Evidence files document data quality results and pipeline execution.
6. The Streamlit dashboard makes quality, risk, lineage, ownership, access, and stewardship visible.

Governance controls included

Control area	What is implemented
Data quality	Completeness, uniqueness, validity, referential integrity, and business-rule checks
Metadata	Data catalog, ownership, classification, PII flag, retention rule, and source-system documentation
Business glossary	Approved terms, definitions, owners, and critical data element flags
Lineage	Raw-to-bronze, bronze-to-silver, and silver-to-gold transformation mapping
Access governance	Role-based and classification-based access control matrix
Stewardship	Data owner, steward, technical custodian, and escalation-path assignments
Data contracts	Required fields, primary keys, freshness expectations, quality rules, and contract owners
Audit evidence	Pipeline evidence report and data quality results export
CI/CD	GitHub Actions validates code, regenerates data, rebuilds metadata, runs the pipeline, and confirms outputs

Data quality approach

The raw data intentionally includes data quality issues. This is deliberate.

The purpose of the lab is to show that the pipeline can detect, report, and control issues before data is promoted into governed outputs.

Example checks include:

• customer ID uniqueness
• missing customer email detection
• approved province validation
• approved risk segment validation
• account-to-customer referential integrity
• non-negative account balance validation
• transaction-to-account referential integrity
• approved transaction type validation
• service request priority validation

Dashboard

The Streamlit dashboard includes:

• executive governance control panel
• data quality check results
• quality results by dimension
• governance risk summary
• catalog and glossary coverage
• lineage records
• access control rules
• stewardship assignments
• data contracts

To run the dashboard locally:

1. Activate the virtual environment: .venv\Scripts\activate
2. Run the dashboard: python -m streamlit run app/streamlit_app.py
3. Open the local browser URL shown by Streamlit.

A live dashboard link can be added here after deployment to Streamlit Community Cloud.

How to run locally

Create and activate a virtual environment:

python -m venv .venv

.venv\Scripts\activate

Install dependencies:

python -m pip install --upgrade pip

python -m pip install -r requirements.txt

Generate synthetic data:

python src/generate_synthetic_data.py

Create governance metadata:

python src/create_governance_metadata.py

Build the governed pipeline:

python src/build_pipeline.py

Run the dashboard:

python -m streamlit run app/streamlit_app.py

Tech stack

• Python
• SQL
• DuckDB
• pandas
• Streamlit
• Plotly
• GitHub Actions
• pytest

Target role alignment

This project is designed to demonstrate practical capability for roles such as:

• Data Governance Engineer
• Technical Data Governance Specialist
• Senior Data Platform Governance Specialist
• Data Quality Engineer
• Metadata and Lineage Engineer
• Data Platform Engineer with governance focus
• Data Risk and Controls Analyst
• Analytics Engineer with governance responsibilities

Project status

The first working version is complete.

Completed:

• synthetic data generation
• bronze/silver/gold pipeline
• data quality checks
• metadata artifacts
• lineage map
• access control matrix
• stewardship model
• data contracts
• audit evidence exports
• Streamlit dashboard
• GitHub Actions CI build

Planned next improvements:

• dashboard screenshots in the README
• Streamlit Cloud deployment
• additional data quality test coverage
• richer lineage visualization
• optional dbt-style model layer
• optional OpenMetadata or DataHub integration

Informatica-Style ETL Skills Layer

This repo includes an Informatica-style ETL documentation layer to show transferable Informatica analyst skills without claiming direct access to the proprietary Informatica platform.

Included artifacts:

• docs/informatica-style-etl/README.md
• docs/informatica-style-etl/source_to_target_mapping.md
• docs/informatica-style-etl/etl_mapping_design.md
• docs/informatica-style-etl/data_quality_validation_matrix.md
• metadata/informatica_style_lineage_matrix.csv
• dq_rules/informatica_style_dq_rules.yml
• sql/informatica_style_validation_queries.sql

This section demonstrates source-to-target mapping, ETL transformation logic, lookup validation, router-style exception handling, SQL reconciliation, metadata documentation, lineage, and data quality rules.