ETL Lab 4: Data Quality Validation with Great Expectations

Project Overview

This project implements a complete ETL pipeline for a retail company. It focuses on data quality validation using Great Expectations, data cleaning, transformation, and dimensional modeling (Star Schema) via pandas and SQLite.

Project Structure

• data/raw/: Original source data (retail_etl_dataset.csv) - Never modified.
• data/processed/: Cleaned data, transformed data, and the final SQLite Data Warehouse (data_warehouse.db).
• src/: Python scripts executing each distinct ETL stage (extract.py, validate_input.py, quality_analysis.py, etc).
• dashboard.py: Streamlit dashboard connected to the final SQLite warehouse.
• reports/: Final KPI PNG visualizations used in the report deliverable.
• logs/: Runtime logs generated by pipeline executions (pipeline.log).
• gx/: Great Expectations suites used for raw-input and cleaned-output validation.

Setup Instructions (using uv)

This project uses uv for lightning-fast dependency management via pyproject.toml.

1. Install uv (if not already installed): Follow the uv installation guide.
2. Place Raw Data: Ensure you place the original retail_etl_dataset.csv file inside the data/raw/ directory before running the pipeline.
3. Sync Dependencies: Navigate to the project root and run to synchronize environment packages:

   uv sync

4. Initialize Great Expectations: (Optional, handled within scripts, but you can build docs anytime):

   uv run great_expectations docs build

Running the Pipeline

You can run the entire sequence end-to-end using the main orchestrator:

uv run python src/main.py

Pipeline Stages Breakdown

If executing individually, maintain the strict order below to guarantee referential integrity:

1. Extract & Profiling: uv run python src/extract.py
2. Input GE Validation: uv run python src/validate_input.py
3. Quality Analysis: uv run python src/quality_analysis.py
4. Clean: uv run python src/clean.py
5. Transform: uv run python src/transform.py
6. Output GE Validation: uv run python src/validate_output.py
7. Dimensional Model: uv run python src/dimensional_model.py
8. Load DW: uv run python src/load_dw.py
9. Analysis KPIs: uv run python src/analysis.py

Running the Dashboard

After the ETL pipeline has created data/processed/data_warehouse.db, you can launch the interactive dashboard with:

uv run streamlit run dashboard.py

Then open the local URL shown by Streamlit in your browser, usually:

http://localhost:8501