Skip to content

About

End-to-end data quality & observability platform monitoring a retail pipeline for nulls, duplicates, schema drift, volume anomalies, and downstream lineage impact — built with Python, Pandas, SQLite, Streamlit, and Docker.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

📡 DataPulse AI

Data Quality & Observability Platform

An end-to-end data observability platform for monitoring retail data pipelines

Monitor data quality, detect volume anomalies, track schema drift, trace dataset lineage, and generate incident reports — all through a rich interactive dashboard.


Python Streamlit Plotly SQLite Docker Pandas


Quality Score Records Checks CI


📸 Dashboard Screenshots

Overview — Pipeline Health

Overview


Data Quality Checks

Data Quality


Anomaly Detection

Anomalies


Schema Drift

Schema Drift


Dataset Lineage

Lineage


Incident Report

Incident Report


🧠 What is DataPulse AI?

DataPulse AI is a data observability platform that monitors a synthetic retail data pipeline end-to-end. It ingests raw CSVs (customers, orders, products, returns), loads them into a SQLite warehouse, and runs automated checks across 7 quality dimensions — surfacing failures, anomalies, and drift in a rich Streamlit dashboard.

The platform ships with deliberately injected failures in orders_bad.csv (nulls, negatives, bad dates, duplicates) to demonstrate real-world observability scenarios.


⚡ Key Features

Feature Description
Quality Checks Null detection, uniqueness, range validation, date parsing, required columns
Volume Anomaly Detection Compares row counts against baseline — flags deviations beyond threshold
Schema Drift Detection Detects added, removed, or type-changed columns across pipeline runs
Dataset Lineage Traces data flow from raw CSVs → warehouse tables → downstream outputs
Incident Reports Auto-generated summaries with root cause and remediation steps
Interactive Dashboard 6-tab Streamlit UI with Plotly bar, pie, line, and Sankey charts
Synthetic Demo Data 500 customers, 5000 orders, 100 products, ~400 returns with injected failures

📊 Quality Rules

Rule Column Description
Null check customer_id Must not be null
Uniqueness order_id Must be unique
Range check quantity Must be > 0
Range check price Must be >= 0
Date check order_date Must be a valid date
Volume check All tables Row count vs baseline
Schema check All tables Expected columns/types vs observed

🏗️ Architecture

Raw Sources (CSV)
      |
      v
 Ingestion Layer
      |
      v
SQLite Warehouse ──── fact_orders
      |                    |
      v                    v
Quality Checks      monthly_revenue
      |
      v
Anomaly Detection ──── Volume / Schema Drift
      |
      v
Lineage Tracker ──── Downstream Impact
      |
      v
Incident Reporter
      |
      v
Streamlit Dashboard (6 tabs)

Lineage Map

customers.csv  --> dim_customer  ──\
products.csv   --> dim_product   ───+--> fact_orders --> monthly_revenue --> executive_dashboard
orders.csv     --> fact_orders   ──/
returns.csv    --> return_metrics

📂 Repository Structure

DataPulse-AI/
│
├── datapulse/              # Core modules
│   ├── config.py           # Paths and baseline config
│   ├── generate_data.py    # Synthetic data generator
│   ├── database.py         # SQLite warehouse + fact tables
│   ├── quality.py          # Quality check engine
│   ├── anomalies.py        # Volume anomaly detection
│   ├── schema.py           # Schema drift detection
│   ├── lineage.py          # Dataset lineage tracker
│   ├── incidents.py        # Incident report generator
│   └── pipeline.py         # Full pipeline orchestrator
│
├── data/raw/               # Source CSVs (500 customers, 5000 orders, etc.)
├── data/warehouse/         # SQLite DB + monitoring report
├── screenshots/            # Dashboard screenshots
├── tests/                  # pytest test suite
├── app.py                  # Streamlit dashboard
├── docker-compose.yml
├── Dockerfile
└── requirements.txt

🚀 Quick Start

# 1. Clone and set up environment
git clone https://github.com/Adityaindurkhya/DataPulseAI.git
cd DataPulse-AI
python -m venv .venv

# Windows
.venv\Scripts\Activate.ps1

pip install -r requirements.txt

# 2. Generate synthetic data
python -m datapulse.generate_data

# 3. Run the pipeline
python -m datapulse.pipeline

# 4. Launch the dashboard
streamlit run app.py

Open: http://localhost:8501


🧪 Run Tests

pytest -q

🐳 Docker

docker compose up --build

Open: http://localhost:8501


🛠️ Tech Stack

Category Technologies
Dashboard Streamlit, Plotly (bar, pie, line, Sankey)
Data Processing Pandas, NumPy
Warehouse SQLite
Quality Engine Custom rule-based checks
Testing pytest
CI/CD GitHub Actions
Infra Docker, Docker Compose

👨‍💻 Author

Aditya Indurkhya

Master of Science in Data Analytics Engineering | B.Tech in Computer Science Engineering

GitHub

About

End-to-end data quality & observability platform monitoring a retail pipeline for nulls, duplicates, schema drift, volume anomalies, and downstream lineage impact — built with Python, Pandas, SQLite, Streamlit, and Docker.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages