An end-to-end data observability platform for monitoring retail data pipelines
Monitor data quality, detect volume anomalies, track schema drift, trace dataset lineage, and generate incident reports — all through a rich interactive dashboard.
DataPulse AI is a data observability platform that monitors a synthetic retail data pipeline end-to-end. It ingests raw CSVs (customers, orders, products, returns), loads them into a SQLite warehouse, and runs automated checks across 7 quality dimensions — surfacing failures, anomalies, and drift in a rich Streamlit dashboard.
The platform ships with deliberately injected failures in orders_bad.csv (nulls, negatives, bad dates, duplicates) to demonstrate real-world observability scenarios.
| Feature | Description |
|---|---|
| Quality Checks | Null detection, uniqueness, range validation, date parsing, required columns |
| Volume Anomaly Detection | Compares row counts against baseline — flags deviations beyond threshold |
| Schema Drift Detection | Detects added, removed, or type-changed columns across pipeline runs |
| Dataset Lineage | Traces data flow from raw CSVs → warehouse tables → downstream outputs |
| Incident Reports | Auto-generated summaries with root cause and remediation steps |
| Interactive Dashboard | 6-tab Streamlit UI with Plotly bar, pie, line, and Sankey charts |
| Synthetic Demo Data | 500 customers, 5000 orders, 100 products, ~400 returns with injected failures |
| Rule | Column | Description |
|---|---|---|
| Null check | customer_id |
Must not be null |
| Uniqueness | order_id |
Must be unique |
| Range check | quantity |
Must be > 0 |
| Range check | price |
Must be >= 0 |
| Date check | order_date |
Must be a valid date |
| Volume check | All tables | Row count vs baseline |
| Schema check | All tables | Expected columns/types vs observed |
Raw Sources (CSV)
|
v
Ingestion Layer
|
v
SQLite Warehouse ──── fact_orders
| |
v v
Quality Checks monthly_revenue
|
v
Anomaly Detection ──── Volume / Schema Drift
|
v
Lineage Tracker ──── Downstream Impact
|
v
Incident Reporter
|
v
Streamlit Dashboard (6 tabs)
customers.csv --> dim_customer ──\
products.csv --> dim_product ───+--> fact_orders --> monthly_revenue --> executive_dashboard
orders.csv --> fact_orders ──/
returns.csv --> return_metrics
DataPulse-AI/
│
├── datapulse/ # Core modules
│ ├── config.py # Paths and baseline config
│ ├── generate_data.py # Synthetic data generator
│ ├── database.py # SQLite warehouse + fact tables
│ ├── quality.py # Quality check engine
│ ├── anomalies.py # Volume anomaly detection
│ ├── schema.py # Schema drift detection
│ ├── lineage.py # Dataset lineage tracker
│ ├── incidents.py # Incident report generator
│ └── pipeline.py # Full pipeline orchestrator
│
├── data/raw/ # Source CSVs (500 customers, 5000 orders, etc.)
├── data/warehouse/ # SQLite DB + monitoring report
├── screenshots/ # Dashboard screenshots
├── tests/ # pytest test suite
├── app.py # Streamlit dashboard
├── docker-compose.yml
├── Dockerfile
└── requirements.txt
# 1. Clone and set up environment
git clone https://github.com/Adityaindurkhya/DataPulseAI.git
cd DataPulse-AI
python -m venv .venv
# Windows
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
# 2. Generate synthetic data
python -m datapulse.generate_data
# 3. Run the pipeline
python -m datapulse.pipeline
# 4. Launch the dashboard
streamlit run app.pyOpen: http://localhost:8501
pytest -qdocker compose up --buildOpen: http://localhost:8501
| Category | Technologies |
|---|---|
| Dashboard | Streamlit, Plotly (bar, pie, line, Sankey) |
| Data Processing | Pandas, NumPy |
| Warehouse | SQLite |
| Quality Engine | Custom rule-based checks |
| Testing | pytest |
| CI/CD | GitHub Actions |
| Infra | Docker, Docker Compose |
Aditya Indurkhya
Master of Science in Data Analytics Engineering | B.Tech in Computer Science Engineering





