📄 Research Publication

This project is backed by a peer-reviewed research paper accepted for presentation and publication at ASIACON 2026 (Asian Conference on Innovation in Technology).

📌 Paper Title DataSonar: A Real-Time Data Pipeline Quality Monitoring & Anomaly Detection Platform

🏛️ Conference ASIACON 2026 – Asian Conference on Innovation in Technology

📖 Research Paper 👉 https://drive.google.com/file/d/1qU0DQiFB3Nt7UA5GDso4TG7aDrOikPoD/view?usp=drive_link

This publication presents the architecture, design decisions, and implementation of DataSonar, a distributed platform for real-time data quality monitoring, pipeline observability, and anomaly detection.

Citation

DataSonar: A Real-Time Data Pipeline Quality Monitoring & Anomaly Detection Platform.
Accepted for presentation and publication at ASIACON 2026
(Asian Conference on Innovation in Technology).

🔍 DataSonar

Real-time data pipeline quality monitoring & anomaly detection platform

🎯 Problem

Bad data costs companies an average of $12.9 million annually (Gartner). Data pipeline failures go undetected for hours, there's no real-time visibility into data quality, and compliance suffers from data integrity issues.

💡 Solution

DataSonar monitors data pipelines in real-time, detects anomalies using ML, and provides instant alerts with root cause analysis.

Key Features

• Real-time Ingestion — Stream pipeline metadata via Kafka
• 6-Dimension Quality Scoring — Completeness, accuracy, consistency, timeliness, validity, uniqueness
• ML Anomaly Detection — Isolation Forest, Prophet, Z-score, DBSCAN
• Instant Alerts — RabbitMQ-powered notifications with deduplication
• Interactive Dashboard — Next.js real-time monitoring UI
• Full Observability — Prometheus, Grafana, ELK Stack

🏗️ Architecture

[Data Sources] → [Kafka] → [Quality Engine] → [Anomaly Detector]
                                                       ↓
[Dashboard] ← [Dashboard API] ← [Alert Service] ← [RabbitMQ]

🛠️ Tech Stack

Layer	Technologies
Frontend	Next.js, React, TailwindCSS, Recharts
API Gateway	Node.js, Express, JWT, OAuth2, Prisma
Streaming	Apache Kafka (KafkaJS), RabbitMQ
ML/Quality	Python, FastAPI, scikit-learn, Prophet, Pandas
Databases	PostgreSQL, MongoDB, Redis, Aerospike
Storage	MinIO (S3-compatible)
Observability	Prometheus, Grafana, ELK Stack
DevOps	Docker, Kubernetes, GitHub Actions

📦 Project Structure

datasonar/
├── services/
│   ├── gateway/            # API Gateway (Node.js + Express + Prisma)
│   ├── ingestion/          # Data Ingestion (Node.js + KafkaJS)
│   ├── quality-engine/     # Quality Scoring (Python + FastAPI)
│   ├── anomaly-detector/   # ML Detection (Python + FastAPI)
│   ├── alert-service/      # Alerting (Node.js + RabbitMQ)
│   └── dashboard-api/      # Dashboard Backend (Node.js + Express)
├── frontend/               # Dashboard UI (Next.js)
├── infra/                  # Kubernetes & Terraform configs
├── monitoring/             # Prometheus, Grafana, ELK configs
├── docs/                   # Architecture docs & ADRs
└── scripts/                # Utility scripts

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Node.js 20+
• Python 3.11+
• Git

Run Locally

# Clone the repository
git clone https://github.com/om-singh-D/DataSonar.git
cd DataSonar

# Start all infrastructure services
docker-compose up -d

# Verify everything is running
docker-compose ps

Start Full App (PowerShell)

Use the launcher script to start Docker services, check API health, open the dashboard, and run the frontend dev server.

powershell -ExecutionPolicy Bypass -File .\start-datasonar.ps1

📅 Development Roadmap

• Project setup & folder structure
• Docker Compose infrastructure
• Ingestion Service (Kafka)
• Gateway Service (Auth + API)
• Quality Engine (FastAPI)
• Anomaly Detection (ML)
• Alert Service (RabbitMQ)
• Dashboard (Next.js)
• Observability (Prometheus + ELK)
• Kubernetes deployment

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

👤 Author

Om Singh — @om-singh-D