🛰️ Sentinel — Production ML Observability Platform

Sentinel is a production-grade, full-stack Machine Learning Observability Platform designed to monitor deployed models in real time. Sentinel tracks model inferences, detects statistical drift, conducts probability calibration, and computes explainability attributions—all visualised through an interactive live dashboard and monitored via a pre-provisioned Prometheus + Grafana stack.

🌟 Core Architecture & Features

1. Real-time Prediction Ingestion & Analytics

Ingest inferences asynchronously and monitor metrics as they flow through the system.
WebSocket support for real-time dashboard state synchronization.
Structured Gzip-compressed responses for large payloads (e.g., historical drift, detailed calibration curves).

2. Multi-Statistic Drift Engine

Population Stability Index (PSI): Catches overall feature distribution shifts compared to baseline data.
Kolmogorov-Smirnov (KS) Test: Detects continuous variable shifts.
Confidence-Weighted PSI: Weights features based on their model feature importance to focus attention on critical covariate shifts.
Adaptive EWMA Control Limits: Employs Exponentially Weighted Moving Averages to dynamically adjust alerting thresholds and prevent false positives.
Actionable Drift Type Classifier: Automatically categorises drift events (e.g., abrupt shifts, slow gradients) to guide resolution.

3. Time-Series STL Decomposition

Decomposes drift metrics into Trend, Seasonality, and Residuals.
Separates transient seasonal fluctuations from permanent model performance degradation.

4. Probability Calibration & Performance Monitoring

Compute probability calibration curves (Brier score, calibration slope) in real time.
Generate receiver operating characteristic (ROC) curves, compute Area Under the Curve (AUC), and find optimal decision thresholds using Youden's J statistic.

5. Explainable AI (SHAP)

Seamlessly integrates SHAP (SHapley Additive exPlanations) pipelines.
Displays global feature importances and lets you drill down to individual predictions for local explanation attributions.

6. Security & Telemetry

API Rate Limiting: Powered by a Redis-backed slowapi limiter protecting authentication (20 req/min), prediction ingestion (100 req/min), and other query endpoints (200 req/min).
Prometheus Scraper: Auto-instruments FastAPI requests while registering custom metrics (total predictions ingested, drift count, alert count, and alert resolution latency).
Grafana Dashboard: Out-of-the-box system dashboards displaying API request rates, database pool statistics, queue lengths, and model latency.

🏗️ System Architecture

flowchart TD
    %% Define Styles
    classDef client fill:#2c3e50,stroke:#f5f6fa,stroke-width:2px,color:#fff;
    classDef backend fill:#009688,stroke:#fff,stroke-width:2px,color:#fff;
    classDef storage fill:#2980b9,stroke:#fff,stroke-width:2px,color:#fff;
    classDef queue fill:#27ae60,stroke:#fff,stroke-width:2px,color:#fff;
    classDef monitor fill:#8e44ad,stroke:#fff,stroke-width:2px,color:#fff;

    subgraph Clients ["🌐 User & Client Layer"]
        ML_Model["🚀 Deployed ML Model<br/>(Inference Engine)"]
        Browser["🖥️ React Dashboard UI<br/>(Browser)"]
        Grafana_UI["📊 Grafana Dashboard<br/>(Browser)"]
    end

    subgraph Core_Backend ["⚙️ Sentinel Core Services"]
        API_Gateway["⚡ FastAPI Backend<br/>(REST & WebSockets)"]
        Rate_Limiter["🛡️ Rate Limiter<br/>(slowapi)"]
        Instrumentator["📈 Prometheus Instrumentator<br/>(FastAPI telemetry)"]
        
        API_Gateway --- Rate_Limiter
        API_Gateway --- Instrumentator
    end

    subgraph Async_Workers ["⏳ Background Processing"]
        RedisBroker["🔴 Redis Message Broker"]
        CeleryWorker["👷 Celery Worker Tasks<br/>(Drift, SHAP, Calibration)"]
        CeleryBeat["⏰ Celery Beat<br/>(Periodic Schedulers)"]
        
        RedisBroker === CeleryWorker
        CeleryBeat -.->|Triggers Tasks| RedisBroker
    end

    subgraph Storage_Layer ["💾 Storage & Cache Engine"]
        Postgres[("🐘 PostgreSQL DB<br/>(Baselines, Inferences, Drift, Alerts)")]
        RedisCache[("🔴 Redis Cache<br/>(Rate Limit Store & Active Sessions)")]
    end

    subgraph Observability ["📊 System & Custom Telemetry"]
        Prometheus[("🔥 Prometheus Server<br/>(Scrapes /metrics)")]
        Grafana["📈 Grafana Dashboards<br/>(Visualizes system telemetry)"]
    end

    %% Flows
    ML_Model -->|1. Ingest Predictions| API_Gateway
    Browser -->|2. Query Live Metrics & WS| API_Gateway
    
    API_Gateway -->|Write & Read Data| Postgres
    API_Gateway -->|Trigger Drift / SHAP Tasks| RedisBroker
    API_Gateway -->|Rate Limiting Checks| RedisCache
    
    CeleryWorker -->|Update Metrics & Alerts| Postgres
    CeleryWorker -->|Pull Baseline Models| Postgres
    
    Prometheus -->|Scrape HTTP & Custom Metrics| API_Gateway
    Grafana -->|Query Prometheus Metrics| Prometheus
    Grafana_UI -->|View System Dashboards| Grafana

    %% Apply Classes
    class ML_Model,Browser,Grafana_UI client;
    class API_Gateway,Rate_Limiter,Instrumentator backend;
    class Postgres,RedisCache storage;
    class RedisBroker,CeleryWorker,CeleryBeat queue;
    class Prometheus,Grafana monitor;

🎨 Premium UI/UX Design

The dashboard is custom-styled with a rich, dark-mode visual aesthetic:

Glassmorphic Cards: Translucent slate components overlaying a deep space background gradient with glowing ambient blur spots.
60px Rail Sidebar: Space-saving navigation icon rail featuring dynamic flyout drawers that expand smoothly on hover or click.
Connection Heartbeat Pulse: An ambient glowing left-border status pulse that turns green when the WebSocket real-time connection is active, and flashes warning-red if the connection is lost.
Vibrant Interactive Charts: Recharts-based responsive components featuring customized tooltips, Youden's J markers, and smooth hover state micro-animations.

🛠️ Service Map & Default Credentials

Service	Address	Credentials (Default)	Description
Sentinel Dashboard UI	http://localhost:3000	`admin@sentinel.dev` / `sentinel`	Primary dashboard for ML observability
Sentinel Backend API	http://localhost:8000	Bearer Token Auth	FastAPI backend, OpenAPI spec at `/docs`
Grafana Dashboard	http://localhost:3001	`admin` / `sentinel`	Core infrastructure & telemetry dashboard
Prometheus	http://localhost:9090	No Auth	Raw metrics repository & scrapers
PostgreSQL	`localhost:5433` (host)	`sentinel` / `sentinel` (db: `sentinel`)	Persistent inferences, baseline models & drift data
Redis	`localhost:6379` (host)	No Auth	Message broker, cache, and rate limit database

🚀 Getting Started

Option A: Run via Docker Compose (Recommended)

This fires up the database, cache, backend api, background worker, scheduler, prometheus, grafana, and frontend UI in a single command.

Navigate to the infrastructure folder:
```
cd infra
```
Spin up the containers:
```
docker-compose up -d
```
Wait ~30 seconds for postgres to apply database migrations and seed baseline models.
Access the dashboard at http://localhost:3000 or view telemetry at http://localhost:3001.

Option B: Run Manually (Local Development)

1. Backend & Worker Setup

Navigate to the backend directory and set up a virtual environment:

cd backend
python -m venv .venv
.\.venv\Scripts\activate  # Windows
# or: source .venv/bin/activate  # macOS/Linux

Install dependencies in editable mode:
```
pip install -e ".[dev]"
```
Run migrations to initialize PostgreSQL schema:
```
alembic upgrade head
```
Seed database with baseline models:
```
python scripts/seed_db.py
```

Run the FastAPI development server:

python -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

(In a separate terminal) Start the Celery Worker tasks:

celery -A app.tasks.celery_app worker --loglevel=info --concurrency=4 -Q default,drift,shap

(In a separate terminal) Start the Celery Beat scheduler:
```
celery -A app.tasks.celery_app beat --loglevel=info
```

2. Frontend Setup

Navigate to the frontend directory and install dependencies:
```
cd ../frontend
npm install
```
Start the Vite development server:
```
npm run dev
```
Dashboard will be available at http://localhost:3000.

3. Live Simulators

Generate artificial predictions and simulate feature/concept drift in real time:

cd backend
python scripts/generate_real_predictions.py

📁 Repository Structure

Sentinel/
├── backend/            # FastAPI Backend API, Celery Tasks, & Database migrations
│   ├── app/            # Main application source code (routers, core metrics/rate limits, models, middleware)
│   ├── scripts/        # Database seeding, reset, and live prediction simulators
│   └── tests/          # Unit & integration tests for rate limiting, drift, and calibration
├── frontend/           # React, TypeScript, and Vite Dashboard
│   ├── src/            # Charting components, Zustand store, pages, layout with rail sidebar
│   └── nginx.conf      # Production Nginx reverse proxy configuration
├── ml/                 # Machine learning models, schemas, and training pipelines
│   └── models/         # Pre-trained XGBoost & Scikit-learn models (1, 2, 3)
├── infra/              # Docker Compose environment and configuration
│   ├── prometheus/     # Prometheus scraping configurations
│   └── grafana/        # Auto-provisioned Datasources and preloaded Sentinel dashboard
└── docs/               # System documentation, design reports & ADRs

✅ System Verification

Check that all system components are communicating correctly:

# 1. Verify backend health and version
curl http://localhost:8000/health
# Expected: {"status": "ok", "version": "0.1.0"}

# 2. Verify Prometheus endpoint is exposing API & custom ML metrics
curl http://localhost:8000/metrics | grep sentinel_
# Expected: Displays active sentinel_predictions_ingested_total, sentinel_drift_events_total, etc.

# 3. Verify PostgreSQL contains initialized tables
psql postgresql://sentinel:sentinel@localhost:5432/sentinel -c "\dt"

# 4. Verify baseline models are properly seeded
psql postgresql://sentinel:sentinel@localhost:5432/sentinel -c "SELECT id, name, version FROM model_registry;"

📖 Reference Documentation

🚀 Quick Start Reference Card — Command cheat-sheet & port references.
⚙️ Local Setup & Configuration Guide — Detailed environment variables and DB configuration.
🖥️ Running Dashboard Instructions — UI walkthrough and metrics explanation.
📝 Work Summary Log — System issue diagnosis, fixes, and sprint progress tracking.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

🛰️ Sentinel — Production ML Observability Platform

🌟 Core Architecture & Features

1. Real-time Prediction Ingestion & Analytics

2. Multi-Statistic Drift Engine

3. Time-Series STL Decomposition

4. Probability Calibration & Performance Monitoring

5. Explainable AI (SHAP)

6. Security & Telemetry

🏗️ System Architecture

🎨 Premium UI/UX Design

🛠️ Service Map & Default Credentials

🚀 Getting Started

Option A: Run via Docker Compose (Recommended)

Option B: Run Manually (Local Development)

1. Backend & Worker Setup

2. Frontend Setup

3. Live Simulators

📁 Repository Structure

✅ System Verification

📖 Reference Documentation

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

🛰️ Sentinel — Production ML Observability Platform

🌟 Core Architecture & Features

1. Real-time Prediction Ingestion & Analytics

2. Multi-Statistic Drift Engine

3. Time-Series STL Decomposition

4. Probability Calibration & Performance Monitoring

5. Explainable AI (SHAP)

6. Security & Telemetry

🏗️ System Architecture

🎨 Premium UI/UX Design

🛠️ Service Map & Default Credentials

🚀 Getting Started

Option A: Run via Docker Compose (Recommended)

Option B: Run Manually (Local Development)

1. Backend & Worker Setup

2. Frontend Setup

3. Live Simulators

📁 Repository Structure

✅ System Verification

📖 Reference Documentation