Sentinel is a production-grade, full-stack Machine Learning Observability Platform designed to monitor deployed models in real time. Sentinel tracks model inferences, detects statistical drift, conducts probability calibration, and computes explainability attributions—all visualised through an interactive live dashboard and monitored via a pre-provisioned Prometheus + Grafana stack.
- Ingest inferences asynchronously and monitor metrics as they flow through the system.
- WebSocket support for real-time dashboard state synchronization.
- Structured Gzip-compressed responses for large payloads (e.g., historical drift, detailed calibration curves).
- Population Stability Index (PSI): Catches overall feature distribution shifts compared to baseline data.
- Kolmogorov-Smirnov (KS) Test: Detects continuous variable shifts.
- Confidence-Weighted PSI: Weights features based on their model feature importance to focus attention on critical covariate shifts.
- Adaptive EWMA Control Limits: Employs Exponentially Weighted Moving Averages to dynamically adjust alerting thresholds and prevent false positives.
- Actionable Drift Type Classifier: Automatically categorises drift events (e.g., abrupt shifts, slow gradients) to guide resolution.
- Decomposes drift metrics into Trend, Seasonality, and Residuals.
- Separates transient seasonal fluctuations from permanent model performance degradation.
- Compute probability calibration curves (Brier score, calibration slope) in real time.
- Generate receiver operating characteristic (ROC) curves, compute Area Under the Curve (AUC), and find optimal decision thresholds using Youden's J statistic.
- Seamlessly integrates SHAP (SHapley Additive exPlanations) pipelines.
- Displays global feature importances and lets you drill down to individual predictions for local explanation attributions.
- API Rate Limiting: Powered by a Redis-backed
slowapilimiter protecting authentication (20 req/min), prediction ingestion (100 req/min), and other query endpoints (200 req/min). - Prometheus Scraper: Auto-instruments FastAPI requests while registering custom metrics (total predictions ingested, drift count, alert count, and alert resolution latency).
- Grafana Dashboard: Out-of-the-box system dashboards displaying API request rates, database pool statistics, queue lengths, and model latency.
flowchart TD
%% Define Styles
classDef client fill:#2c3e50,stroke:#f5f6fa,stroke-width:2px,color:#fff;
classDef backend fill:#009688,stroke:#fff,stroke-width:2px,color:#fff;
classDef storage fill:#2980b9,stroke:#fff,stroke-width:2px,color:#fff;
classDef queue fill:#27ae60,stroke:#fff,stroke-width:2px,color:#fff;
classDef monitor fill:#8e44ad,stroke:#fff,stroke-width:2px,color:#fff;
subgraph Clients ["🌐 User & Client Layer"]
ML_Model["🚀 Deployed ML Model<br/>(Inference Engine)"]
Browser["🖥️ React Dashboard UI<br/>(Browser)"]
Grafana_UI["📊 Grafana Dashboard<br/>(Browser)"]
end
subgraph Core_Backend ["⚙️ Sentinel Core Services"]
API_Gateway["⚡ FastAPI Backend<br/>(REST & WebSockets)"]
Rate_Limiter["🛡️ Rate Limiter<br/>(slowapi)"]
Instrumentator["📈 Prometheus Instrumentator<br/>(FastAPI telemetry)"]
API_Gateway --- Rate_Limiter
API_Gateway --- Instrumentator
end
subgraph Async_Workers ["⏳ Background Processing"]
RedisBroker["🔴 Redis Message Broker"]
CeleryWorker["👷 Celery Worker Tasks<br/>(Drift, SHAP, Calibration)"]
CeleryBeat["⏰ Celery Beat<br/>(Periodic Schedulers)"]
RedisBroker === CeleryWorker
CeleryBeat -.->|Triggers Tasks| RedisBroker
end
subgraph Storage_Layer ["💾 Storage & Cache Engine"]
Postgres[("🐘 PostgreSQL DB<br/>(Baselines, Inferences, Drift, Alerts)")]
RedisCache[("🔴 Redis Cache<br/>(Rate Limit Store & Active Sessions)")]
end
subgraph Observability ["📊 System & Custom Telemetry"]
Prometheus[("🔥 Prometheus Server<br/>(Scrapes /metrics)")]
Grafana["📈 Grafana Dashboards<br/>(Visualizes system telemetry)"]
end
%% Flows
ML_Model -->|1. Ingest Predictions| API_Gateway
Browser -->|2. Query Live Metrics & WS| API_Gateway
API_Gateway -->|Write & Read Data| Postgres
API_Gateway -->|Trigger Drift / SHAP Tasks| RedisBroker
API_Gateway -->|Rate Limiting Checks| RedisCache
CeleryWorker -->|Update Metrics & Alerts| Postgres
CeleryWorker -->|Pull Baseline Models| Postgres
Prometheus -->|Scrape HTTP & Custom Metrics| API_Gateway
Grafana -->|Query Prometheus Metrics| Prometheus
Grafana_UI -->|View System Dashboards| Grafana
%% Apply Classes
class ML_Model,Browser,Grafana_UI client;
class API_Gateway,Rate_Limiter,Instrumentator backend;
class Postgres,RedisCache storage;
class RedisBroker,CeleryWorker,CeleryBeat queue;
class Prometheus,Grafana monitor;
The dashboard is custom-styled with a rich, dark-mode visual aesthetic:
- Glassmorphic Cards: Translucent slate components overlaying a deep space background gradient with glowing ambient blur spots.
- 60px Rail Sidebar: Space-saving navigation icon rail featuring dynamic flyout drawers that expand smoothly on hover or click.
- Connection Heartbeat Pulse: An ambient glowing left-border status pulse that turns green when the WebSocket real-time connection is active, and flashes warning-red if the connection is lost.
- Vibrant Interactive Charts: Recharts-based responsive components featuring customized tooltips, Youden's J markers, and smooth hover state micro-animations.
| Service | Address | Credentials (Default) | Description |
|---|---|---|---|
| Sentinel Dashboard UI | http://localhost:3000 | admin@sentinel.dev / sentinel |
Primary dashboard for ML observability |
| Sentinel Backend API | http://localhost:8000 | Bearer Token Auth | FastAPI backend, OpenAPI spec at /docs |
| Grafana Dashboard | http://localhost:3001 | admin / sentinel |
Core infrastructure & telemetry dashboard |
| Prometheus | http://localhost:9090 | No Auth | Raw metrics repository & scrapers |
| PostgreSQL | localhost:5433 (host) |
sentinel / sentinel (db: sentinel) |
Persistent inferences, baseline models & drift data |
| Redis | localhost:6379 (host) |
No Auth | Message broker, cache, and rate limit database |
This fires up the database, cache, backend api, background worker, scheduler, prometheus, grafana, and frontend UI in a single command.
- Navigate to the infrastructure folder:
cd infra - Spin up the containers:
docker-compose up -d
- Wait ~30 seconds for postgres to apply database migrations and seed baseline models.
- Access the dashboard at
http://localhost:3000or view telemetry athttp://localhost:3001.
- Navigate to the backend directory and set up a virtual environment:
cd backend python -m venv .venv .\.venv\Scripts\activate # Windows # or: source .venv/bin/activate # macOS/Linux
- Install dependencies in editable mode:
pip install -e ".[dev]" - Run migrations to initialize PostgreSQL schema:
alembic upgrade head
- Seed database with baseline models:
python scripts/seed_db.py
- Run the FastAPI development server:
python -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
- (In a separate terminal) Start the Celery Worker tasks:
celery -A app.tasks.celery_app worker --loglevel=info --concurrency=4 -Q default,drift,shap
- (In a separate terminal) Start the Celery Beat scheduler:
celery -A app.tasks.celery_app beat --loglevel=info
- Navigate to the frontend directory and install dependencies:
cd ../frontend npm install - Start the Vite development server:
Dashboard will be available at http://localhost:3000.
npm run dev
Generate artificial predictions and simulate feature/concept drift in real time:
cd backend
python scripts/generate_real_predictions.pySentinel/
├── backend/ # FastAPI Backend API, Celery Tasks, & Database migrations
│ ├── app/ # Main application source code (routers, core metrics/rate limits, models, middleware)
│ ├── scripts/ # Database seeding, reset, and live prediction simulators
│ └── tests/ # Unit & integration tests for rate limiting, drift, and calibration
├── frontend/ # React, TypeScript, and Vite Dashboard
│ ├── src/ # Charting components, Zustand store, pages, layout with rail sidebar
│ └── nginx.conf # Production Nginx reverse proxy configuration
├── ml/ # Machine learning models, schemas, and training pipelines
│ └── models/ # Pre-trained XGBoost & Scikit-learn models (1, 2, 3)
├── infra/ # Docker Compose environment and configuration
│ ├── prometheus/ # Prometheus scraping configurations
│ └── grafana/ # Auto-provisioned Datasources and preloaded Sentinel dashboard
└── docs/ # System documentation, design reports & ADRs
Check that all system components are communicating correctly:
# 1. Verify backend health and version
curl http://localhost:8000/health
# Expected: {"status": "ok", "version": "0.1.0"}
# 2. Verify Prometheus endpoint is exposing API & custom ML metrics
curl http://localhost:8000/metrics | grep sentinel_
# Expected: Displays active sentinel_predictions_ingested_total, sentinel_drift_events_total, etc.
# 3. Verify PostgreSQL contains initialized tables
psql postgresql://sentinel:sentinel@localhost:5432/sentinel -c "\dt"
# 4. Verify baseline models are properly seeded
psql postgresql://sentinel:sentinel@localhost:5432/sentinel -c "SELECT id, name, version FROM model_registry;"- 🚀 Quick Start Reference Card — Command cheat-sheet & port references.
- ⚙️ Local Setup & Configuration Guide — Detailed environment variables and DB configuration.
- 🖥️ Running Dashboard Instructions — UI walkthrough and metrics explanation.
- 📝 Work Summary Log — System issue diagnosis, fixes, and sprint progress tracking.