Argus — Distributed Observability Platform

A self-hosted observability stack with ML anomaly detection. Three instrumented microservices emit logs, metrics and traces to Kafka; Spark Structured Streaming turns them into 10-second windows in TimescaleDB and searchable documents in Elasticsearch; a FastAPI backend scores every window with an Isolation Forest and an LSTM autoencoder, raises alerts, and streams it all to a React dashboard.

Results

Live, end to end (python fault_injection_test.py --trials 2 --rps 6): real traffic through the whole stack, faults switched on through each service's own /admin/chaos endpoint, anomalies read back from the API. Detection delay is measured from the moment the fault is switched on to the anomaly appearing in the API, so it includes Kafka, the Spark window, the TimescaleDB write and the API's 10 s inference poll.

12 incidents (2 trials × 3 services × 2 fault kinds, 90 s each, 6 req/s per service), results in results/live_detection.json:

Fault	Detected	Detection delay
Latency spike (chaos spike mode)	6 / 6	0.6 – 19.5 s
Error burst (20 % errors, latency unchanged)	6 / 6	6.3 – 22.7 s
All	12 / 12	median 9.9 s, max 22.7 s

The false-alarm rate from this run is not representative: the trials started as soon as the first production models existed, and those are trained on the trainer's minimum of 60 windows (10 minutes), where the benchmark models see an hour. Use the benchmark below for false alarms.

Two bugs this run found are fixed: ingestion/producer.py waited for a Kafka ack on every message, which blocked the services' event loops (order-service p99 sat at the 5 s auth timeout), and retraining dropped only the windows the forest flagged, so error bursts the autoencoder caught leaked into the next model.

Labelled benchmark (python -m ml.evaluation.detection_benchmark --seeds 5): 5 seeds × 3 services, each trained on 1 hour of normal traffic and tested on 6 hours with 24 labelled incidents across eight fault kinds, four severe and four subtle (latency ×1.6, 4–8 % errors, a slow latency creep, traffic halved). The traffic is generated from the demo services' own latency and error model, so treat it as a comparison of the detectors on known faults, not production data.

Detector	Incident recall	Median delay	False alarms / hour	Window precision
Isolation Forest	0.953 ± 0.052	10 s	2.26 ± 1.34	0.959
LSTM autoencoder	0.989 ± 0.018	5 s	1.33 ± 0.74	0.962
Either (what the API alerts on)	0.994 ± 0.014	0 s	2.83 ± 1.33	0.941

The two detectors miss different things: the forest scores each window on its own and misses a quarter of the mild error bursts (recall 0.756), while the autoencoder sees the sequence and catches all of them. Neither is perfect on traffic halving. Per-kind numbers are in ml/evaluation/results/detection_benchmark.json.

Architecture

 auth-service ◄── order-service    (order calls auth and propagates x-trace-id / x-parent-span-id)
 payment-service
        │  logs · metrics · traces
        ▼
 Kafka:  logs · metrics · traces
        │
        ▼
 Spark Structured Streaming
   • 10 s windows per service: p50/p95/p99 latency, error rate, throughput ──► TimescaleDB service_metrics
   • custom service metrics (CPU, memory, queue depth, revenue) per window ──► TimescaleDB service_custom_metrics
   • traces with span and parent span ids, logs                            ──► Elasticsearch
        │
        ▼
 FastAPI (port 8000)                                ml-trainer (own container)
   • inference loop every 10 s                        • trains both models per service from TimescaleDB
       Isolation Forest (sklearn pipeline)            • exports the LSTM to ONNX
       LSTM autoencoder (ONNX via onnxruntime)        • registers both in MLflow, moves the `production` alias
   • reloads `production` models from MLflow ◄──────  • hourly retrain, PSI drift check every 5 min
   • alert engine (Redis state, webhooks with retry)  • on-demand retrain from a Redis queue
   • service dependency graph built from traces
   • REST + WebSocket /ws/dashboard
        │
        ▼
 React dashboard (port 3000): live charts, anomaly overlays, alerts, logs, traces, dependency graph

ML pipeline

• Isolation Forest: scaler and forest in one sklearn pipeline, contamination 0.01. Scores single windows, so it is good at point anomalies.
• LSTM autoencoder (PyTorch): reconstructs 30-window sequences of the five features. The score is the reconstruction error of the newest 3 windows, so a fault shows up as soon as it starts instead of being averaged over the whole sequence. The threshold is the 99.5th percentile of training error. The model is exported to ONNX, so the API serves it with onnxruntime and does not need PyTorch.
• Model registry: each service has argus-iforest-<service> and argus-lstm-<service> in the MLflow Model Registry. A training run registers both and only then moves the production alias, so the API never pairs a new forest with an old autoencoder. The API polls the alias and hot-swaps models; rolling back means moving the alias. GET /admin/models shows what is serving.
• Retraining: the ml-trainer container retrains every hour and drops the windows the current production forest flags, so an incident does not become the new normal. POST /admin/retrain?service=… queues an immediate retrain.
• Drift: every 5 minutes, PSI per feature between the last 30 minutes and the production model's training baseline, logged to the MLflow experiment argus-drift. PSI above 0.25 on three consecutive checks triggers a retrain; a single incident does not. On stable traffic the noise floor is about 0.03 (see ml/drift.py).

Tech stack

Layer	Technology
Message bus	Apache Kafka
Stream processing	Apache Spark Structured Streaming
Time-series DB	TimescaleDB
Logs and traces	Elasticsearch
Alert state, retrain queue	Redis
ML	scikit-learn, PyTorch, ONNX Runtime
Model registry, drift tracking	MLflow
API	FastAPI
Frontend	React 18, Recharts, React Flow
Instrumentation	OpenTelemetry SDK
Infra	Docker Compose

Quick start

cp .env.example .env
docker compose up -d          # about 2 minutes until everything is healthy
open http://localhost:3000

# the first production models appear after ~10 minutes of traffic
python load-testing/load_test.py --duration 600
curl localhost:8000/admin/models

Tests

pip install -r requirements/api.txt -r requirements/trainer.txt pytest
pytest      # detectors, ONNX export, drift, registry round trip, alert engine

Repository structure

argus/
├── ingestion/              # Kafka topics and message schemas
├── processing/spark/       # windowed aggregation, custom metrics, trace and log indexing
├── ml/
│   ├── models/             # Isolation Forest, LSTM autoencoder, ONNX scorer
│   ├── training/           # trainer and the ml-trainer worker loop
│   ├── evaluation/         # labelled detection benchmark and results
│   ├── drift.py            # PSI drift scoring
│   └── registry.py         # MLflow Model Registry publish and load
├── api/                    # FastAPI routers, services, ml_serving
├── frontend/               # React dashboard
├── demo-services/          # instrumented auth, order and payment services
├── infra/                  # Dockerfiles, DB init scripts
├── load-testing/           # traffic generator
├── fault_injection_test.py # live end-to-end detection evaluation
└── docker-compose.yml

See Instructions.md for a step-by-step demo walkthrough.