diff --git a/README.md b/README.md index fb33181..9160526 100644 --- a/README.md +++ b/README.md @@ -2,36 +2,80 @@ **AI-Powered Distributed Log Intelligence & Anomaly Detection Platform** -SentinelOps ingests real-time logs and metrics from distributed microservices, detects anomalies using statistical and ML-based models, and exposes a natural-language query interface (LLM/RAG) for fast incident triage. +[![CI](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml/badge.svg)](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml) + +SentinelOps ingests real-time logs from distributed microservices, detects anomalies using statistical models (z-score + EWMA), and will expose a natural-language incident-query interface (LLM/RAG) for fast triage. Built as a polyglot system (Go, Rust, Python) to demonstrate production patterns across the stack: event-driven ingestion, self-migrating services, batched writes, full observability, and a real CI/CD pipeline with branch protection. -## Architecture +## What's actually running today ```mermaid flowchart LR - A[Microservice Producers] -->|events| B[(Kafka)] - B --> C[Go Ingestion Service] - C --> D[(TimescaleDB)] - C --> E[Rust Anomaly Engine] - E --> D - D --> F[Python LLM/RAG Service] - F --> G[Next.js Dashboard] - C --> H[Redis Cache] - F --> H - subgraph Observability - I[Prometheus] --> J[Grafana] - end - C -.metrics.-> I - E -.metrics.-> I - F -.metrics.-> I + A[Log Producer
Python] -->|JSON events| B[(Redpanda
Kafka-compatible)] + B --> C[Ingestion Service
Go] + C --> D[(TimescaleDB
hypertables)] + E[Anomaly Engine
Rust] -->|polls 1min aggregate| D + E -->|writes detections| D + C -.Prometheus metrics.-> F[":9100/metrics"] + E -.Prometheus metrics.-> G[":9200/metrics"] ``` -## Status -[![CI](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml/badge.svg)](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml) +| Service | Language | Role | Status | +|---|---|---|---| +| `log-producer` | Python | Simulates 4 microservices, injects realistic anomaly bursts | ✅ Running | +| `ingestion-go` | Go | Kafka consumer → batched writes to TimescaleDB, self-migrating schema | ✅ Running | +| `anomaly-rust` | Rust | Polls continuous aggregate, z-score + EWMA anomaly detection | ✅ Running | +| `llm-query-python` | Python | Natural-language RAG query interface over detected anomalies | 🚧 Not started | +| `dashboard-nextjs` | TypeScript | Real-time dashboard | 🚧 Not started | + +See [docs/MILESTONES.md](docs/MILESTONES.md) for the full roadmap and what's shipped so far. + +## Quickstart + +Requires Docker and Docker Compose. + +```bash +git clone https://github.com/yahiaongh/sentinelops.git +cd sentinelops/infra +cp .env.example .env +docker compose up -d +``` + +Wait ~20 seconds for health checks, then verify: + +```bash +# Confirm all containers are healthy +docker compose ps -🚧 Early development — see [milestones](docs/MILESTONES.md). +# Watch log events land in TimescaleDB +docker exec -it sentinelops-timescaledb psql -U sentinelops -d sentinelops \ + -c "SELECT count(*) FROM log_events;" -## Local Development -See `infra/docker-compose.yml` (coming in Milestone 1). +# Watch anomalies get detected in real time +docker compose logs -f anomaly-rust + +# Browse the raw event stream +open http://localhost:8080 # Redpanda Console + +# Check ingestion service metrics +curl http://localhost:9100/metrics | grep sentinelops + +# Check anomaly engine metrics +curl http://localhost:9200/metrics | grep sentinelops +``` + +## Engineering highlights + +A few things in this repo worth a closer look if you're reviewing it: + +- **Self-migrating schema, in two languages.** Both `ingestion-go` and `anomaly-rust` embed their SQL migrations at compile time (`go:embed`, `include_str!`) and apply them idempotently on every startup — no manual `psql` step, no drift between environments. This came from a real incident during development: an early version relied on Postgres's `docker-entrypoint-initdb.d`, which silently no-ops on a pre-existing volume. Fixed once, in a way that can't regress. +- **Directional vs. bidirectional anomaly detection.** The anomaly engine originally flagged latency z-scores in both directions, meaning a service running *faster* than its baseline was logged as a "warning" alongside a service running slower. Caught by reading live detection output critically rather than trusting a green build, root-caused, fixed, and covered by a regression test — see `services/anomaly-rust/src/detector/mod.rs`. +- **Idempotent batched writes under at-least-once delivery.** The ingestion service uses `pgx.Batch` to pipeline inserts in a single round trip, with `ON CONFLICT DO NOTHING` on a composite key — verified with a live integration test that duplicate Kafka delivery doesn't create duplicate rows. +- **7-job CI pipeline with branch protection.** Lint + test + build for Go and Rust, lint for Python, a database-integration test against a real TimescaleDB service container, and a Docker build validation for all three images — all required to pass before merge into `main`. + +## Local development + +Each service also runs standalone for local iteration — see the README in each service directory (`services//`) for language-specific setup. The full stack is defined in [`infra/docker-compose.yml`](infra/docker-compose.yml). ## License + MIT — see [LICENSE](LICENSE). \ No newline at end of file diff --git a/docs/MILESTONES.md b/docs/MILESTONES.md new file mode 100644 index 0000000..4002087 --- /dev/null +++ b/docs/MILESTONES.md @@ -0,0 +1,54 @@ +# Milestones + +Status and scope for each phase of SentinelOps. Dates are approximate; this is a portfolio project built incrementally, not on a fixed sprint schedule. + +## ✅ Milestone 1 — Infrastructure foundation +- Redpanda (Kafka-compatible event bus) with dual internal/external listeners +- TimescaleDB with hypertable schema and continuous aggregates +- Redis (provisioned, not yet consumed by a service) +- Python log producer simulating 4 microservices with realistic anomaly injection +- Docker Compose stack with health checks + +## ✅ Milestone 1b — Go ingestion service +- Kafka consumer with batched writes to TimescaleDB (`pgx.Batch`, idempotent via `ON CONFLICT`) +- Self-migrating schema via `go:embed` +- Full Prometheus instrumentation (`/metrics`, `/healthz`) +- Distroless multi-stage Docker build + +## ✅ Milestone 1c — CI/CD pipeline +- GitHub Actions: lint (`golangci-lint`, `ruff`), test, Docker build validation +- Fan-in `CI Success` required status check +- Branch protection on `main` (no force-push, no delete, status checks required) +- Adopted PR-based workflow after discovering direct-push conflicts with required status checks + +## ✅ Milestone 2 — Rust anomaly-detection engine +- Polls the `log_events_1min` continuous aggregate every 15s +- Z-score detection on p95 and max latency (directional — only flags getting slower) +- EWMA-based error-rate deviation detection (bidirectional — flags spikes and suspicious drops) +- Self-migrating schema (same pattern as the Go service, via `include_str!`) +- Caught and fixed a real directional-severity bug in detection logic, with regression tests +- Extended CI to 7 jobs: added Rust lint/test/fmt and a third Docker image build + +## 🚧 Milestone 3 — Python LLM/RAG query service (next) +- FastAPI service exposing natural-language queries over detected anomalies and log context +- RAG pipeline: retrieve relevant log/anomaly context, summarize via LLM +- Incident summarization ("what happened to checkout between 2-3am?") + +## 🚧 Milestone 4 — Next.js dashboard +- Real-time anomaly feed +- Natural-language query chat interface +- Service health overview + +## 🚧 Milestone 5 — Observability stack +- Prometheus scraping all services (currently exposed but not scraped) +- Grafana dashboards + +## 🚧 Milestone 6 — Kubernetes deployment +- Manifests for all services +- Local cluster deployment (k3d/kind) as a production-shaped deploy path + +## Known gaps (tracked, not blocking) +- `anomaly-rust`: `detect()` has no end-to-end unit test (only its component math functions are tested) +- `anomaly-rust`: EWMA baseline isn't persisted in `AnomalyRecord` for error-burst anomalies +- `ingestion-go`: `store.go` write path is integration-tested but lacks a unit test with a mocked pool +- No Grafana/Prometheus scrape config yet — metrics are exposed but not yet centrally collected \ No newline at end of file