docs: honest README + milestones doc

README.md

@@ -2,36 +2,80 @@
 
 **AI-Powered Distributed Log Intelligence & Anomaly Detection Platform**
 
-SentinelOps ingests real-time logs and metrics from distributed microservices, detects anomalies using statistical and ML-based models, and exposes a natural-language query interface (LLM/RAG) for fast incident triage.
+[![CI](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml/badge.svg)](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml)
+
+SentinelOps ingests real-time logs from distributed microservices, detects anomalies using statistical models (z-score + EWMA), and will expose a natural-language incident-query interface (LLM/RAG) for fast triage. Built as a polyglot system (Go, Rust, Python) to demonstrate production patterns across the stack: event-driven ingestion, self-migrating services, batched writes, full observability, and a real CI/CD pipeline with branch protection.
 
-## Architecture
+## What's actually running today
 
 ```mermaid
 flowchart LR
-    A[Microservice Producers] -->|events| B[(Kafka)]
-    B --> C[Go Ingestion Service]
-    C --> D[(TimescaleDB)]
-    C --> E[Rust Anomaly Engine]
-    E --> D
-    D --> F[Python LLM/RAG Service]
-    F --> G[Next.js Dashboard]
-    C --> H[Redis Cache]
-    F --> H
-    subgraph Observability
-      I[Prometheus] --> J[Grafana]
-    end
-    C -.metrics.-> I
-    E -.metrics.-> I
-    F -.metrics.-> I
+    A[Log Producer<br/>Python] -->|JSON events| B[(Redpanda<br/>Kafka-compatible)]
+    B --> C[Ingestion Service<br/>Go]
+    C --> D[(TimescaleDB<br/>hypertables)]
+    E[Anomaly Engine<br/>Rust] -->|polls 1min aggregate| D
+    E -->|writes detections| D
+    C -.Prometheus metrics.-> F[":9100/metrics"]
+    E -.Prometheus metrics.-> G[":9200/metrics"]
 ```
 
-## Status
-[![CI](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml/badge.svg)](https://github.com/yahiaongh/sentinelops/actions/workflows/ci.yml)
+| Service | Language | Role | Status |
+|---|---|---|---|
+| `log-producer` | Python | Simulates 4 microservices, injects realistic anomaly bursts | ✅ Running |
+| `ingestion-go` | Go | Kafka consumer → batched writes to TimescaleDB, self-migrating schema | ✅ Running |
+| `anomaly-rust` | Rust | Polls continuous aggregate, z-score + EWMA anomaly detection | ✅ Running |
+| `llm-query-python` | Python | Natural-language RAG query interface over detected anomalies | 🚧 Not started |
+| `dashboard-nextjs` | TypeScript | Real-time dashboard | 🚧 Not started |
+
+See [docs/MILESTONES.md](docs/MILESTONES.md) for the full roadmap and what's shipped so far.
+
+## Quickstart
+
+Requires Docker and Docker Compose.
+
+```bash
+git clone https://github.com/yahiaongh/sentinelops.git
+cd sentinelops/infra
+cp .env.example .env
+docker compose up -d
+```
+
+Wait ~20 seconds for health checks, then verify:
+
+```bash
+# Confirm all containers are healthy
+docker compose ps
 
-🚧 Early development — see [milestones](docs/MILESTONES.md).
+# Watch log events land in TimescaleDB
+docker exec -it sentinelops-timescaledb psql -U sentinelops -d sentinelops \
+  -c "SELECT count(*) FROM log_events;"
 
-## Local Development
-See `infra/docker-compose.yml` (coming in Milestone 1).
+# Watch anomalies get detected in real time
+docker compose logs -f anomaly-rust
+
+# Browse the raw event stream
+open http://localhost:8080   # Redpanda Console
+
+# Check ingestion service metrics
+curl http://localhost:9100/metrics | grep sentinelops
+
+# Check anomaly engine metrics
+curl http://localhost:9200/metrics | grep sentinelops
+```
+
+## Engineering highlights
+
+A few things in this repo worth a closer look if you're reviewing it:
+
+- **Self-migrating schema, in two languages.** Both `ingestion-go` and `anomaly-rust` embed their SQL migrations at compile time (`go:embed`, `include_str!`) and apply them idempotently on every startup — no manual `psql` step, no drift between environments. This came from a real incident during development: an early version relied on Postgres's `docker-entrypoint-initdb.d`, which silently no-ops on a pre-existing volume. Fixed once, in a way that can't regress.
+- **Directional vs. bidirectional anomaly detection.** The anomaly engine originally flagged latency z-scores in both directions, meaning a service running *faster* than its baseline was logged as a "warning" alongside a service running slower. Caught by reading live detection output critically rather than trusting a green build, root-caused, fixed, and covered by a regression test — see `services/anomaly-rust/src/detector/mod.rs`.
+- **Idempotent batched writes under at-least-once delivery.** The ingestion service uses `pgx.Batch` to pipeline inserts in a single round trip, with `ON CONFLICT DO NOTHING` on a composite key — verified with a live integration test that duplicate Kafka delivery doesn't create duplicate rows.
+- **7-job CI pipeline with branch protection.** Lint + test + build for Go and Rust, lint for Python, a database-integration test against a real TimescaleDB service container, and a Docker build validation for all three images — all required to pass before merge into `main`.
+
+## Local development
+
+Each service also runs standalone for local iteration — see the README in each service directory (`services/<name>/`) for language-specific setup. The full stack is defined in [`infra/docker-compose.yml`](infra/docker-compose.yml).
 
 ## License
+
 MIT — see [LICENSE](LICENSE).

docs/MILESTONES.md

@@ -0,0 +1,54 @@
+# Milestones
+
+Status and scope for each phase of SentinelOps. Dates are approximate; this is a portfolio project built incrementally, not on a fixed sprint schedule.
+
+## ✅ Milestone 1 — Infrastructure foundation
+- Redpanda (Kafka-compatible event bus) with dual internal/external listeners
+- TimescaleDB with hypertable schema and continuous aggregates
+- Redis (provisioned, not yet consumed by a service)
+- Python log producer simulating 4 microservices with realistic anomaly injection
+- Docker Compose stack with health checks
+
+## ✅ Milestone 1b — Go ingestion service
+- Kafka consumer with batched writes to TimescaleDB (`pgx.Batch`, idempotent via `ON CONFLICT`)
+- Self-migrating schema via `go:embed`
+- Full Prometheus instrumentation (`/metrics`, `/healthz`)
+- Distroless multi-stage Docker build
+
+## ✅ Milestone 1c — CI/CD pipeline
+- GitHub Actions: lint (`golangci-lint`, `ruff`), test, Docker build validation
+- Fan-in `CI Success` required status check
+- Branch protection on `main` (no force-push, no delete, status checks required)
+- Adopted PR-based workflow after discovering direct-push conflicts with required status checks
+
+## ✅ Milestone 2 — Rust anomaly-detection engine
+- Polls the `log_events_1min` continuous aggregate every 15s
+- Z-score detection on p95 and max latency (directional — only flags getting slower)
+- EWMA-based error-rate deviation detection (bidirectional — flags spikes and suspicious drops)
+- Self-migrating schema (same pattern as the Go service, via `include_str!`)
+- Caught and fixed a real directional-severity bug in detection logic, with regression tests
+- Extended CI to 7 jobs: added Rust lint/test/fmt and a third Docker image build
+
+## 🚧 Milestone 3 — Python LLM/RAG query service (next)
+- FastAPI service exposing natural-language queries over detected anomalies and log context
+- RAG pipeline: retrieve relevant log/anomaly context, summarize via LLM
+- Incident summarization ("what happened to checkout between 2-3am?")
+
+## 🚧 Milestone 4 — Next.js dashboard
+- Real-time anomaly feed
+- Natural-language query chat interface
+- Service health overview
+
+## 🚧 Milestone 5 — Observability stack
+- Prometheus scraping all services (currently exposed but not scraped)
+- Grafana dashboards
+
+## 🚧 Milestone 6 — Kubernetes deployment
+- Manifests for all services
+- Local cluster deployment (k3d/kind) as a production-shaped deploy path
+
+## Known gaps (tracked, not blocking)
+- `anomaly-rust`: `detect()` has no end-to-end unit test (only its component math functions are tested)
+- `anomaly-rust`: EWMA baseline isn't persisted in `AnomalyRecord` for error-burst anomalies
+- `ingestion-go`: `store.go` write path is integration-tested but lacks a unit test with a mocked pool
+- No Grafana/Prometheus scrape config yet — metrics are exposed but not yet centrally collected