Streaming Metrics System

Real-time metrics pipeline: Kafka → Spark Streaming → Redis → Grafana

Architecture

flowchart LR
    Producer --> Kafka
    Kafka --> Spark[Spark Streaming]
    Spark -->|1min windows| Redis
    subgraph Serving Layer
        Redis[(Redis)] --> API
    end
    API --> Grafana

Loading

Quick Start

make setup             # One-command setup
make port-forward      # Grafana at localhost:3000 (admin/admin)

Manual Setup

make start-minikube    # Start cluster
make install-helm      # Install Kafka, Redis, Grafana
make build-images      # Build Docker images
make create-topic      # Create Kafka topic
make deploy-proxy      # Deploy API
make deploy-producer   # Deploy producer
make deploy-spark      # Deploy Spark job

Commands

make logs-producer     # Producer logs
make logs-spark        # Spark logs
make clean             # Cleanup