🌤️ Distributed Weather Streaming System

A fault-tolerant, real-time weather data streaming pipeline built with Kafka, gRPC, and Docker. Processes 10 messages/sec across 4 partitions using Protobuf serialization and offset checkpointing to achieve exactly-once processing with crash recovery.

Features

• High-throughput ingestion — 10 messages/sec across 4 Kafka partitions
• Protobuf serialization — compact binary encoding via Protocol Buffers
• Exactly-once processing — offset checkpointing ensures no duplicates on crash recovery
• gRPC API — typed, low-latency service interface for weather data queries
• Dockerized — fully containerized with Docker Compose for local dev and deployment
• Fault tolerance — consumer groups with automatic partition rebalancing

Architecture

Weather Sources (simulated)
        │
        ▼
┌───────────────────┐
│  Kafka Producer   │  Protobuf serialization
│  4 partitions     │  offset tracking
└────────┬──────────┘
         │
    ┌────▼────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐
    │ Part. 0 │  │ Part. 1 │  │ Part. 2 │  │ Part. 3 │
    └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘
         └────────────┴────────────┴─────────────┘
                           │
                  ┌────────▼────────┐
                  │  Kafka Consumer │  exactly-once
                  │  (consumer grp) │  crash recovery
                  └────────┬────────┘
                           │
                  ┌────────▼────────┐
                  │   gRPC Server   │  typed API
                  │  WeatherService │
                  └─────────────────┘

Tech Stack

Component	Technology
Message broker	Apache Kafka
Serialization	Protocol Buffers (Protobuf)
API layer	gRPC
Containerization	Docker, Docker Compose
Language	Python 3.11
Consumer	confluent-kafka

Getting Started

Prerequisites

• Docker & Docker Compose
• Python 3.11+

Run with Docker Compose

docker-compose up

This starts Zookeeper, Kafka (4 partitions), the producer, consumer, and gRPC server.

Run locally

pip install -r requirements.txt
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. weather.proto

# Terminal 1 — Producer
python producer.py

# Terminal 2 — Consumer
python consumer.py

# Terminal 3 — gRPC server
python server.py

Query via gRPC client

python client.py --city "Madison" --limit 10

Kafka Configuration

KAFKA_TOPIC = 'weather-events'
NUM_PARTITIONS = 4
REPLICATION_FACTOR = 1
MESSAGES_PER_SEC = 10

Protobuf Schema

message WeatherEvent {
  string city = 1;
  float temperature = 2;
  float humidity = 3;
  float wind_speed = 4;
  string condition = 5;
  string timestamp = 6;
  int32 partition = 7;
  int64 offset = 8;
}

Exactly-Once Semantics

Offset checkpointing is implemented by committing offsets only after successful processing:

consumer.commit(asynchronous=False)  # synchronous commit after processing

On crash recovery, the consumer resumes from the last committed offset — ensuring no message is processed twice.

Author

Rakshith Sriraman Krishnaraj · LinkedIn · Google Scholar