⚡ Real-Time Dynamic Pricing Engine with Apache Kafka

🚀 Overview

A high-throughput, event-driven real-time data streaming architecture built to simulate high-frequency surge pricing models (similar to Uber or airline booking engines). This system decouples data production from consumption using Apache Kafka, ensuring fault-tolerant, low-latency processing under massive load spikes.

🧠 Architecture & Data Flow

1. The Producer (producer.py): Continuously simulates live user transactions (ride requests, geolocation variables, driver availability) across multiple metropolitan zones and streams JSON payloads into Kafka at high frequency.
2. The Message Broker (Apache Kafka): Acts as an indestructible, partitioned buffer running via Homebrew (KRaft mode), handling event streaming synchronization and offset tracking.
3. The Consumer & Pricing Algorithm (consumer.py): Asynchronously polls the stream, computes real-time supply-to-demand ratios, applies business logic multipliers (Normal, Moderate Surge, High Surge), and computes dynamic pricing.
4. Data Persistence (pricing_ledger.csv): Automatically flushes real-time transformed streaming data into a persistent analytics ledger for downstream business intelligence.

🛠️ Tech Stack

• Streaming Core: Apache Kafka (KRaft mode)
• Python Client: confluent-kafka (C-optimized enterprise wrapper)
• Data Manipulation: Pandas, NumPy
• Storage: Local CSV Partition Ledger / Append-only stream log

💻 Local Setup & Execution

1. Install Apache Kafka (macOS):

brew install kafka
brew services start kafka