Distributed Image Processing Pipeline (Project 3)

This project uses Apache Kafka to create a distributed pipeline that processes images

A master node splits an image into tiles and publishes them as tasks.
Multiple worker nodes consume these tasks, process the image tiles (e.g., grayscale), and publish the results.
Finally, the master node reconstructs the final image.

---

Team Roles

• Node 1: Master/Client
• Node 2: Kafka Broker
• Node 3: Worker 1
• Node 4: Worker 2

---

How to Run

1. Clone the Repo

git clone https://github.com/kshirinshetty/100_Project3_BD.git
cd 100_Project3_BD

---

2. Person 1 (Broker)

• Start ZooKeeper.

  bin/zookeeper-server-start.sh config/zookeeper.properties
  

• Start the Kafka serer.

  bin/kafka-server-start.sh config/server.properties
  

• Create the topics: tasks, results, heartbeats.
• Share BROKER_IP with everyone.

---

3. Master & Workers

• Update the broker_ip in master.py or worker.py.
• Create and activate a Python virtual environment.
• Install dependencies:

pip install -r requirements.txt

---

4. Run Order

1. NODE 2: (Kafka is running)
2. NODE 3: python worker.py worker-1
3. NODE 4: python worker.py worker-2
4. NODE 1: python master.py
5. NODE 1 Open http://127.0.0.1:5000 in a browser to use the application.

---