Kafka Stack with Docker Compose [Beginner]

This project sets up a Kafka stack using Docker Compose, including Kafka and Conduktor for easy management and visualization.

---

🚀 Prerequisites

• Docker installed and running on your machine
• Git installed

---

📥 Clone the Repository

git clone https://github.com/conduktor/kafka-stack-docker-compose.git
cd kafka-stack-docker-compose

---

📥 Start Kafka and Conduktor

docker compose -f conduktor-kafka-single.yml up

• Kafka will be available at $DOCKER_HOST_IP:9092
• Conduktor will be available at $DOCKER_HOST_IP:8080

---

📡 Kafka Producer: Wikimedia Stream

This project also includes a Kafka producer that streams live changes from Wikimedia into Kafka.

▶️ Steps to Run

1. Navigate to the project directory

cd conductor-platform

2. Start required services (if not already running)

docker compose -f your-file.yml up -d

3. Run the Wikimedia Producer

Once everything is up and running, start your WikiMediaChangesProducer application from your IDE or terminal.

---

📌 What this does

• Connects to the Wikimedia live stream
• Processes real-time change events
• Publishes them into a Kafka topic

---

✅ Verify

• Open Conduktor at http://localhost:8080
• Check your Kafka topics
• You should see live messages flowing in 🎉

---

📡 Kafka Consumer: Opensearch

This module includes a Kafka consumer that streams live data from a Kafka topic into OpenSearch.

▶️ Steps to Run

1. Navigate to the project directory

cd conductor-platform

cd kafka-consumer-opensearch

2. Start required services (if not already running)

docker compose -f your-file.yml up -d

3. Run the OpenSearch Consumer

Once everything is up and running, start your OpenSearchConsumer application from your IDE or terminal.

---

📌 What this does

• Consumes messages from the configured Kafka topic
• Indexes the data into OpenSearch
• Processes real-time change events

---

✅ Verify

1. Check the logs of the Kafka consumer and locate an IndexResponse ID
   Example:

   y6gYNp0Ba2tgHH6QrCPb
   

2. Open OpenSearch Dashboard:

   http://localhost:5601
   

3. Navigate to Dev Tools and run:

   GET /wikimedia/_doc/y6gYNp0Ba2tgHH6QrCPb

4. Verify that the document is successfully indexed.

---