• Features
• Architecture
• Tech Stack
• Prerequisites
• Installation
• Usage
• Project Structure
• API Endpoints
• Database Schema
• Use Cases
- Dynamic Topic Management: Create, approve, and manage Kafka topics through a centralized control system
- Real-Time Streaming: Multi-threaded producer architecture for continuous data ingestion
- Interactive Consumers: Users can subscribe/unsubscribe to topics dynamically without restart
- Admin Dashboard: Next.js interface for monitoring and controlling the entire ecosystem
- Centralized Metadata: MySQL for topics, subscriptions, and user management
| Component | Technology | Purpose |
|---|---|---|
| Message Broker | Apache Kafka | Real-time data streaming |
| Backend | Flask + Python | REST API for admin operations |
| Frontend | Next.js 16 + TypeScript | Admin dashboard |
| Database | MySQL | Metadata registry |
| Producer | Python (multi-threaded) | Data ingestion |
| Consumer | Python (kafka-python) | Message consumption |
| UI Components | Radix UI + Tailwind CSS | Modern, accessible interface |
| Authentication | JWT + bcrypt | Secure admin access |
- Python 3.8+
- Node.js 18+ and npm
- Apache Kafka 2.8+
- Apache ZooKeeper (comes with Kafka)
- MySQL 8.0+ or SQLite 3
git clone https://github.com/yourusername/KStreamHive.git
cd KStreamHivewget https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz
tar -xzf kafka_2.13-3.6.0.tgz
cd kafka_2.13-3.6.0bin/zookeeper-server-start.sh config/zookeeper.propertiesbin/kafka-server-start.sh config/server.propertiesNote: Update
brokerconfig.propertiesin the project root with your Kafka broker address if different from default.
mysql -u root -pCREATE DATABASE bigdata;
CREATE USER 'admin'@'localhost' IDENTIFIED BY 'your_password';
GRANT ALL PRIVILEGES ON bigdata.* TO 'admin'@'localhost';
FLUSH PRIVILEGES;
EXIT;mysql -u admin -p bigdata < db.sqlUSE bigdata;
INSERT INTO admins (email, password_hash)
VALUES ('admin@example.com', '$2b$12$YourBcryptHashHere');Tip: Generate bcrypt hash using Python:
from flask_bcrypt import Bcrypt bcrypt = Bcrypt() print(bcrypt.generate_password_hash('your_password').decode('utf-8'))
pip install -r requirements.txtEdit the following files with your database credentials:
app.py- Flask backendproducer.py- Producer scriptconsumer.py- Consumer script
DB_CONFIG = {
'host': 'localhost',
'user': 'admin',
'password': 'your_password',
'database': 'bigdata',
'port': 3306
}cd topic_admin
npm installCreate .env.local in topic_admin/ directory:
NEXT_PUBLIC_API_URL=http://localhost:5001# Terminal 1: ZooKeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
# Terminal 2: Kafka Broker
bin/kafka-server-start.sh config/server.properties# Terminal 3
python app.pyBackend will run on http://localhost:5001
# Terminal 4
cd topic_admin
npm run devFrontend will run on http://localhost:3000
# Terminal 5
python producer.py# Terminal 6
python consumer.pyCreate/edit topics.txt in the project root:
/path/to/dataset1.csv topic-name-1
/path/to/dataset2.csv topic-name-2
The producer will:
- Detect new entries in
topics.txt - Insert topic into database with
pendingstatus - Wait for admin approval
- Create topic in Kafka once approved
- Stream CSV data to the topic
- Login at
http://localhost:3000/loginwith admin credentials - View Dashboard - See statistics of all topics
- Approve Topics - Review pending topics and approve/reject
- Activate Topics - Move approved topics to active status
- Manage Active/Inactive - Toggle topics on/off dynamically
- View Subscriptions - Monitor user-topic relationships
When you run consumer.py, you'll see an interactive menu:
═══════════════════════════════════════════
DYNAMIC TOPIC SUBSCRIPTION MANAGER
═══════════════════════════════════════════
Active Topics:
[1] topic-name-1 [Subscribed ✓]
[2] topic-name-2 [Not Subscribed]
Options:
[s <id>] Subscribe to topic
[u <id>] Unsubscribe from topic
[r] Refresh topic list
[q] Quit
KStreamHive/
├── app.py # Flask REST API backend
├── producer.py # Multi-threaded Kafka producer
├── consumer.py # Interactive Kafka consumer
├── db.sql # MySQL database schema
├── brokerconfig.properties # Kafka broker configuration
├── topics.txt # Input file for producer (CSV paths)
├── requirements.txt # Python dependencies
├── architecture.avif # System architecture diagram
├── instructions.txt # Project requirements/documentation
│
└── topic_admin/ # Next.js Admin Dashboard
├── app/
│ ├── page.tsx # Main dashboard page
│ ├── login/ # Admin login page
│ ├── manage-topics/ # Active/inactive topic management
│ └── api/ # Next.js API routes
│ └── admin/
│ ├── all/ # Fetch all topics
│ ├── pending/ # Fetch pending topics
│ ├── decide/ # Approve/reject topics
│ └── subscriptions/ # User subscriptions
├── components/
│ ├── dashboard-header.tsx
│ ├── stats-overview.tsx
│ ├── topics-list.tsx
│ ├── status-timeline.tsx
│ ├── user-subscriptions.tsx
│ └── inactive-topics.tsx
├── lib/
│ ├── api-config.ts # API endpoint configuration
│ └── utils.ts # Utility functions
└── package.json # Node.js dependencies
| Method | Endpoint | Description |
|---|---|---|
POST |
/admin/login |
Admin authentication (returns JWT) |
GET |
/admin/all |
Fetch all topics (all statuses) |
GET |
/admin/pending |
Fetch pending topics |
POST |
/admin/decide |
Approve/reject topic |
POST |
/admin/toggle_topic |
Toggle topic active/inactive |
GET |
/admin/subscriptions |
Get user subscriptions with topic details |
CREATE TABLE topics (
topic_id INT PRIMARY KEY AUTO_INCREMENT,
topic_name VARCHAR(255) UNIQUE NOT NULL,
status ENUM('pending', 'approved', 'active', 'rejected', 'inactive') DEFAULT 'pending',
description TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);CREATE TABLE user_subscriptions (
subscription_id INT PRIMARY KEY AUTO_INCREMENT,
user_id VARCHAR(100) NOT NULL,
topic_id INT NOT NULL,
subscribed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (topic_id) REFERENCES topics(topic_id),
UNIQUE(user_id, topic_id)
);CREATE TABLE admins (
id INT PRIMARY KEY AUTO_INCREMENT,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);- Real-Time Data Pipelines: Stream financial data, IoT sensor readings, or social media feeds
- Event-Driven Architectures: Build microservices that react to specific data topics
- Log Aggregation: Collect and distribute logs from multiple sources
- Dynamic Content Distribution: Manage content streams that change based on user demand
- Data Analytics: Feed real-time data into analytics platforms
