Skip to content

About

A dynamic Kafka-based content streaming system

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

KStreamHive

Apache Kafka Python Next.js MySQL Flask

A Dynamic, Real-Time Content Streaming Platform with Kafka for empowering adaptive topic management, multi-threaded data ingestion, and interactive subscription handling.

• Features
• Architecture
• Tech Stack
• Prerequisites
• Installation
• Usage
• Project Structure
• API Endpoints
• Database Schema
• Use Cases

Features

  • Dynamic Topic Management: Create, approve, and manage Kafka topics through a centralized control system
  • Real-Time Streaming: Multi-threaded producer architecture for continuous data ingestion
  • Interactive Consumers: Users can subscribe/unsubscribe to topics dynamically without restart
  • Admin Dashboard: Next.js interface for monitoring and controlling the entire ecosystem
  • Centralized Metadata: MySQL for topics, subscriptions, and user management

Architecture

System Architecture

Tech Stack

Component Technology Purpose
Message Broker Apache Kafka Real-time data streaming
Backend Flask + Python REST API for admin operations
Frontend Next.js 16 + TypeScript Admin dashboard
Database MySQL Metadata registry
Producer Python (multi-threaded) Data ingestion
Consumer Python (kafka-python) Message consumption
UI Components Radix UI + Tailwind CSS Modern, accessible interface
Authentication JWT + bcrypt Secure admin access

Prerequisites

  • Python 3.8+
  • Node.js 18+ and npm
  • Apache Kafka 2.8+
  • Apache ZooKeeper (comes with Kafka)
  • MySQL 8.0+ or SQLite 3

Installation

1. Clone the Repository

git clone https://github.com/yourusername/KStreamHive.git
cd KStreamHive

2. Set Up Apache Kafka

Download and Extract Kafka

wget https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz
tar -xzf kafka_2.13-3.6.0.tgz
cd kafka_2.13-3.6.0

Start ZooKeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka Broker (in a new terminal)

bin/kafka-server-start.sh config/server.properties

Note: Update brokerconfig.properties in the project root with your Kafka broker address if different from default.

3. Set Up MySQL Database

Create Database and User

mysql -u root -p
CREATE DATABASE bigdata;
CREATE USER 'admin'@'localhost' IDENTIFIED BY 'your_password';
GRANT ALL PRIVILEGES ON bigdata.* TO 'admin'@'localhost';
FLUSH PRIVILEGES;
EXIT;

Import Database Schema

mysql -u admin -p bigdata < db.sql

Create Admin User (for login)

USE bigdata;
INSERT INTO admins (email, password_hash) 
VALUES ('admin@example.com', '$2b$12$YourBcryptHashHere');

Tip: Generate bcrypt hash using Python:

from flask_bcrypt import Bcrypt
bcrypt = Bcrypt()
print(bcrypt.generate_password_hash('your_password').decode('utf-8'))

4. Configure Python Environment

Install Python Dependencies

pip install -r requirements.txt

Update Database Configuration

Edit the following files with your database credentials:

  • app.py - Flask backend
  • producer.py - Producer script
  • consumer.py - Consumer script
DB_CONFIG = {
    'host': 'localhost',
    'user': 'admin',
    'password': 'your_password',
    'database': 'bigdata',
    'port': 3306
}

5. Set Up Next.js Frontend

cd topic_admin
npm install

Create Environment Variables

Create .env.local in topic_admin/ directory:

NEXT_PUBLIC_API_URL=http://localhost:5001

Usage

Starting the System

1. Start Kafka Infrastructure (if not already running)

# Terminal 1: ZooKeeper
bin/zookeeper-server-start.sh config/zookeeper.properties

# Terminal 2: Kafka Broker
bin/kafka-server-start.sh config/server.properties

2. Start Flask Backend

# Terminal 3
python app.py

Backend will run on http://localhost:5001

3. Start Next.js Admin Dashboard

# Terminal 4
cd topic_admin
npm run dev

Frontend will run on http://localhost:3000

4. Start Producer

# Terminal 5
python producer.py

5. Start Consumer(s)

# Terminal 6
python consumer.py

Creating and Streaming Topics

Add Topics to Stream

Create/edit topics.txt in the project root:

/path/to/dataset1.csv topic-name-1
/path/to/dataset2.csv topic-name-2

The producer will:

  1. Detect new entries in topics.txt
  2. Insert topic into database with pending status
  3. Wait for admin approval
  4. Create topic in Kafka once approved
  5. Stream CSV data to the topic

Admin Dashboard Workflow

  1. Login at http://localhost:3000/login with admin credentials
  2. View Dashboard - See statistics of all topics
  3. Approve Topics - Review pending topics and approve/reject
  4. Activate Topics - Move approved topics to active status
  5. Manage Active/Inactive - Toggle topics on/off dynamically
  6. View Subscriptions - Monitor user-topic relationships

Consumer Interaction

When you run consumer.py, you'll see an interactive menu:

═══════════════════════════════════════════
   DYNAMIC TOPIC SUBSCRIPTION MANAGER
═══════════════════════════════════════════

Active Topics:
  [1] topic-name-1 [Subscribed ✓]
  [2] topic-name-2 [Not Subscribed]
  
Options:
  [s <id>] Subscribe to topic
  [u <id>] Unsubscribe from topic
  [r] Refresh topic list
  [q] Quit

Project Structure

KStreamHive/
├── app.py                      # Flask REST API backend
├── producer.py                 # Multi-threaded Kafka producer
├── consumer.py                 # Interactive Kafka consumer
├── db.sql                      # MySQL database schema
├── brokerconfig.properties     # Kafka broker configuration
├── topics.txt                  # Input file for producer (CSV paths)
├── requirements.txt            # Python dependencies
├── architecture.avif           # System architecture diagram
├── instructions.txt            # Project requirements/documentation
│
└── topic_admin/                # Next.js Admin Dashboard
    ├── app/
    │   ├── page.tsx            # Main dashboard page
    │   ├── login/              # Admin login page
    │   ├── manage-topics/      # Active/inactive topic management
    │   └── api/                # Next.js API routes
    │       └── admin/
    │           ├── all/        # Fetch all topics
    │           ├── pending/    # Fetch pending topics
    │           ├── decide/     # Approve/reject topics
    │           └── subscriptions/ # User subscriptions
    ├── components/
    │   ├── dashboard-header.tsx
    │   ├── stats-overview.tsx
    │   ├── topics-list.tsx
    │   ├── status-timeline.tsx
    │   ├── user-subscriptions.tsx
    │   └── inactive-topics.tsx
    ├── lib/
    │   ├── api-config.ts       # API endpoint configuration
    │   └── utils.ts            # Utility functions
    └── package.json            # Node.js dependencies

API Endpoints

Method Endpoint Description
POST /admin/login Admin authentication (returns JWT)
GET /admin/all Fetch all topics (all statuses)
GET /admin/pending Fetch pending topics
POST /admin/decide Approve/reject topic
POST /admin/toggle_topic Toggle topic active/inactive
GET /admin/subscriptions Get user subscriptions with topic details

Database Schema

Topics Table

CREATE TABLE topics (
    topic_id INT PRIMARY KEY AUTO_INCREMENT,
    topic_name VARCHAR(255) UNIQUE NOT NULL,
    status ENUM('pending', 'approved', 'active', 'rejected', 'inactive') DEFAULT 'pending',
    description TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

User Subscriptions Table

CREATE TABLE user_subscriptions (
    subscription_id INT PRIMARY KEY AUTO_INCREMENT,
    user_id VARCHAR(100) NOT NULL,
    topic_id INT NOT NULL,
    subscribed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (topic_id) REFERENCES topics(topic_id),
    UNIQUE(user_id, topic_id)
);

Admins Table

CREATE TABLE admins (
    id INT PRIMARY KEY AUTO_INCREMENT,
    email VARCHAR(255) UNIQUE NOT NULL,
    password_hash VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Use Cases

  • Real-Time Data Pipelines: Stream financial data, IoT sensor readings, or social media feeds
  • Event-Driven Architectures: Build microservices that react to specific data topics
  • Log Aggregation: Collect and distribute logs from multiple sources
  • Dynamic Content Distribution: Manage content streams that change based on user demand
  • Data Analytics: Feed real-time data into analytics platforms

About

A dynamic Kafka-based content streaming system

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages