This project implements a data center in a distributed context. Data to be managed and processed include structured data (5 relational tables – User, Article, Read, Be-Read and Popular-Rank), and unstructured data (text, images and videos).
The system is deployed using Docker for containerization, MongoDB (as a sharded cluster) for structured data, Hadoop HDFS for unstructured data, and Redis for in-memory caching. Web application, built using Node.js, queries MongoDB cluster through the Router. Redis caches frequently accessed data to reduce MongoDB load.
Available functionality:
- Bulk load – one time mass insert into User, Article, and Read tables, construction of Be-Read and Popular-Rank tables
- Query execution – insert, update read statistics, top-5 popular articles
- Monitoring – running status, operations overview, resource memory usage, data distribution
- Advanced features – expansion allowing a new DBMS server to join; dropping a DBMS server at will
Folders structure:
data-generation– samples and scripts for generating datafrontend– web application serving as a single entry point; it starts the server and handles HTTP requests, including managing queriesscripts– JavaScript files for initializing tables
First time initialization
chmod +x init.sh (to make init.sh script executable)
./init.sh (automates data center setup)
To start the data center for subsequent launches
docker compose up
To start the web application
docker compose up frontend
Accessing the Web UI
http://localhost:3000 – web application
http://localhost:9870 – Hadoop
The data center can also be connected to and managed using MongoDB Compass. To connect, create a new connection and use the following URI: mongodb://localhost:27100/.