📦 Distributed File System (DFS)

A scalable and fault-tolerant Distributed File System built using Go and gRPC. This system allows uploading, downloading, and replicating files across multiple distributed storage nodes.

---

🚀 Features

• 📤 Upload files to distributed nodes
• 📥 Download files using parallel chunking
• 🔁 Automatic replication for fault tolerance
• ❤️ Heartbeat mechanism for node health monitoring
• ⚡ Concurrent file transfer for better performance

---

🏗️ System Architecture

The system is composed of three main components:

🧠 Master Tracker

• Maintains metadata of files and nodes
• Assigns nodes for uploads and downloads
• Monitors node health using heartbeats
• Handles replication logic

💾 Data Keeper (Storage Node)

• Stores files locally
• Handles upload and download requests
• Replicates files to other nodes
• Sends periodic heartbeat signals to master

🖥️ Client

• Uploads files to the system
• Downloads files from multiple nodes in parallel

---

📁 Project Structure

Distributed_File_System/
│
├── master_tracker/     # Master node logic
├── data_keeper/        # Storage node implementation
├── client/             # Client operations
├── dfs/                # Protobuf definitions
└── README.md

---

⚙️ Setup & Installation

Make sure you have the following installed:

• Go
• Protocol Buffers (protoc)
• gRPC dependencies

---

▶️ How to Run

1️⃣ Start the Master Tracker

cd master_tracker
go run main.go

Runs on:

localhost:3000

---

2️⃣ Start Data Keeper Nodes

cd data_keeper
go run main.go <MasterAddress>

Example:

go run main.go localhost:3001

💡 You can start multiple nodes for better replication and fault tolerance.

---

3️⃣ Run the Client

cd client
go run main.go <master_address> <upload|download> <filename> [local_filepath]

---

📤 Upload Example

go run main.go localhost:3000 upload myfile.txt ./myfile.txt

---

📥 Download Example

go run main.go localhost:3000 download myfile.txt

Or specify a custom save path:

go run maint.go localhost:3000 download myfile.txt ./downloads/myfile.txt

---

🔁 Replication Strategy

• Each file is replicated to up to 3 nodes
• Replication is triggered automatically by the master
• Ensures availability even if nodes fail

---

❤️ Heartbeat Mechanism

• Data nodes send heartbeat every 1 second
• Master marks nodes as dead after 3 seconds of inactivity

---

⚡ Parallel Download Mechanism

• File is split into chunks
• Each node serves a chunk
• Client downloads chunks concurrently and reconstructs the file

---