🌐 Distributed File System (Go-DFS)

A high-performance, fault-tolerant distributed file system built with Go, gRPC, and Protocol Buffers. This system is designed to handle video files (MP4) with automated replication and real-time node monitoring.

🚀 Architectural Overview

The system follows a Master-Worker (Tracker-Keeper) architecture:

1. Master Tracker

The central nervous system of the DFS.

• Node Registry: Manages dynamic registration of Data Keeper nodes.
• Metadata Management: Tracks file locations across the cluster using a consistent record system.
• Heartbeat Monitor: Uses gRPC streams to monitor node health in real-time, automatically marking dead nodes for recovery.
• Auto-Replication Engine: A background orchestrator that ensures every file is replicated across at least 3 distinct nodes to guarantee high availability.

🧱 System Architecture

    +------------------+          1. Query          +------------------+
    |                  | ------------------------>  |                  |
    |   Smart Client   |                            |  Master Tracker  |
    |                  | <------------------------  |                  |
    +-------+----------+          2. Metadata       +--------+---------+
            |                                                ^
            |                                                |
            | 3. High-Speed TCP Stream                       | 4. Heartbeats
            v                                                | (gRPC)
    +-------+----------+                            +--------+---------+
    |                  |                            |                  |
    |  Data Keeper 1   | ------------------------>  |  Data Keeper 2   |
    |                  |      5. Replication        |                  |
    +------------------+          (P2P TCP)         +------------------+

Flow Description:

1. Handshake: Client queries Master for node availability.
2. Coordination: Master returns metadata for viable Data Keepers.
3. Data Transfer: Client streams data directly to a Keeper via high-speed TCP.
4. Monitoring: Keepers maintain constant health signals to Master via gRPC.
5. Redundancy: Master triggers P2P replication between Keepers to maintain factor of 3.

2. Data Keeper Nodes

The storage backbone of the system.

• Dual-Protocol Communication: Uses gRPC for control plane operations (registration, replication commands) and TCP for high-speed data plane operations (file transfers).
• Dynamic Port Management: Handles multiple concurrent uploads/downloads by dynamically allocating free ports.
• Self-Scanning: Automatically indexes local MP4 files upon startup and syncs with the Master.

3. Smart Client

• Protocol Handshake: Communicates with the Master to find the optimal node for upload/download.
• Hybrid Transfer: Orchestrates the multi-stage transfer process, handling serialization and high-speed TCP streaming.

🛠️ Tech Stack & Skills

• Language: Go (Golang)
• Communication: gRPC, Protobuf (Protocol Buffers)
• Networking: TCP/IP, Socket Programming
• Concurrency: Goroutines, Channels, Mutexes, Sync WaitGroups
• Architecture: Distributed Systems, Master-Slave, Fault Tolerance

⚡ Key Features

• Fault Tolerance: Automatic detection of node failure and redistribution of lost replicas.
• Replication Strategy: Ensures a replication factor of 3 for all stored content.
• Concurrency: Highly concurrent I/O using Go's lightweight threading model.
• Efficient Serialization: Uses Protocol Buffers for minimal overhead in control messages.

🔧 Getting Started

Prerequisites

• Go 1.19+
• gRPC and Protobuf tools

Running the System

1. Start the Master Tracker:

   cd internals/master_tracker
   go run main.go

2. Start multiple Data Keeper Nodes:

   cd internals/data_keeper_node
   go run main.go # Run this in multiple terminals

3. Use the Client:
   • Upload: go run main.go u <filename.mp4>
   • Download: go run main.go d <filename.mp4>

---

Contributers

• Mamdouh Attia
• Ahmed Asaad

---

This project demonstrates deep understanding of distributed consensus, networked I/O, and concurrent system design.