Kyronix Vault - Distributed File System (DFS)

Kyronix Vault is a distributed file system built in Go that showcases chunk-based storage, streaming gRPC replication, and centralized metadata coordination. Engineered as a learning-focused project while incorporating architectural patterns commonly found in production-scale storage systems.

📖 Complete Documentation: System Design | Architecture | Protocol | Configuration | Development Roadmap

Features

Core Capabilities ✅

High-Performance File Transfers — End-to-end upload and download workflows powered by chunking and parallel data processing.
File & Directory Management — Support for file deletion, garbage collection workflows, and recursive directory listing APIs.
Chunk-Oriented Storage Architecture — Files are split into configurable chunks (8 MB by default, up to 64 MB) for efficient distribution and scalability.
Fault-Tolerant Replication — Configurable replication strategy with a default factor of three (one primary and two replicas), leveraging parallel replication for durability and availability.
Streaming Data Pipeline — Bidirectional upload streams and optimized download streams over gRPC, featuring back-pressure handling and SHA-256 integrity verification.
Dynamic Cluster Management — Automatic DataNode registration, heartbeat monitoring, health tracking, and cluster membership coordination.
Resilient Client Connectivity — Intelligent client connection pooling with automatic failover across multiple DataNodes.
Transactional Session Management — Dedicated metadata and streaming sessions designed to ensure consistency and operation atomicity.

Architecture Overview

Coordinator Service — Centralized metadata management responsible for file-to-chunk mapping, namespace coordination, and cluster state tracking.
DataNode Service — Distributed storage nodes that handle chunk persistence, peer-to-peer replication, and data distribution.
Go Client SDK — Developer-friendly SDK providing efficient parallel upload and download capabilities.
gRPC & Protocol Buffers — High-performance communication built on Protocol Buffers serialization and gRPC over HTTP/2.
Containerized Test Environment — Comprehensive Docker-based integration setup with one Coordinator and six DataNodes for end-to-end validation.

Roadmap 🚧

Advanced File Discovery — Native support for file and directory listing operations.
Persistent Metadata Layer — Integration with etcd (or a similar distributed store) to replace the current in-memory metadata implementation.
Production-Ready Garbage Collection — Enhanced cleanup workflows backed by persistent metadata and extensive validation.
Observability Stack — Metrics, distributed tracing, centralized logging, and operational monitoring.
Enterprise Security — TLS encryption, JWT-based authentication, and role-based access control (RBAC).
Data-at-Rest Encryption — Optional chunk-level encryption for enhanced storage security.
API Gateway — RESTful and HTTP-based access layer with integrated authentication and authorization.

See Complete Feature List: docs/roadmap.md | Technical Details: docs/architecture.md

Prerequisites

Tool	Version	Purpose
Go	1.24.4+	Building binaries and running unit tests
Docker	Latest	Local multi-node cluster and e2e testing
Protocol Buffers	Latest	Required only when modifying `.proto` files

Development Tools (Optional)

# Install protobuf tools for gRPC development
make dev-setup

Quick Start

Launch Full Cluster

# Start 1 coordinator + 6 datanodes with e2e tests
make e2e

Monitor and Manage

# View aggregated logs
tail -f ./logs/e2e_run.log

# Stop and cleanup
make e2e-down

Development Workflow

# Run unit tests
make test

# Generate protobuf (after .proto changes)
make clean && make proto

Detailed Commands: PROJECT.mdc | Architecture Guide: docs/architecture.md

Current Architecture Overview

System Components

flowchart TB
 subgraph subGraph0["Client Layer"]
        CLI["CLI Client"]
        SDK["Go SDK"]
        CPOOL["Client Pool<br>• Rotating connections<br>• Failover handling<br>• Retry logic"]
  end
 subgraph subGraph1["Control Plane"]
        COORD["Coordinator<br>• Metadata Management<br>• Node Selection<br>• Session Management"]
  end
 subgraph subGraph2["Data Plane"]
        DN1["DataNode 1<br>• Chunk Storage<br>• Session Management"]
        DN2["DataNode 2<br>• Chunk Storage<br>• Session Management"]
        DN3["DataNode N<br>• Chunk Storage<br>• Session Management"]
  end
    CLI --> CPOOL
    SDK --> CPOOL
    CPOOL <--> COORD & DN1
    DN1 -. replicate .-> DN2 & DN3
    DN2 -. replicate .-> DN3
    DN1 -. heartbeat .-> COORD
    DN2 -. heartbeat .-> COORD
    DN3 -. heartbeat .-> COORD

    style CPOOL fill:#fff3e0
    style COORD fill:#e1f5fe
    style DN1 fill:#f3e5f5
    style DN2 fill:#f3e5f5
    style DN3 fill:#f3e5f5

Core Concepts

Chunk-Based Storage — Files are partitioned into fixed-size chunks (8 MB by default), each protected with SHA-256 checksums for integrity verification.
Distributed Replication — Every chunk is replicated across multiple DataNodes, with primary-to-replica streaming ensuring redundancy and fault tolerance.
Transactional Sessions — Separate metadata and streaming sessions coordinate operations to maintain consistency and atomicity.
Resilient Client Pooling — Intelligent connection management with automatic failover, load distribution, and retry mechanisms across DataNodes.
Heartbeat-Driven Coordination — Periodic heartbeat exchanges enable health monitoring, node discovery, and incremental cluster state synchronization.

Protocol Workflows

Kyronix Vault implements three primary workflows for managing data throughout its lifecycle.

Upload Workflow

Metadata Initialization — The client requests an upload session from the Coordinator and receives chunk placement information.
Distributed Data Transfer — Chunks are streamed from the client to primary DataNodes, which concurrently replicate data to designated replicas.
Commit & Finalization — Upon successful transfer, the client commits the upload, allowing metadata to be finalized and exposed for future access.

Download Workflow

Metadata Discovery — The client retrieves file metadata and available chunk locations from the Coordinator.
Parallel Chunk Retrieval — Chunks are downloaded directly from DataNodes using efficient server-side streaming.
Integrity Verification & Assembly — Downloaded chunks are validated using checksums and reassembled into the original file.

Delete Workflow

Logical Deletion — The Coordinator marks the file as deleted within the metadata layer.
Background Reclamation — Garbage collection processes identify and remove obsolete chunks from storage nodes.
Orphan Cleanup — Local cleanup routines periodically scan for and remove orphaned data that is no longer referenced by metadata.

Complete Architecture (including planned): docs/architecture.md | Detailed Protocol Flows: docs/protocol.md

Configuration

The system uses environment variables for service discovery and YAML files for operational configuration:

Service Discovery (Environment Variables)

# Coordinator location (required by all nodes)
COORDINATOR_HOST=coordinator
COORDINATOR_PORT=8080

# New: global log level (default=error, dev=info)
LOG_LEVEL=info

# Node registration (required by datanodes)  
DATANODE_HOST=datanode1
DATANODE_PORT=8081

Operational Settings (YAML Configuration)

# configs/coordinator.yaml
coordinator:
  chunk_size: 8388608      # 8MB default
  metadata:
    commit_timeout: "5m"

# configs/datanode.yaml  
node:
  replication:
    timeout: "2m"
  session:
    timeout: "1m"

Complete Configuration Guide: docs/configuration.md

Project Structure

dfs/
├── cmd/                   # Entry points
│   ├── coordinator/       # Coordinator service main
│   ├── datanode/          # DataNode service main
│   └── client/            # Client CLI main
├── internal/              # Private application code
│   ├── client/            # Client SDK (uploader, downloader)
│   ├── clients/           # gRPC client wrappers
│   ├── cluster/           # Node management & selection
│   ├── common/            # Shared types, proto -> internal type conversion & validation
│   ├── config/            # Configuration management
│   ├── coordinator/       # Coordinator business logic
│   ├── datanode/          # DataNode business logic
│   └── storage/           # Storage interfaces and implementations
│       ├── chunk/         # Chunk storage (disk-based)
│       ├── encoding/      # Serialization interface and protocol (protobuf)
│       └── metadata/      # Metadata storage (currently in-memory)
├── pkg/                   # Public library code
│   ├── proto/             # Generated protobuf files
│   ├── logging/           # Structured logging utilities
│   ├── streamer/          # Streaming utilities
│   ├── utils/             # General utilities, small functions, prototyping
│   ├── client_pool/       # Client connection pools (rotating etc.)
│   ├── testutils/         # Test utilities
├── tests/                 # Test files
│   └── e2e/              # End-to-end tests
├── configs/              # Example yaml configuration files
├── deploy/               # Deployment configurations
├── docs/                 # Documentation
└── logs/                 # Log output directory, local testing

Detailed Structure: PROJECT.mdc

Development

Making Changes

# 1. Feature development
make test                   # Run unit tests
make e2e                    # Run full integration tests

# 2. Protocol changes
make clean && make proto    # Regenerate after .proto edits
make test                   # Verify changes

Testing

Test Coverage

Unit Tests: make test - >80% coverage target with race detection
End-to-End: make e2e - Full cluster scenarios with varying file sizes
Integration: Component interaction testing

Test Environment

# Local development
make test

# Full cluster simulation  
make e2e

# Continuous integration
# All tests run automatically on pull requests

Debugging Tests

# Enable debug logging - modify in .env file
LOG_LEVEL=info # debug, info, warn, error
ENVIRONMENT=development # development, production
DEBUG_E2E_TESTS=false # true, false

# View specific logs
docker logs dfs_coordinator_1
docker logs dfs_datanode_1

Testing Strategy: docs/architecture.md#testing-infrastructure

Documentation

Complete Documentation Index

Document	Purpose	Audience
README.md	Project overview and quick start	All users
docs/design.md	System design and future architecture	Architects, senior developers
docs/architecture.md	Current implementation details	Developers, contributors
docs/protocol.md	Wire protocol and API specifications	Protocol developers
docs/configuration.md	Configuration reference and examples	Operators, DevOps
docs/roadmap.md	Development phases and feature planning	Project managers, stakeholders
PROJECT.mdc	Developer reference and AI assistant guide	Developers, AI tools

Quick Navigation

🚀 Getting Started: Prerequisites → Quick Start
🏗️ Understanding the System: Architecture Overview → docs/architecture.md
⚙️ Configuration: Configuration → docs/configuration.md
🔧 Development: Development → PROJECT.mdc
📋 Planning: Roadmap → docs/roadmap.md

Roadmap

Development Progress

✅ Phase 0 — Core Storage Foundation
Implemented distributed file uploads and downloads, chunk-based storage, replication, streaming protocols, and cluster coordination.
🚧 Phase 1 — Storage Lifecycle & Durability
Complete file management operations, strengthen garbage collection workflows, and introduce persistent metadata storage.
📋 Phase 2 — Developer Experience & APIs
Expand the CLI experience and introduce an HTTP/REST gateway for broader integration support.
📋 Phase 3 — Security & Access Control
Add transport encryption, authentication, authorization, and role-based access controls.
📋 Phase 4 — Scalability & Performance
Optimize data placement, replication efficiency, throughput, and cluster scalability.
📋 Phase 5 — Observability & Operations
Introduce metrics, distributed tracing, monitoring, alerting, and operational tooling.

Near-Term Priorities

Complete File Management APIs — Finalize file deletion, directory listing, and metadata lifecycle operations.
Introduce Persistent Metadata Storage — Integrate etcd (or a similar distributed datastore) to eliminate reliance on in-memory metadata.
Strengthen Garbage Collection — Implement robust orphaned-chunk detection and automated cleanup workflows.
Enhance the CLI Experience — Provide a more intuitive command-line interface with improved usability and diagnostics.

Long-Term Vision

Production-Grade Distributed Storage Platform — Evolve Kyronix Vault into a resilient, fault-tolerant, and operationally mature distributed file system.
Multi-Cloud Storage Integration — Support external storage backends such as Amazon S3, Google Cloud Storage, and Azure Blob Storage.
Advanced Data Services — Add encryption, compression, deduplication, and intelligent storage optimization features.
Operational Excellence — Deliver comprehensive monitoring, alerting, automated recovery, and cluster management capabilities.

Detailed Roadmap: docs/roadmap.md | Technical Planning: docs/design.md

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.vscode		.vscode
dfs		dfs
docs		docs
.gitignore		.gitignore
PROJECT.mdc		PROJECT.mdc
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation