File Structure

[MaxFdev]

Suggested project file structure:

src/
├── main/
│   ├── java/
│   │   └── edu/
│   │       └── yu/
│   │           └── capstone/
│   │               └── DistributedSecretsVault/
│   │                   ├── DistributedSecretsVaultApplication.java
│   │                   │
│   │                   ├── config/
│   │                   │   ├── SecurityConfig.java
│   │                   │   ├── ClusterConfig.java
│   │                   │   ├── NetworkConfig.java
│   │                   │   └── StorageConfig.java
│   │                   │
│   │                   ├── controller/
│   │                   │   ├── SecretController.java
│   │                   │   ├── HealthController.java
│   │                   │   └── ClusterController.java
│   │                   │
│   │                   ├── service/
│   │                   │   ├── secret/
│   │                   │   │   ├── SecretService.java
│   │                   │   │   ├── SecretServiceImpl.java
│   │                   │   │   ├── SecretSharingService.java
│   │                   │   │   └── SecretReconstructionService.java
│   │                   │   ├── cluster/
│   │                   │   │   ├── ClusterManager.java
│   │                   │   │   ├── NodeDiscoveryService.java
│   │                   │   │   ├── GossipProtocolService.java
│   │                   │   │   ├── HeartbeatService.java
│   │                   │   │   └── FailureDetectionService.java
│   │                   │   ├── versioning/
│   │                   │   │   ├── VersionService.java
│   │                   │   │   └── LamportClockService.java
│   │                   │   └── communication/
│   │                   │       ├── MulticastService.java
│   │                   │       └── NodeCommunicationService.java
│   │                   │
│   │                   ├── repository/
│   │                   │   ├── SecretPartRepository.java
│   │                   │   ├── NodeStateRepository.java
│   │                   │   └── impl/
│   │                   │       ├── InMemorySecretPartRepository.java
│   │                   │       └── InMemoryNodeStateRepository.java
│   │                   │
│   │                   ├── domain/
│   │                   │   ├── model/
│   │                   │   │   ├── Secret.java
│   │                   │   │   ├── SecretPart.java
│   │                   │   │   ├── SecretKey.java
│   │                   │   │   ├── SecretVersion.java
│   │                   │   │   ├── Node.java
│   │                   │   │   └── NodeState.java
│   │                   │   └── enums/
│   │                   │       └── NodeStatus.java
│   │                   │
│   │                   ├── dto/
│   │                   │   ├── request/
│   │                   │   │   ├── StoreSecretRequest.java
│   │                   │   │   └── UpdateSecretRequest.java
│   │                   │   ├── response/
│   │                   │   │   ├── SecretResponse.java
│   │                   │   │   ├── SecretVersionResponse.java
│   │                   │   │   └── ClusterStatusResponse.java
│   │                   │   └── internal/
│   │                   │       ├── SecretPartMessage.java
│   │                   │       ├── HeartbeatMessage.java
│   │                   │       └── GossipMessage.java
│   │                   │
│   │                   ├── encrypt/
│   │                   │   ├── ShamirSecretSharing.java
│   │                   │   ├── SecretSplitter.java
│   │                   │   └── SecretReconstructor.java
│   │                   │
│   │                   ├── network/
│   │                   │   ├── MulticastHandler.java
│   │                   │   ├── UdpMessageHandler.java
│   │                   │   └── TcpConnectionManager.java
│   │                   │
│   │                   ├── exception/
│   │                   │   ├── GlobalExceptionHandler.java
│   │                   │   ├── SecretNotFoundException.java
│   │                   │   ├── InsufficientPartsException.java
│   │                   │   ├── NodeCommunicationException.java
│   │                   │   └── VersionConflictException.java
│   │                   │
│   │                   └── util/
│   │                       ├── ClockUtil.java
│   │                       ├── SecretKeyGenerator.java
│   │                       └── NetworkUtil.java
│   │
│   └── resources/
│       ├── application.properties
│       ├── application-dev.properties
│       ├── application-prod.properties
│       ├── cluster.properties
│       ├── static/
│       └── templates/
│
└── test/
    └── java/
        └── edu/
            └── yu/
                └── capstone/
                    └── DistributedSecretsVault/
                        ├── DistributedSecretsVaultApplicationTests.java
                        ├── controller/
                        │   └── SecretControllerTest.java
                        ├── service/
                        │   ├── secret/
                        │   │   ├── SecretServiceTest.java
                        │   │   └── SecretSharingServiceTest.java
                        │   ├── cluster/
                        │   │   ├── GossipProtocolServiceTest.java
                        │   │   └── HeartbeatServiceTest.java
                        │   └── versioning/
                        │       └── LamportClockServiceTest.java
                        ├── encrypt/
                        │   └── ShamirSecretSharingTest.java
                        ├── integration/
                        │   ├── SecretStorageIntegrationTest.java
                        │   └── ClusterCommunicationIntegrationTest.java
                        └── util/
                            └── TestDataBuilder.java

[MaxFdev]

Key organizational principles:

1. Controller Layer - REST endpoints for secret operations and cluster management
2. Service Layer - Organized by domain (secret, cluster, versioning, communication)
3. Repository Layer - Data access abstraction with implementation flexibility
4. Domain Layer - Core business entities and enums
5. DTO Layer - Separated into request/response for API and internal for cluster communication
6. Crypto Package - Shamir's Secret Sharing implementation
7. Network Package - Low-level network communication handlers
8. Exception Package - Centralized exception handling
9. Config Package - All Spring configuration classes
10. Util Package - Helper utilities

Testing structure mirrors the main structure for easy navigation and follows the same organizational patterns.

[MaxFdev]

More in depth from claude:
I'll provide a comprehensive explanation of each folder's purpose, tailored for developers who know Java but are learning Spring Boot.

Core Application

DistributedSecretsVaultApplication.java

Purpose: The entry point of your Spring Boot application.

• Contains the main() method that launches the application
• The @SpringBootApplication annotation does three things:
  • @Configuration: Marks this as a source of bean definitions
  • @EnableAutoConfiguration: Tells Spring Boot to auto-configure based on dependencies in pom.xml
  • @ComponentScan: Tells Spring to scan this package and sub-packages for components

---

Layer-by-Layer Breakdown

📁 config/ - Configuration Classes

Spring Boot Concept: Instead of XML configuration files, Spring Boot uses Java classes annotated with @Configuration to define how the application should be set up.

What goes here:

• SecurityConfig.java: Configures authentication, authorization, OAuth2 settings

  • Annotated with @Configuration and @EnableWebSecurity
  • Defines which endpoints require authentication
  • Sets up password encoders, authentication providers

• ClusterConfig.java: Cluster-specific settings

  • Number of nodes in the cluster
  • Threshold for Shamir's secret sharing
  • Cluster membership configuration

• NetworkConfig.java: Network communication settings

  • UDP multicast addresses and ports
  • TCP connection pool configuration
  • Timeout values for network operations

• StorageConfig.java: Storage backend configuration

  • In-memory storage settings
  • Future: Database connection pools if using persistent storage

Example:

@Configuration
public class ClusterConfig {
    @Value("${cluster.threshold}")
    private int threshold;
    
    @Bean
    public ClusterProperties clusterProperties() {
        return new ClusterProperties(threshold, ...);
    }
}

---

📁 controller/ - REST API Controllers

Spring Boot Concept: Controllers handle HTTP requests and return HTTP responses. They're the "entry point" for external clients to interact with your application.

What goes here:

• SecretController.java: Handles all secret-related endpoints

  • Annotated with @RestController (combines @Controller + @ResponseBody)
  • Methods annotated with @GetMapping, @PostMapping, @PutMapping, @DeleteMapping
  • Example endpoints:
    • PUT /api/v1/secrets - Store a new secret
    • GET /api/v1/secrets/{key} - Retrieve a secret
    • GET /api/v1/secrets/{key}/versions - Get all versions

• HealthController.java: Health check endpoints

  • GET /health - Is this node alive?
  • GET /health/cluster - Cluster health status

• ClusterController.java: Cluster management endpoints (admin only)

  • GET /api/v1/cluster/nodes - List all nodes
  • GET /api/v1/cluster/status - Cluster status

Key Pattern: Controllers should be thin - they receive requests, delegate to services, and return responses. No business logic here.

Example:

@RestController
@RequestMapping("/api/v1/secrets")
public class SecretController {
    
    @Autowired
    private SecretService secretService;
    
    @PostMapping
    public ResponseEntity<SecretVersionResponse> storeSecret(
            @RequestBody StoreSecretRequest request,
            @AuthenticationPrincipal User user) {
        SecretVersion version = secretService.storeSecret(
            user.getUsername(), 
            request.getKey(), 
            request.getValue()
        );
        return ResponseEntity.ok(new SecretVersionResponse(version));
    }
}

---

📁 service/ - Business Logic Layer

Spring Boot Concept: Services contain the core business logic. They're annotated with @Service, which marks them as Spring-managed beans that can be injected into controllers and other services.

This is where the "real work" happens. Organized into sub-packages by domain:

service/secret/ - Secret Management Logic

• SecretService.java: Interface defining the contract

  • SecretVersion storeSecret(String user, String key, String value)
  • String getSecret(String user, String key, Long version)
  • Map<Long, String> getAllVersions(String user, String key)

• SecretServiceImpl.java: Implementation of SecretService

  • Orchestrates the entire process of storing/retrieving secrets
  • Coordinates between other services (versioning, sharing, cluster communication)
  • Annotated with @Service

• SecretSharingService.java: Implements Shamir's Secret Sharing

  • List<SecretPart> split(String secret, int threshold, int totalParts)
  • Wraps the crypto package with service-level logic

• SecretReconstructionService.java: Rebuilds secrets from parts

  • String reconstruct(List<SecretPart> parts)
  • Validates that enough parts are available

Why separate interfaces from implementations? Makes testing easier (you can mock the interface) and allows multiple implementations.

service/cluster/ - Cluster Coordination

• ClusterManager.java: High-level cluster orchestration

  • Maintains the list of active nodes
  • Coordinates node joins and leaves
  • Decides which nodes get which secret parts

• NodeDiscoveryService.java: Finds other nodes in the cluster

  • Uses multicast to discover peers
  • Maintains a registry of known nodes

• GossipProtocolService.java: Implements gossip protocol

  • Periodically exchanges node state information with random peers
  • Propagates information about node failures
  • Uses eventual consistency model

• HeartbeatService.java: Sends/receives heartbeats

  • Annotated with @Scheduled for periodic execution
  • Sends "I'm alive" messages to other nodes
  • Tracks when other nodes last sent heartbeats

• FailureDetectionService.java: Detects when nodes fail

  • Monitors heartbeat timeouts
  • Combines gossip information with local observations
  • Triggers recovery procedures

service/versioning/ - Version Management

• VersionService.java: Manages secret versions

  • Retrieves specific versions or latest version

• LamportClockService.java: Implements Lamport logical clocks

  • Ensures causal ordering in a distributed system
  • Each node maintains its own clock
  • Increments on events, synchronizes with received messages

service/communication/ - Node-to-Node Communication

• MulticastService.java: UDP multicast for broadcasting

  • Sends requests to all nodes simultaneously
  • Used when retrieving secret parts

• NodeCommunicationService.java: Direct node-to-node messages

  • TCP communication for reliable message delivery
  • Handles retries and timeouts

---

📁 repository/ - Data Access Layer

Spring Boot Concept: Repositories abstract away data storage details. If you later switch from in-memory to a database, you only change the repository implementation, not the services.

What goes here:

• SecretPartRepository.java: Interface for storing/retrieving secret parts

  public interface SecretPartRepository {
      void save(SecretKey key, SecretPart part);
      Optional<SecretPart> findByKey(SecretKey key);
      List<SecretPart> findAllByUserAndKey(String user, String key);
      void delete(SecretKey key);
  }

• NodeStateRepository.java: Interface for node state data

• impl/InMemorySecretPartRepository.java: Implementation using ConcurrentHashMap

  • Thread-safe in-memory storage
  • Fast but volatile (data lost on restart)

Why this pattern? Tomorrow you might want PostgreSQL instead. You'd create PostgresSecretPartRepository.java implementing the same interface, and services wouldn't need any changes.

---

📁 domain/model/ - Core Business Objects

Java Concept: These are your domain entities - the core objects your business logic operates on. They're POJOs (Plain Old Java Objects) with no Spring annotations.

What goes here:

• Secret.java: Represents a complete secret

  public class Secret {
      private String user;
      private String key;
      private String value;
      private Long version;
      private LocalDateTime createdAt;
      // getters, setters, constructors
  }

• SecretPart.java: One shard of a secret after Shamir splitting

  public class SecretPart {
      private int partNumber;  // Which part (1 of N)
      private byte[] data;      // The actual shard
      private String algorithm; // "SHAMIR_SSS"
      // ...
  }

• SecretKey.java: Composite key for looking up secrets

  public class SecretKey {
      private String user;
      private String key;
      private Long version;
      // Used as map key, needs equals() and hashCode()
  }

• Node.java: Represents a cluster node

  public class Node {
      private String nodeId;
      private String ipAddress;
      private int port;
      private NodeStatus status;
      private long lastHeartbeat;
  }

📁 domain/enums/ - Enumerations

• NodeStatus.java:

  public enum NodeStatus {
      ACTIVE,
      SUSPECTED,  // Gossip indicates might be down
      FAILED,
      JOINING,
      LEAVING
  }

---

📁 dto/ - Data Transfer Objects

Spring Boot Concept: DTOs are objects specifically designed for transferring data over the network. They're different from domain models because:

1. They match your API contract (what clients see)
2. They can combine/flatten data from multiple domain objects
3. They can hide sensitive fields
4. They can include validation annotations

dto/request/ - Incoming API Requests

• StoreSecretRequest.java:

  public class StoreSecretRequest {
      @NotBlank(message = "Key cannot be empty")
      private String key;
      
      @NotBlank(message = "Value cannot be empty")
      private String value;
      
      // Jackson will deserialize JSON to this object
  }

dto/response/ - Outgoing API Responses

• SecretResponse.java:

  public class SecretResponse {
      private String key;
      private String value;
      private Long version;
      private LocalDateTime retrievedAt;
      
      // Jackson will serialize this object to JSON
  }

dto/internal/ - Node-to-Node Messages

These are NOT for external clients - they're for cluster communication.

• SecretPartMessage.java: When requesting/sending secret parts between nodes
• HeartbeatMessage.java: Heartbeat payload
• GossipMessage.java: Gossip protocol messages

Why separate DTOs from domain models?

• Domain models can change internally without breaking your API
• API responses can aggregate multiple domain objects
• Different endpoints can return different views of the same data

---

📁 encrypt/ - Cryptography

Pure Java implementations of cryptographic operations. No Spring dependencies here.

• ShamirSecretSharing.java: Core Shamir's algorithm implementation

  • Splits secrets using polynomial interpolation
  • Reconstructs secrets from threshold number of parts

• SecretSplitter.java: High-level API for splitting

• SecretReconstructor.java: High-level API for reconstruction

Why separate this? This is complex mathematical code that should be independently testable and reusable.

---

📁 network/ - Low-Level Networking

Handles the actual network I/O. These might not be Spring beans - they might be plain Java classes or use Java NIO.

• MulticastHandler.java: Sets up UDP multicast sockets

  • Joins multicast groups
  • Sends/receives datagrams

• UdpMessageHandler.java: Handles UDP message serialization

• TcpConnectionManager.java: Manages TCP connection pool

  • Reuses connections to other nodes
  • Handles connection failures and retries

---

📁 exception/ - Exception Handling

Spring Boot Concept: Centralized exception handling makes your controllers cleaner.

• GlobalExceptionHandler.java:

  @RestControllerAdvice  // Applied to all controllers
  public class GlobalExceptionHandler {
      
      @ExceptionHandler(SecretNotFoundException.class)
      public ResponseEntity<ErrorResponse> handleSecretNotFound(
              SecretNotFoundException ex) {
          return ResponseEntity
              .status(HttpStatus.NOT_FOUND)
              .body(new ErrorResponse(ex.getMessage()));
      }
      
      @ExceptionHandler(InsufficientPartsException.class)
      public ResponseEntity<ErrorResponse> handleInsufficientParts(
              InsufficientPartsException ex) {
          return ResponseEntity
              .status(HttpStatus.SERVICE_UNAVAILABLE)
              .body(new ErrorResponse(ex.getMessage()));
      }
  }

• Custom Exceptions: Each represents a specific business error

  • Extend RuntimeException for unchecked exceptions
  • Include meaningful error messages
  • Can include error codes for clients

---

📁 util/ - Utility Classes

Helper classes with static methods - no state, just pure functions.

• ClockUtil.java: Date/time utilities
• SecretKeyGenerator.java: Generate unique keys
• NetworkUtil.java: IP address parsing, validation

Pattern: Keep these classes stateless and make methods static when possible.

---

Resources Folder

📁 resources/

• application.properties: Main configuration file

  server.port=8080
  cluster.threshold=3
  cluster.total-nodes=5
  multicast.address=239.255.0.1
  multicast.port=4446

• application-dev.properties: Development-specific settings

  • Activated with spring.profiles.active=dev
  • Might have verbose logging, test data

• application-prod.properties: Production settings

  • Activated with spring.profiles.active=prod
  • Performance tuned, minimal logging

• cluster.properties: Node-specific configuration

  • Each node might have different settings

---

Test Structure

📁 test/java/.../

Mirrors your main structure for easy navigation.

• controller/: Controller tests

  • Use @WebMvcTest to test only the web layer
  • Mock the service layer with @MockBean

• service/: Service tests

  • Use @SpringBootTest for integration tests
  • Use plain JUnit + Mockito for unit tests
  • Mock repositories and other dependencies

• integration/: Full integration tests

  • @SpringBootTest with real Spring context
  • Might start actual network services
  • Tests multiple layers working together

• util/TestDataBuilder.java: Test data factory

  • Builder pattern for creating test objects
  • Makes tests more readable

---

How These Layers Interact

Client HTTP Request
    ↓
Controller (validates, delegates)
    ↓
Service (business logic, orchestration)
    ↓ ↓ ↓
├─ Repository (data access)
├─ Other Services (collaborators)
└─ Crypto/Network (utilities)
    ↓
Response back up the chain

Key Spring Boot Principles:

1. Dependency Injection: Don't use new for Spring beans. Use @Autowired or constructor injection:

   @Service
   public class SecretServiceImpl {
       private final SecretPartRepository repository;
       
       // Constructor injection (preferred)
       public SecretServiceImpl(SecretPartRepository repository) {
           this.repository = repository;
       }
   }

2. Separation of Concerns: Each layer has one job:

   • Controllers: HTTP ↔ Java objects
   • Services: Business logic
   • Repositories: Data access
   • Domain: Pure business objects

3. Configuration over Code: Use application.properties instead of hardcoding values

4. Convention over Configuration: Follow Spring Boot's expected structure and it auto-wires a lot for you

This structure will scale as your project grows and makes it clear where new code should go!

[NoamBenS]

Config:

1. We want to also create a system that allows invokers to set data like the Shamir threshold and node cluster size. We can implement this via setting variables in external and organized ways, e.g.:

• Keep using Spring @value or @ConfigurationProperties and read from application.yml / .properties.
• Users override values via environment variables or command-line args (--cluster.threshold=3).

2. Storage Config should also store database data like Postgres configuration data

Controller:

1. Cluster Health could be implemented via getting all data from one node, or getting individual data from each node. Requires some kind of authoritative state on cluster subscriptions.