📁 File Portal (Microservices-Based File Management System)

A distributed, microservices-based company document portal that enables secure uploading, organizing, processing, and sharing of internal documents among authenticated users.

Built for the Web Applications II Final Project.

📑 Table of Contents

✨ Key Features
🏗️ System Architecture
🔒 Security & Authentication
📂 Microservices Breakdown
🔄 End-to-End Operational Flows
📊 Observability, Metrics & Tracing
🚀 How to Run the Project
🛠️ Environment Configuration
🔌 Port Summary

✨ Key Features

Secure File Management: Role-based access control with sharing capabilities.
Resumable Uploads: Robust file uploads powered by the TUS protocol.
AI-Powered Insights: Automated metadata generation and text extraction using Google Gemini.
Real-Time Processing: Kafka-driven asynchronous workflows for seamless operations.
Comprehensive Observability: Full tracing and monitoring via OpenTelemetry, Prometheus, Grafana, and Langtrace.

🏗️ System Architecture

Our system is built using a modern, distributed microservices architecture to ensure high scalability, availability, and maintainability.

graph TB
    subgraph "Frontend"
        FE["PortalFrontend<br/>(React + Vite)<br/>:5173"]
    end

    subgraph "API Layer"
        GW["API Gateway<br/>(Spring Cloud Gateway)<br/>:8082"]
    end

    subgraph "Backend Microservices"
        FS["FileService<br/>(Spring Boot)<br/>:8080"]
        UDS["UserDetailService<br/>(Spring Boot)<br/>:8081"]
        AI["AIProcessingService<br/>(Spring Boot)<br/>:8083"]
    end

    subgraph "Infrastructure"
        KC["Keycloak<br/>(IAM/SSO)<br/>:9090"]
        PG["PostgreSQL<br/>(Multi-DB)<br/>:5432"]
        MINIO["MinIO<br/>(Object Storage)<br/>:9000"]
        KAFKA["Apache Kafka<br/>(Message Broker)<br/>:9092"]
        KCONN["Kafka Connect<br/>+ Debezium<br/>:8089"]
    end

    subgraph "Observability"
        PROM["Prometheus<br/>:9091"]
        GRAF["Grafana<br/>:3000"]
        TEMPO["Tempo<br/>:3200"]
        OTEL["OTel Collector<br/>:4319"]
        LANG["Langtrace<br/>:3001"]
    end

    FE -->|"HTTP (via browser)"| GW
    GW -->|"TokenRelay + Routing"| FS
    GW -->|"TokenRelay + Routing"| UDS
    GW -->|"/api/auth/userinfo"| GW

    FS -->|"REST (M2M)"| UDS
    AI -->|"REST (M2M)"| FS

    FS -->|"Outbox → DB"| PG
    KCONN -->|"CDC (Debezium)"| PG
    KCONN -->|"Produce events"| KAFKA
    KAFKA -->|"Consume events"| AI

    FS -->|"Upload/Download"| MINIO
    AI -->|"Download (via FS)"| MINIO

    GW -->|"OAuth2 Login"| KC
    FS -->|"JWT Validation"| KC
    UDS -->|"JWT Validation"| KC
    AI -->|"Client Credentials"| KC

    FS -->|"Metrics"| PROM
    UDS -->|"Metrics"| PROM
    GW -->|"Metrics"| PROM
    AI -->|"Metrics"| PROM

    FS -->|"Traces"| OTEL
    UDS -->|"Traces"| OTEL
    AI -->|"Traces"| OTEL
    GW -->|"Traces"| OTEL
    OTEL -->|"Forward"| TEMPO
    OTEL -->|"AI Traces"| LANG
    PROM --> GRAF
    TEMPO --> GRAF

🔒 Security & Authentication

The system uses Keycloak (Port 9090) as the central Identity and Access Management (IAM) provider, configuring a single realm (file-management).

Roles (RBAC)

admin: Full access to the entire system.
manager: Full management over files belonging to their department.
staff: Base access to own files, shared files, and department files (read-only).
serviceaccount-aiservice & serviceaccount-fileservice: Dedicated machine-to-machine (M2M) roles.

Authentication Flow

OAuth2 Login: The Frontend redirects unauthenticated users to the API Gateway (/oauth2/authorization/keycloak), which initiates the Authorization Code Flow with PKCE via Keycloak.
Session Management: Upon successful login, the API Gateway establishes a secure session (via JSESSIONID cookie) with the browser.
Token Relay: For every request routed to backend microservices, the API Gateway automatically propagates the user's JWT access token using the TokenRelay filter.
JWT Validation: Each backend service (Resource Server) independently validates the JWT's signature and issuer against Keycloak's public keys. A custom JWTConverter maps Keycloak realm roles into Spring Security GrantedAuthority objects (e.g., ROLE_admin).
Machine-to-Machine (M2M): Background services like AIProcessingService authenticate using the Client Credentials Flow to obtain their own M2M JWT tokens to securely call other internal services (e.g., downloading files from FileService).

📂 Microservices Breakdown

1. API Gateway (Port 8082)

Tech: Spring Cloud Gateway MVC
Role: Single entry point. Handles OAuth2 login, cookie-based session management, CORS configuration for the React frontend, and dynamic HTTP routing to the backend services.

2. FileService (Port 8080)

Tech: Spring Boot + Kotlin + Spring Data JPA + MinIO SDK
Role: The core monolith managing the entire document lifecycle.
Features:
- TUS Protocol Uploads: Supports resumable, chunked file uploads directly to MinIO.
- Transactional Outbox Pattern: When a file is uploaded, an OutboxEvent is saved to PostgreSQL in the exact same transaction. Debezium reads the WAL and guarantees exactly-once publication to Kafka, preventing missing events.
- Cleanup Service: A scheduled background task sweeps orphaned TUS chunks and expired sessions every hour.
- Access Control List (ACL): Validates read/write permissions based on ownership, department, or explicit file-sharing rules.

3. UserDetailService (Port 8081)

Tech: Spring Boot + Kotlin + Spring Data JPA
Role: Manages extended organizational user profiles (departments, quotas, specific notes) beyond the basic Keycloak identity. It exposes APIs for cross-service department resolution and user lookups.

4. AIProcessingService (Port 8083)

Tech: Spring Boot + Spring AI (Google Gemini) + Kafka Consumer
Role: A dedicated, asynchronous worker that listens for Kafka events (file-upload.completed.*).
Features:
- Downloads the file from MinIO, extracts text (via PDFBox or Apache POI), and sends a structured prompt to Google Gemini.
- Generates AI metadata: Category, Description, Summary, Tags, and Sensitivity.
- Fault Tolerance: Utilizes Kafka retry backoffs for transient errors (e.g., API rate limits) and routes permanently failed messages to a Dead Letter Topic (DLT) for fallback processing, ensuring the system never hangs.

5. PortalFrontend (Port 5173)

Tech: React 18 + TypeScript + Vite + TailwindCSS
Role: The Single Page Application UI. Communicates exclusively with the API Gateway. Implements the TUS client for robust uploads and provides a rich aesthetic for viewing, sharing, and searching files.
Features:
- Single-pane Admin Panel: Consolidated user management and real-time system observability metrics via embedded Grafana.
- Granular Shared File Permissions: File actions (Edit Metadata, View History) are automatically adjusted per-file based on specific READ/WRITE sharing permissions.
- Real-time AI Polling: Uses reactive polling across all views (Personal, Department, Global, and Shared) to dynamically update file lists the moment backend AI processing completes, providing a seamless user experience.

🔄 End-to-End Operational Flows

1. Document Upload & AI Processing (Sync + Async)

sequenceDiagram
    actor U as User
    participant FE as Frontend
    participant GW as API Gateway
    participant FS as FileService
    participant PG as PostgreSQL
    participant S3 as MinIO
    participant DEB as Debezium
    participant KF as Kafka
    participant AI as AIProcessingService
    participant LLM as Google Gemini

    U->>FE: Select file & click "Upload"
    FE->>GW: POST /api/uploads (TUS headers)
    GW->>FS: POST /uploads (Token Relay)
    FS->>PG: Save UploadSession
    FS-->>FE: 201 + Location

    loop Per chunk
        FE->>GW: PATCH /api/uploads/{id} (chunk bytes)
        GW->>FS: PATCH /uploads/{id}
        FS->>S3: putObject (chunk)
        FS->>PG: Update offset
        FS-->>FE: 204
    end

    Note over FE,FS: Finalization
    FE->>GW: PATCH /api/uploads/{id} (last chunk)
    GW->>FS: PATCH /uploads/{id}
    FS->>S3: Combine chunks → final file
    FS->>PG: Create File + FileVersion
    FS->>PG: INSERT OutboxEvent (same transaction)
    FS->>S3: Delete temporary chunks
    FS-->>FE: 204

    Note over DEB,LLM: Asynchronous AI Pipeline
    DEB->>PG: Read WAL (CDC)
    DEB->>KF: Publish to topic
    KF->>AI: Consume event
    AI->>FS: GET /{id}/versions/{vid}/download (M2M)
    FS->>S3: Fetch file
    FS-->>AI: File bytes
    AI->>LLM: Analyze with structured prompt
    LLM-->>AI: JSON (category, summary, tags, ...)
    AI->>FS: POST /{vid}/ai-metadata (M2M)
    FS->>PG: Update FileVersion with AI metadata

2. Advanced Search & Department Resolution

sequenceDiagram
    actor U as User
    participant FE as Frontend
    participant GW as API Gateway
    participant FS as FileService
    participant UDS as UserDetailService
    participant PG as PostgreSQL

    U->>FE: Enters query in SearchBar
    FE->>GW: GET /api/v1/files/search?query=...&scope=department
    GW->>FS: GET (Token Relay)
    FS->>UDS: GET /{userId}/department (M2M)
    UDS-->>FS: "Engineering"
    FS->>UDS: GET /department/Engineering (M2M)
    UDS-->>FS: [uuid1, uuid2, uuid3]
    FS->>PG: Dynamic JPA Query (query, tags, sensitivity, scope, department)
    FS-->>FE: Page<FileDTO>

📊 Observability, Metrics & Tracing

The system implements a comprehensive observability stack based on three pillars: metrics, traces, and AI logs.

The Flow

Instrumentation & Tracing: OpenTelemetry automatically generates a unique Trace ID when a request enters the Gateway. Every downstream service hop (HTTP or Kafka) creates a Span. All microservices push their spans directly to the OpenTelemetry (OTel) Collector (port 4319).
Trace Routing: The OTel Collector acts as a proxy router, forwarding every trace to Tempo (for end-to-end system distributed tracing) and a filtered subset of traces (excluding background Spring Boot noise) to Langtrace (for specialized AI/LLM tracing).
Metrics Collection: Spring Boot Actuator and Micrometer expose in-memory metrics on the /actuator/prometheus endpoint. This includes custom metrics like ai.processing.success and ai.processing.duration.
Metrics Scraping: Prometheus scrapes these endpoints every 15 seconds.
Visualization: Developers use Grafana (port 3000) to view real-time system health and TraceQL queries, and Langtrace UI (port 3001) to inspect exact LLM prompts, token usage, and AI model latency.

🚀 How to Run the Project

We have built a custom, interactive CLI tool to make running this complex microservices architecture incredibly simple.

Prerequisites

Ensure you have the following installed on your system:

Docker & Docker Compose (Ensure the Docker daemon is running)
Java 21 (or your preferred JDK distribution)
Node.js (v18+)
Make (Pre-installed on macOS/Linux; available via choco install make on Windows)

Quick Start

Open your terminal at the root of the project and run:

make quick-menu

This will launch the File Portal Control Center, an interactive terminal interface where you can:

1: Start all background services safely (Infrastructure + Spring Boot + Frontend).
2: Stop all services cleanly.
3-6: Restart individual microservices independently without taking down the whole system.
9: Clean logs, build folders, and node_modules safely.
0: Purge the entire system (WARNING: Destroys databases and Docker volumes).

Manual Makefile Commands

If you prefer standard terminal commands without the interactive menu:

make start-infra (Starts only Docker containers: Postgres, MinIO, Kafka, Keycloak)
make start-all (Starts infrastructure, then compiles and runs all Spring Boot and React apps in the background)
make start-verbose (Runs everything in the foreground using concurrently so you can see all logs side-by-side)
make stop-all (Gracefully stops all JVMs, node servers, and docker containers)

🛠️ Environment Configuration

You must create a .env file at the root of the project with the following keys for the AI Service to work:

GOOGLE_API_KEY=your_gemini_api_key_here

(The Makefile will automatically read this file and inject it into the Spring Boot processes).

🔌 Port Summary

Service	Port	Description
FileService	`8080`	Main Backend
UserDetailService	`8081`	User Management
API Gateway	`8082`	Entry Point
AIProcessingService	`8083`	AI Worker
PortalFrontend	`5173`	React SPA
PostgreSQL	`5432`	Database
MinIO	`9000` / `9001`	Object Storage / UI
Keycloak	`9090`	IAM/SSO
Kafka / Kafka UI	`9092` / `8088`	Message Broker / UI
Kafka Connect	`8089`	CDC Connector
Prometheus / Grafana	`9091` / `3000`	Metrics & Dashboards
Tempo / OTel Collector	`3200` / `4319`	Tracing Backend / Collector
Langtrace	`3001`	AI Monitoring

Name		Name	Last commit message	Last commit date
AIProcessingService		AIProcessingService
ApiGateway		ApiGateway
FileService		FileService
PortalFrontend		PortalFrontend
UserDetailService		UserDetailService
.gitignore		.gitignore
Dockerfile.connect		Dockerfile.connect
Makefile		Makefile
README.md		README.md
compose.yaml		compose.yaml
guidance.md		guidance.md
init-multiple-databases.sh		init-multiple-databases.sh
master-realm.json		master-realm.json
quick-menu.sh		quick-menu.sh
realm-export.json		realm-export.json
test.sh		test.sh

Navigation Menu

Repository navigation

About

Repository files navigation

📁 File Portal (Microservices-Based File Management System)

📑 Table of Contents

✨ Key Features

🏗️ System Architecture

🔒 Security & Authentication

Roles (RBAC)

Authentication Flow

📂 Microservices Breakdown

1. API Gateway (Port 8082)

2. FileService (Port 8080)

3. UserDetailService (Port 8081)

4. AIProcessingService (Port 8083)

5. PortalFrontend (Port 5173)

🔄 End-to-End Operational Flows

1. Document Upload & AI Processing (Sync + Async)

2. Advanced Search & Department Resolution

📊 Observability, Metrics & Tracing

The Flow

🚀 How to Run the Project

Prerequisites

Quick Start

Manual Makefile Commands

🛠️ Environment Configuration

🔌 Port Summary

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

📁 File Portal (Microservices-Based File Management System)

📑 Table of Contents

✨ Key Features

🏗️ System Architecture

🔒 Security & Authentication

Roles (RBAC)

Authentication Flow

📂 Microservices Breakdown

1. API Gateway (Port 8082)

2. FileService (Port 8080)

3. UserDetailService (Port 8081)

4. AIProcessingService (Port 8083)

5. PortalFrontend (Port 5173)

🔄 End-to-End Operational Flows

1. Document Upload & AI Processing (Sync + Async)

2. Advanced Search & Department Resolution

📊 Observability, Metrics & Tracing

The Flow

🚀 How to Run the Project

Prerequisites

Quick Start

Manual Makefile Commands

🛠️ Environment Configuration

🔌 Port Summary

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages