DBMS - Database Management System

A comprehensive web-based application for uploading, managing, and viewing documents in AWS S3 with advanced folder structure support, search functionality, and secure authentication.

🚀 Project Overview

The DBMS (Database Management System) is a Flask-based web application that provides a secure, user-friendly interface for managing files and folders in AWS S3. It features real-time upload progress tracking, advanced search capabilities, and a modern responsive UI.

✨ Key Features

• 🔐 Secure Authentication: JWT-based login system with session management
• 🔍 Advanced Search: Real-time search through files and folders with filtering
• 📁 Multiple Folder Upload: Bulk upload with folder structure preservation
• 📂 Folder Management: Hierarchical folder organization and management
• 📄 File Browser: Modern, responsive interface for file operations
• ☁️ Direct S3 Integration: Seamless AWS S3 bucket integration
• ⚡ File Operations: View, download, and delete files with confirmation
• 📊 Progress Tracking: Real-time upload progress with detailed statistics
• 🎨 Modern UI: Beautiful, responsive design with excellent UX
• 🔒 Security: Environment-based AWS credentials and secure file handling
• 📈 Analytics: Folder size tracking and file count statistics
• 📋 Reporting: Excel report generation for S3 folder analysis

🏗️ Project Architecture

Core Components

1. Flask Web Application (app.py)

• Main Application Server: Flask-based web server handling HTTP requests
• Authentication System: JWT-based session management with login/logout functionality
• Route Handlers: RESTful API endpoints for file operations
• S3 Integration: Direct AWS S3 client integration for file operations
• File Validation: Security checks for file types and upload validation

2. S3 Folder Checker (s3_folder_checker.py)

• Analytics Module: Standalone utility for S3 bucket analysis
• Excel Reporting: Automated Excel report generation with folder statistics
• Data Processing: S3 metadata extraction and processing
• Configuration Management: Environment variable handling for AWS credentials

3. Frontend Templates

• Login Interface (templates/login.html): Secure authentication form with modern UI
• Main Dashboard (templates/index.html): Comprehensive file management interface
• Responsive Design: Mobile-friendly layouts with CSS Grid and Flexbox
• Interactive Elements: JavaScript-powered upload progress and file operations

4. Configuration & Deployment

• Environment Configuration: .env file for secure credential management
• Dependency Management: requirements.txt with version-pinned packages
• Startup Scripts: start_app.bat for Windows deployment automation
• Static Assets: CSS styling and JavaScript functionality

Technical Stack

• Backend: Python 3.7+, Flask 2.3.3, Boto3 1.28.62
• Frontend: HTML5, CSS3, Vanilla JavaScript
• Cloud: AWS S3, AWS IAM
• Security: JWT tokens, Environment variables, File validation
• Reporting: OpenPyXL for Excel generation
• Deployment: Windows batch scripts, pip package management

Prerequisites

• Python 3.7 or higher
• AWS S3 bucket with appropriate permissions
• AWS access credentials (Access Key ID, Secret Access Key)
• AWS IAM user with S3 read/write permissions

Setup Instructions

1. Install Dependencies

Run the batch file to install all required packages:

start_app.bat

Or manually install:

pip install -r requirements.txt

2. Configure Environment Variables

Your .env file is already configured with:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION
• S3_BUCKET_NAME
• JWT_SECRET

3. Run the Application

Option 1: Use the batch file

start_app.bat

Option 2: Run manually

python app.py

4. Access the Application

Open your web browser and go to:

http://localhost:5000

5. Login Credentials

Use the following credentials to access the system:

• Username: theops
• Password: theopss

Usage

Using the System

1. Login: Enter your credentials to access the system
2. Search: Use the search bar to find specific files or folders
3. Upload Files:
   • Select a folder from the dropdown (or leave empty for root)
   • Choose multiple files to upload
   • Click "Upload Files" button
4. Create Folders: Use the "Create New Folder" section to organize your files
5. Manage Files: View, download, or delete files using the action buttons

Managing Files

• View Files: Click the eye icon to view files in a new tab
• Download Files: Click the download icon to download files
• Delete Files: Click the trash icon to delete files (with confirmation)

Creating Folders

1. Click the "+" button next to the folder dropdown
2. Enter the folder name in the modal
3. Click "Create Folder"

File Types Supported

The application supports various file types including:

• Documents: PDF, DOC, DOCX, TXT, CSV
• Images: PNG, JPG, JPEG, GIF
• Spreadsheets: XLS, XLSX
• Presentations: PPT, PPTX
• Archives: ZIP, RAR, 7Z
• Media: MP4, AVI, MOV, MP3, WAV
• Data: JSON, XML

🔌 API Endpoints

Authentication Endpoints

• GET /login - Display login page

  • Response: HTML login form
  • Authentication: Not required

• POST /login - Authenticate user credentials

  • Request Body: username, password
  • Response: Redirect to dashboard or error message
  • Authentication: Not required

• GET /logout - Logout and clear session

  • Response: Redirect to login page
  • Authentication: Required

Main Application Endpoints

• GET / - Main dashboard interface

  • Response: HTML dashboard with folders and files
  • Authentication: Required
  • Features: Search functionality, file management interface

• POST /upload - Upload individual files

  • Request Body: files[], target_folder (optional)
  • Response: Redirect with success/error message
  • Authentication: Required
  • File Types: Allowed extensions only

• POST /upload_folder - Upload entire folder structure

  • Request Body: folder_files[] (multiple files with folder structure)
  • Response: Redirect with upload statistics
  • Authentication: Required
  • Features: Duplicate folder detection, progress tracking

• GET /delete/<file_key> - Delete file from S3

  • Parameters: file_key (S3 object key)
  • Response: Redirect with success/error message
  • Authentication: Required

API Data Endpoints

• GET /api/folders - Get folders list (JSON)

  • Response: JSON array of folder objects
  • Authentication: Required
  • Data: Folder name, size, file count, last modified

• GET /api/files - Get files list (JSON)

  • Response: JSON array of file objects
  • Authentication: Required
  • Data: File key, name, folder, size, URL, last modified

• POST /api/upload_progress - Real-time upload progress

  • Request Body: folder_name, progress, status
  • Response: JSON success/error response
  • Authentication: Required
  • Purpose: Progress tracking for large uploads

Security Notes

• Authentication Required: All routes except login require authentication
• Session Management: Secure session handling with JWT secret
• AWS Credentials: Loaded from environment variables for security
• File Validation: Uploads are validated for allowed extensions
• Filename Sanitization: All filenames are sanitized for security
• HTTPS Recommended: Use HTTPS in production environments

Troubleshooting

Common Issues

1. AWS Credentials Error: Verify your .env file has correct AWS credentials
2. Bucket Access Error: Ensure your AWS user has S3 permissions for the bucket
3. File Upload Fails: Check file size limits and network connection
4. Port Already in Use: The app runs on port 5000 by default

Error Messages

• Check the flash messages at the top of the page for detailed error information
• Server logs are printed to the console where you ran the application

📁 Project Structure

DBMS-Project/
├── 📄 app.py                    # Main Flask application server
│   ├── Authentication & Session Management
│   ├── S3 Integration & File Operations
│   ├── Route Handlers & API Endpoints
│   └── File Validation & Security
│
├── 📄 s3_folder_checker.py      # S3 Analytics & Reporting Module
│   ├── Bucket Analysis Functions
│   ├── Excel Report Generation
│   ├── Metadata Processing
│   └── Configuration Management
│
├── 📁 templates/                # Frontend Templates
│   ├── 📄 login.html           # Authentication Interface
│   │   ├── Login Form & Validation
│   │   ├── Password Toggle Feature
│   │   ├── Flash Message Display
│   │   └── Responsive Design
│   │
│   └── 📄 index.html           # Main Dashboard Interface
│       ├── File Upload Interface
│       ├── Folder Management
│       ├── Search & Filtering
│       ├── Progress Tracking
│       ├── File Operations (View/Download/Delete)
│       └── Real-time Updates
│
├── 📁 static/                   # Static Assets (if any)
│   ├── CSS files
│   ├── JavaScript files
│   └── Images/Icons
│
├── 📄 requirements.txt          # Python Dependencies
│   ├── Flask==2.3.3
│   ├── boto3==1.28.62
│   ├── python-dotenv==1.0.0
│   └── Werkzeug==2.3.7
│
├── 📄 .env                      # Environment Configuration
│   ├── AWS_ACCESS_KEY_ID
│   ├── AWS_SECRET_ACCESS_KEY
│   ├── AWS_DEFAULT_REGION
│   ├── S3_BUCKET_NAME
│   └── JWT_SECRET
│
├── 📄 start_app.bat             # Windows Deployment Script
│   ├── Dependency Installation
│   ├── Environment Setup
│   └── Application Startup
│
├── 📁 __pycache__/              # Python Cache Files
│   └── Compiled Python modules
│
└── 📄 README.md                 # Project Documentation
    ├── Setup Instructions
    ├── API Documentation
    ├── Architecture Overview
    └── Workflow Diagrams

Component Relationships

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Frontend      │    │   Flask App      │    │   AWS S3        │
│   (Templates)   │◄──►│   (app.py)       │◄──►│   (Cloud)       │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   User          │    │   S3 Checker     │    │   File Storage  │
│   Interface     │    │   (Analytics)    │    │   & Metadata    │
└─────────────────┘    └──────────────────┘    └─────────────────┘

🔄 Detailed Workflow Chart

The following diagram illustrates the complete workflow of the DBMS application, showing all major processes, decision points, and data flows:

graph TD
    A[User Access] --> B{Authentication Required?}
    B -->|No| C[Login Page]
    B -->|Yes| D[Main Dashboard]
    
    C --> E[Enter Credentials]
    E --> F{Valid Credentials?}
    F -->|No| G[Show Error Message]
    F -->|Yes| H[Create Session]
    G --> C
    H --> D
    
    D --> I[Load S3 Data]
    I --> J[Display Folders & Files]
    J --> K{User Action}
    
    K -->|Search| L[Filter Results]
    L --> J
    
    K -->|Upload Files| M[Select Files]
    M --> N[Choose Target Folder]
    N --> O[Validate Files]
    O --> P{Valid Files?}
    P -->|No| Q[Show Error]
    P -->|Yes| R[Upload to S3]
    Q --> D
    R --> S[Update Progress]
    S --> T{Upload Complete?}
    T -->|No| S
    T -->|Yes| U[Show Success Message]
    U --> D
    
    K -->|Upload Folders| V[Select Folder Structure]
    V --> W[Analyze Folder Contents]
    W --> X[Check Existing Folders]
    X --> Y{Duplicate Folders?}
    Y -->|Yes| Z[Skip Existing]
    Y -->|No| AA[Upload New Folders]
    Z --> BB[Show Skip Message]
    AA --> CC[Track Progress]
    CC --> DD[Update Statistics]
    BB --> D
    DD --> D
    
    K -->|View File| EE[Generate S3 URL]
    EE --> FF[Open in New Tab]
    
    K -->|Download File| GG[Generate Download URL]
    GG --> HH[Trigger Download]
    
    K -->|Delete File| II[Confirm Deletion]
    II --> JJ{Confirmed?}
    JJ -->|No| D
    JJ -->|Yes| KK[Delete from S3]
    KK --> LL[Show Success Message]
    LL --> D
    
    K -->|Logout| MM[Clear Session]
    MM --> C
    
    %% S3 Analytics Workflow
    NN[Run S3 Checker] --> OO[Load Environment]
    OO --> PP[Connect to S3]
    PP --> QQ[List Bucket Contents]
    QQ --> RR[Process Folder Data]
    RR --> SS[Generate Excel Report]
    SS --> TT[Save Report File]
    
    %% Error Handling
    R --> UU{Upload Error?}
    UU -->|Yes| VV[Show Error Details]
    VV --> D
    
    KK --> WW{Delete Error?}
    WW -->|Yes| XX[Show Error Details]
    XX --> D
    
    %% Styling
    classDef startEnd fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    classDef process fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    classDef decision fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px
    classDef success fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px
    
    class A,C,D startEnd
    class E,H,I,J,M,N,O,R,S,V,W,X,AA,CC,DD,EE,FF,GG,HH,II,KK,LL,MM,OO,PP,QQ,RR,SS,TT process
    class B,F,K,P,T,Y,JJ,UU,WW decision
    class G,Q,VV,XX error
    class U,BB,LL success

Loading

Workflow Components Explained

🔐 Authentication Flow

1. User Access: User navigates to the application
2. Session Check: System verifies if user is already authenticated
3. Login Process: If not authenticated, user enters credentials
4. Validation: System validates username/password against stored credentials
5. Session Creation: Upon successful authentication, JWT session is created
6. Dashboard Access: User gains access to main application interface

📁 File Upload Workflow

1. File Selection: User selects files from local system
2. Folder Selection: User chooses target folder (optional)
3. File Validation: System checks file types against allowed extensions
4. S3 Upload: Valid files are uploaded directly to AWS S3
5. Progress Tracking: Real-time progress updates with statistics
6. Completion: Success/error messages displayed to user

📂 Folder Upload Workflow

1. Folder Selection: User selects entire folder structure
2. Content Analysis: System analyzes folder contents and structure
3. Duplicate Check: System checks for existing folders in S3
4. Selective Upload: Only new folders are uploaded, existing ones are skipped
5. Progress Monitoring: Individual folder progress tracking
6. Statistics Update: Real-time folder and file count updates

🔍 Search & Management

1. Search Input: User enters search query
2. Real-time Filtering: Results are filtered as user types
3. File Operations: Users can view, download, or delete files
4. Confirmation Dialogs: Delete operations require user confirmation
5. S3 Integration: All operations interact directly with S3

📊 Analytics & Reporting

1. S3 Connection: Analytics module connects to S3 bucket
2. Data Extraction: Folder metadata and statistics are extracted
3. Report Generation: Excel report is created with comprehensive data
4. File Output: Report is saved locally with timestamp

Development

To modify the application:

1. Backend Changes: Edit app.py for server-side functionality
2. Frontend Changes: Edit templates/index.html for UI modifications
3. Dependencies: Add new packages to requirements.txt
4. Analytics: Modify s3_folder_checker.py for reporting features

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Copyright 2025 DBMS Project

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

🔒 Security Note

This project is for internal use. Ensure AWS credentials are kept secure and not shared.