CoughOverflow - Scalable Pathogen Analysis Service

Overview

CoughOverflow is a cloud-native, highly scalable service designed to analyze saliva samples for detecting COVID-19 and H5N1 (avian influenza) pathogens. The platform uses machine learning algorithms to process images of cell samples and provides a RESTful API for labs and healthcare providers to submit analysis requests and retrieve results.

This project demonstrates the implementation of a microservice architecture deployed on AWS using infrastructure as code (Terraform) with auto-scaling capabilities designed to handle variable load patterns from routine testing to epidemic-level demand.

Architecture

The system is built as a cloud-native application with the following components:

Key Components

1. FastAPI Backend:

   • RESTful API built with Python's FastAPI framework
   • Asynchronous request handling with background task processing
   • Request validation and error handling

2. Database Layer:

   • PostgreSQL database for storing analysis job metadata
   • Configured with proper indexing and connection pooling
   • Scalable storage with auto-growth capabilities (20GB base, up to 1TB)

3. Analysis Engine Integration:

   • Integration with the QDHeC-developed overflowengine for image analysis
   • Concurrent processing using ThreadPoolExecutor to manage computational load
   • Error handling and retry mechanisms for failed analyses

4. AWS Infrastructure:

   • ECS (Elastic Container Service) for containerized deployment
   • Application Load Balancer for traffic distribution
   • Auto-scaling based on CPU utilization metrics
   • RDS (Relational Database Service) for reliable data persistence

Technical Implementation

RESTful API

The API follows a clean RESTful design with the following endpoints:

• GET /api/v1/health - Service health check
• GET /api/v1/labs - List all labs that have submitted analyses
• GET /api/v1/labs/results/{lab_id} - Get analyses for a specific lab with filtering
• GET /api/v1/labs/results/{lab_id}/summary - Get summary statistics for a lab
• GET /api/v1/patients/results - Get analyses for a specific patient
• GET /api/v1/analysis - Get a specific analysis by ID
• POST /api/v1/analysis - Submit a new analysis request
• PUT /api/v1/analysis - Update the lab associated with an analysis

The API includes comprehensive validation for all parameters and returns appropriate HTTP status codes and error messages.

Data Persistence Strategy

1. Database Design:

   • Efficient schema for storing analysis metadata
   • PostgreSQL database with proper indexing
   • Transaction management to ensure data integrity

2. Image Storage:

   • Images stored on persistent file storage
   • Unique UUIDs for each analysis job and associated image

3. Connection Management:

   • Connection pooling with SQLAlchemy
   • Pool size optimization based on load testing
   • Pre-ping mechanism to detect stale connections

Scalability Features

1. Auto-scaling Configuration:

   • Min capacity: 1 instance
   • Max capacity: 12 instances
   • Scale-out threshold: 25% CPU utilization
   • Implementation using AWS AppAutoScaling

2. Load Balancing:

   • Application Load Balancer for distributing traffic
   • Health checks for instance viability
   • Connection draining for graceful scaling

3. Concurrent Processing:

   • Thread pool for concurrent analysis of images
   • Configurable worker limits to prevent resource exhaustion
   • Priority processing for urgent analysis requests

4. Database Scaling:

   • Auto-growth configuration (up to 1TB)
   • Properly sized instance class for cost-effective performance

Containerization & Deployment

The application is containerized using Docker and deployed using Terraform:

1. Docker Configuration:

   • Multi-stage build for optimized image size
   • Environment variable configuration
   • Health check endpoints

2. Terraform Implementation:

   • Infrastructure as Code (IaC) for repeatable deployments
   • Modular design for maintainable configuration
   • Resource tagging for proper organization
   • Automatic output of service endpoint

3. CI/CD Integration:

   • Deployment scripts for automated deployment
   • Environment-specific configuration

Performance Optimization

1. Database Optimizations:

   • Connection pooling (20 connections with 20 max overflow)
   • Query optimization for filtering operations
   • Proper indexing strategy

2. Computation Efficiency:

   • Background task processing for analysis to free up web workers
   • Configurable thread pool size for analysis jobs
   • Bounded task queues to prevent memory issues under heavy load

3. Network Optimizations:

   • Minimized payload sizes
   • Proper HTTP caching headers
   • Efficient serialization/deserialization

Deployment Instructions

Local Development

To run the service locally:

# Clone the repository
git clone https://github.com/yourusername/coughoverflow.git
cd coughoverflow

# Run the local deployment script
./local.sh

The service will be available at http://localhost:8080 with API documentation at http://localhost:8080/docs.

Cloud Deployment

To deploy to AWS:

# Ensure AWS credentials are configured
# Create a credentials file in the repository root

# Run the deployment script
./deploy.sh

The deployment script will:

1. Initialize Terraform
2. Apply the Terraform configuration
3. Output the API endpoint URL to api.txt

Testing

The service includes extensive automated testing:

1. Unit Tests:

   • Component-level testing for all service modules
   • Mocking of external dependencies

2. Integration Tests:

   • End-to-end API testing
   • Database integration testing

3. Load Testing:

   • Concurrent request handling (test_concurrent.py)
   • Simulated epidemic scenarios
   • Auto-scaling validation

Future Enhancements

1. Machine Learning Improvements:

   • Integration with more advanced analysis algorithms
   • Support for additional pathogens

2. Monitoring & Observability:

   • Enhanced logging with structured log formats
   • Metrics collection for operational insights
   • Alerting based on performance thresholds

3. Security Enhancements:

   • Enhanced authentication mechanisms
   • Data encryption at rest and in transit
   • Advanced access control policies

Technologies Used

• Backend: Python, FastAPI, SQLAlchemy, ThreadPoolExecutor
• Database: PostgreSQL 14
• Containerization: Docker
• Cloud Infrastructure: AWS (ECS, RDS, ALB, Auto Scaling)
• Infrastructure as Code: Terraform
• Testing: Pytest, concurrent.futures

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

https://csse6400.uqcloud.net/assessment/coughoverflow/#api-Pas-submitRequest

https://fastapi.tiangolo.com/tutorial/handling-errors/?h=exception#override-request-validation-exceptions

^^^ USE THIS LINK FOR VALIDATION ERRROS STUFF..

Exploring the inside of docker containers pog!!

docker exec -it coughoverflow-antimatter543-coughoverflow-api-1 /bin/bash

To directly access the postgresql

docker exec -it coughoverflow-db psql -U administrator -d coughoverflow

SELECT * FROM analysis_jobs; etc.

docker logs op af