[Feature]: Integrate Redis for Caching and Background Processing

[RishiGoswami-code]

Description

The application currently performs all operations directly through the database and backend services. As usage grows, this can lead to increased response times, unnecessary database load, and slower processing of resource-intensive tasks.

Integrate Redis to improve performance, scalability, and reliability across the platform.

Objectives

• Introduce Redis as a centralized caching layer.
• Reduce repeated database queries.
• Support background job processing.
• Improve overall application responsiveness.

Potential Use Cases

API Response Caching

Cache frequently accessed data such as:

• Dashboard statistics
• Assessment summaries
• User profiles
• Institution metadata
• Question banks

Example:

Request
   ↓
Redis Cache
   ↓ (Cache Hit)
Return Response

OR

Redis Cache
   ↓ (Cache Miss)
Database
   ↓
Store in Redis
   ↓
Return Response

Session Management

Store:

• Authentication sessions
• Refresh tokens
• User activity states

Background Job Queue

Use Redis as a queue backend for:

• OCR processing
• AI grading
• Bulk answer-sheet evaluation
• Report generation
• Notification delivery

Rate Limiting

Implement API rate limiting using Redis to prevent abuse and protect services.

Real-Time Features

Enable:

• Live grading progress updates
• Notification systems
• WebSocket state management

Technical Requirements

• Add Redis configuration support through environment variables.
• Create Redis connection utilities.
• Implement health checks for Redis connectivity.
• Provide graceful fallback when Redis is unavailable.
• Add documentation for local and production deployment.

Acceptance Criteria

• Redis connection established and configurable via environment variables.
• Cache layer implemented for selected endpoints.
• Background job queue support added.
• Health check endpoint includes Redis status.
• Documentation updated with setup instructions.
• Existing functionality remains backward compatible.
• Unit and integration tests added.

Suggested Stack

• Redis
• FastAPI Redis Client
• Celery / RQ / Dramatiq (optional)
• Docker Compose Redis Service

Benefits

• Faster API responses
• Reduced database load
• Better scalability
• Improved user experience
• Foundation for asynchronous AI and OCR processing

Type: Enhancement
Difficulty: Intermediate
Labels: enhancement, backend, performance, redis, help wanted