Skip to content
BranchesTags

Repository files navigation

CiteConnect Backend

AI-powered academic paper recommendation system with advanced ML capabilities, citation network analysis, and personalized user experiences.

Table of Contents

Overview

CiteConnect is a sophisticated research paper recommendation system that leverages machine learning, citation networks, and user behavior to provide personalized academic paper recommendations. The system supports multiple embedding models, real-time evaluation, and comprehensive bias detection.

Key Capabilities

  • Multi-Model Embeddings: Supports both MiniLM (384-dim) and SPECTER2 (768-dim) embedding models
  • Citation Network Analysis: Analyzes paper citations, co-citations, and bibliographic coupling
  • Personalized Recommendations: User profile-based recommendations with real-time adaptation
  • Ground Truth Evaluation: Built-in evaluation framework with bias detection
  • MLflow Integration: Comprehensive experiment tracking and model versioning
  • Real-time Clustering: Dynamic paper clustering for discovery

Model Development Pipeline

This section documents the model development process as per the Model Development Guidelines. Since CiteConnect uses pre-trained embedding models, the focus is on validation, bias detection, and model selection rather than training.

1. Docker Format Implementation

The entire model development process is containerized using Docker for reproducibility and portability:

# Build the containerized application
docker-compose build

# Run model validation in container
docker-compose run --rm api python scripts/run_experiment.py

# Run bias detection
docker-compose run --rm api python scripts/offline_evaluation.py

Key Files:

  • Dockerfile: Containerizes the application with all dependencies
  • docker-compose.yml: Orchestrates services (API, Redis, MLflow)
  • All model code runs in isolated containers

2. Loading Data from Data Pipeline

Data is loaded from the PostgreSQL database (output of the data pipeline) with proper versioning:

Implementation:

  • Repository Pattern: app/db/repositories/ - Abstracts data access
  • Data Loading: Papers, embeddings, and user data loaded from Supabase PostgreSQL
  • Versioning: Database migrations via Alembic track schema versions

Code Location:

# app/db/repositories/paper_repo.py
async def get_papers_for_evaluation(self) -> List[Paper]:
    """Load papers from data pipeline for model evaluation"""
    
# app/db/repositories/embedding_repo.py  
async def get_paper_embeddings(self, model_name: str) -> List[Embedding]:
    """Load embeddings generated by data pipeline"""

Usage:

# Validate data pipeline output
docker-compose exec api python scripts/validate_data.py

3. Model Training and Selection

Note: CiteConnect uses pre-trained models (all-MiniLM-L6-v2 and allenai/specter2_base), so traditional training is not required. However, the system implements:

Model Selection Process:

  • Multiple Models: Supports both MiniLM (384-dim) and SPECTER2 (768-dim)
  • Performance Comparison: Models are evaluated on validation metrics
  • Selection Logic: app/services/runtime/recommendation_orchestrator.py selects best model based on:
    • Embedding quality (coverage, dimension correctness)
    • Recommendation performance metrics
    • User segment performance

Code Location:

# app/services/bootstrap/embedding_service.py
class EmbeddingService:
    async def initialize(self):
        """Load and validate embedding models"""
        # Health check both models
        # Select best performing model per use case

4. Model Validation

Comprehensive validation on separate validation datasets with relevant metrics:

Validation Metrics:

  • Cold-Start Users: Profile alignment, Ground truth quality
  • Mature Users: Precision@10, Recall@10, MRR, Click-through rate

Implementation:

# app/services/runtime/evaluation_service.py
class EvaluationService:
    async def evaluate_cold_start(self, recommendations, user_profile):
        """2-metric evaluation for cold-start users"""
        
    async def evaluate_mature(self, recommendations, user_interactions):
        """Richer metrics for mature users"""

Validation Scripts:

# Run validation experiments
docker-compose exec api python scripts/run_experiment.py

# Offline evaluation with ground truth
docker-compose exec api python scripts/offline_evaluation.py

Output: Metrics logged to MLflow for tracking and comparison

5. Bias Checking

Bias detection using data slicing techniques with reports and visualizations:

Slicing Dimensions:

  • Research stage (early, mid, senior)
  • Domain (healthcare, fintech, quantum_computing)
  • Reading level
  • Years of experience

Implementation:

# app/services/runtime/evaluation_service.py
async def detect_bias(
    self,
    recommendation_events: List[Dict],
    slicing_dimensions: List[str]
) -> Dict:
    """
    Detect bias across user segments using slicing techniques.
    Generates bias reports with visualizations.
    """

Bias Detection Process:

  1. Group recommendations by slicing dimension
  2. Calculate metrics per slice (precision, recall, CTR)
  3. Detect variance > threshold (default: 20%)
  4. Generate bias reports with identified biases

Output:

  • bias_reports.json: Structured bias reports
  • MLflow artifacts: Bias visualizations and metrics
  • Per-user bias flags in evaluation results

Usage:

# Run bias detection
docker-compose exec api python scripts/offline_evaluation.py

# View bias reports
cat bias_reports.json

6. Model Selection After Bias Checking

Final model selection considers both validation performance AND bias analysis:

Selection Criteria:

  1. Performance Metrics: Precision@10, Recall@10, MRR
  2. Bias Analysis: Variance across user segments < threshold
  3. Fairness: Balanced performance across domains

Implementation:

# app/services/runtime/recommendation_orchestrator.py
class RecommendationOrchestrator:
    async def select_best_model(
        self,
        performance_metrics: Dict,
        bias_report: Dict
    ) -> str:
        """
        Select best model considering:
        - Validation performance
        - Bias detection results
        - Fairness across segments
        """
        # Reject models with high bias
        if bias_report['max_variance'] > threshold:
            return None
        
        # Select model with best performance AND low bias
        return best_model

Decision Flow:

Performance Metrics → Bias Check → Final Selection
     ↓                    ↓              ↓
  Precision@10      Variance < 20%   Best Model
  Recall@10         Fair Distribution
  MRR               No Domain Bias

7. Pushing Model to Artifact Registry

Models and artifacts are pushed to MLflow Model Registry (with GCP Artifact Registry support):

MLflow Integration:

# app/services/bootstrap/experiment_service.py
class ExperimentService:
    async def log_model_artifact(
        self,
        model_name: str,
        model_path: str,
        metrics: Dict,
        bias_report: Dict
    ):
        """Log model to MLflow with metadata"""
        with mlflow.start_run():
            mlflow.log_artifact(model_path)
            mlflow.log_metrics(metrics)
            mlflow.log_dict(bias_report, "bias_report.json")
            
            # Register model
            mlflow.register_model(
                model_uri=f"runs:/{run_id}/model",
                name=model_name
            )

GCP Artifact Registry Support:

  • Configuration in app/config.py: 
    GOOGLE_CLOUD_ARTIFACT_REGISTRY: str
GOOGLE_CLOUD_PROJECT: str
  • Dependencies: google-cloud-artifact-registry==1.11.0

Model Registry Workflow:

  1. Experiment Tracking: Log runs to MLflow
  2. Model Validation: Validate on test set
  3. Bias Check: Ensure fairness
  4. Model Registration: Register validated models
  5. Versioning: Track model versions
  6. Deployment: Promote models to production

Usage:

# Run experiment and register model
docker-compose exec api python scripts/run_experiment.py

# View registered models in MLflow UI
# http://localhost:5000

Model Artifacts Stored:

  • Model weights (if fine-tuned)
  • Embedding vectors
  • Evaluation metrics
  • Bias reports
  • Configuration files

Architecture

The backend follows a clean architecture pattern with clear separation of concerns:

┌─────────────────────────────────────────────────────────┐
│                    FastAPI Application                   │
│                  (app/main.py)                           │
└─────────────────────────────────────────────────────────┘
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
┌───────▼────────┐  ┌───────▼────────┐  ┌───────▼────────┐
│  API Layer     │  │  Service Layer  │  │ Repository     │
│  (app/api/v1)  │  │  (app/services) │  │ (app/db/repos)  │
└────────────────┘  └─────────────────┘  └────────────────┘
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
┌───────▼────────┐  ┌───────▼────────┐  ┌───────▼────────┐
│  PostgreSQL     │  │  Redis Cache   │  │  Weaviate      │
│  (Supabase)     │  │                │  │  (Vector DB)   │
└─────────────────┘  └────────────────┘  └────────────────┘

Service Architecture

  • Bootstrap Services: Initialize models, ground truth, and experiments

    • EmbeddingService: Manages embedding models (MiniLM, SPECTER2)
    • GroundTruthService: Handles evaluation ground truth data
    • ExperimentService: Manages MLflow experiments

  • Runtime Services: Handle real-time operations

    • RecommendationOrchestrator: Coordinates recommendation generation
    • UserStateService: Manages user profiles and preferences
    • EvaluationService: Performs real-time evaluation and bias detection

Features

1. Paper Recommendations

  • Semantic similarity-based recommendations
  • Citation network-based recommendations
  • Hybrid recommendation strategies
  • Real-time personalization

2. User Management

  • User profile creation and management
  • Interest tracking and hierarchy
  • Interaction history
  • Profile embeddings for personalization

3. Search & Discovery

  • Full-text search across papers
  • Semantic search using embeddings
  • Citation graph exploration
  • Paper clustering

4. Evaluation & Monitoring

  • Ground truth-based evaluation
  • Bias detection and reporting
  • MLflow experiment tracking
  • Performance metrics (MRR, NDCG, Precision, Recall)

5. Data Management

  • Paper ingestion and validation
  • Embedding generation and storage
  • Citation network construction
  • Ground truth initialization

Tech Stack

Core Framework

  • FastAPI: Modern, fast web framework for building APIs
  • Python 3.11: Latest Python features and performance

Databases

  • PostgreSQL (Supabase): Primary relational database with pgvector extension
  • Redis: Caching and session management
  • Weaviate: Vector database for semantic search (optional)

Machine Learning

  • PyTorch: Deep learning framework
  • Sentence Transformers: Embedding models
    • all-MiniLM-L6-v2: 384-dimensional embeddings
    • allenai/specter2_base: 768-dimensional embeddings
  • scikit-learn: Clustering and similarity calculations

MLOps

  • MLflow: Experiment tracking and model registry
  • Celery: Background task processing
  • Flower: Celery monitoring

Other Technologies

  • Alembic: Database migrations
  • Pydantic: Data validation
  • structlog: Structured logging
  • Docker: Containerization

Project Structure

citeconnect-backend/
├── app/
│   ├── api/                    # API endpoints
│   │   └── v1/
│   │       ├── auth.py         # Authentication endpoints
│   │       ├── users.py         # User management
│   │       ├── papers.py        # Paper operations
│   │       ├── recommendations.py  # Recommendation endpoints
│   │       ├── search.py        # Search endpoints
│   │       ├── interactions.py  # User interactions
│   │       ├── clusters.py      # Clustering endpoints
│   │       └── graph.py         # Citation graph
│   │
│   ├── core/                   # Core functionality
│   │   ├── config.py           # Configuration management
│   │   ├── security.py         # Security utilities
│   │   └── exceptions.py       # Custom exceptions
│   │
│   ├── db/                     # Database layer
│   │   ├── connection.py       # Database connection pool
│   │   ├── repositories/       # Data access layer
│   │   │   ├── base.py         # Base repository
│   │   │   ├── user_repo.py    # User data access
│   │   │   ├── paper_repo.py   # Paper data access
│   │   │   ├── embedding_repo.py  # Embedding data access
│   │   │   ├── ground_truth_repo.py  # Ground truth data
│   │   │   └── interaction_repo.py  # Interaction data
│   │   ├── redis_client.py     # Redis client
│   │   └── weaviate_client.py  # Weaviate client
│   │
│   ├── services/               # Business logic
│   │   ├── bootstrap/          # Initialization services
│   │   │   ├── embedding_service.py
│   │   │   ├── ground_truth_service.py
│   │   │   └── experiment_service.py
│   │   ├── runtime/           # Runtime services
│   │   │   ├── recommendation_orchestrator.py
│   │   │   ├── user_state_service.py
│   │   │   └── evaluation_service.py
│   │   ├── user_service.py
│   │   ├── paper_service.py
│   │   ├── search_service.py
│   │   ├── recommendation_service.py
│   │   └── clustering_service.py
│   │
│   ├── models/                 # Data models
│   ├── schemas/                # Pydantic schemas
│   ├── tasks/                  # Celery tasks
│   ├── utils/                  # Utility functions
│   └── main.py                 # Application entry point
│
├── scripts/                    # Utility scripts
│   ├── initialize_ground_truth.py  # Initialize ground truth data
│   ├── validate_data.py       # Data validation
│   ├── seed_data.py           # Seed database
│   └── ...
│
├── tests/                      # Test suite
├── alembic/                    # Database migrations
├── docker-compose.yml          # Docker Compose configuration
├── Dockerfile                   # Docker image definition
├── requirements.txt            # Python dependencies
└── setup.sh                    # Setup script

Setup & Installation

Prerequisites

  • Python 3.11+
  • Docker and Docker Compose
  • PostgreSQL database (Supabase recommended)
  • Redis (optional, for caching)

Quick Start

  1. Clone the repository

    git clone <repository-url>
cd CiteConnect-ModelPipeline/citeconnect-backend

  2. Set up environment variables

    cp .env.example .env
# Edit .env with your configuration

  3. Run setup script

    chmod +x setup.sh
./setup.sh

    Or manually:

  4. Build and start services

    docker-compose build
docker-compose up -d

  5. Initialize database

    # Run migrations
docker-compose exec api alembic upgrade head

# Validate data
docker-compose exec api python scripts/validate_data.py

# Initialize ground truth (optional)
docker-compose exec api python scripts/initialize_ground_truth.py

  6. Verify installation

    curl http://localhost:8000/health

Manual Setup (Without Docker)

  1. Create virtual environment

    python3.11 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

  2. Install dependencies

    pip install -r requirements.txt

  3. Configure environment

    cp .env.example .env
# Edit .env with your database credentials

  4. Run database migrations

    alembic upgrade head

  5. Start the application

    uvicorn app.main:app --reload

Configuration

Environment Variables

Key configuration variables in .env:

# Application
ENVIRONMENT=production
DEBUG=false
LOG_LEVEL=INFO

# Database
DATABASE_URL=postgresql://user:password@host:port/database
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=your-supabase-key

# Redis
REDIS_HOST=localhost
REDIS_PORT=6379

# ML Models
EMBEDDING_MODEL_MINILM=sentence-transformers/all-MiniLM-L6-v2
EMBEDDING_MODEL_SPECTER=allenai/specter2_base

# MLflow
MLFLOW_TRACKING_URI=http://mlflow:5000

# Ground Truth
MIN_GROUND_TRUTH_CITATIONS=10
MAX_GROUND_TRUTH_CITATIONS=100
MIN_REFERENCE_COVERAGE=0.3

See app/config.py for all available configuration options.

API Documentation

Once the application is running, API documentation is available at:

Main Endpoints

Health & Status

  • GET /health - Health check endpoint

Authentication

  • POST /api/v1/auth/register - User registration
  • POST /api/v1/auth/login - User login
  • POST /api/v1/auth/logout - User logout

Users

  • GET /api/v1/users/me - Get current user profile
  • PUT /api/v1/users/me - Update user profile
  • GET /api/v1/users/{user_id} - Get user by ID

Papers

  • GET /api/v1/papers - List papers with filtering
  • GET /api/v1/papers/{paper_id} - Get paper details
  • GET /api/v1/papers/{paper_id}/citations - Get paper citations
  • GET /api/v1/papers/{paper_id}/references - Get paper references

Recommendations

  • GET /api/v1/recommendations - Get personalized recommendations
  • GET /api/v1/recommendations/explain - Get recommendation explanations

Search

  • GET /api/v1/search - Search papers (semantic + keyword)
  • GET /api/v1/search/semantic - Semantic search only

Interactions

  • POST /api/v1/interactions - Record user interaction
  • GET /api/v1/interactions - Get user interaction history

Services

Bootstrap Services

Services that initialize on application startup:

EmbeddingService

  • Loads and manages embedding models (MiniLM, SPECTER2)
  • Handles model health checks
  • Provides embedding generation capabilities

GroundTruthService

  • Loads ground truth papers and relationships
  • Provides evaluation data for recommendations
  • Manages canonical papers

ExperimentService

  • Manages MLflow experiments
  • Tracks model versions and metrics
  • Handles experiment configuration

Runtime Services

Services that handle real-time operations:

RecommendationOrchestrator

  • Coordinates recommendation generation
  • Combines multiple recommendation strategies
  • Handles A/B testing and experiments

UserStateService

  • Manages user profiles and preferences
  • Tracks user interaction history
  • Updates user embeddings

EvaluationService

  • Performs real-time evaluation
  • Detects bias in recommendations
  • Calculates performance metrics

Database Schema

Core Tables

  • users: User accounts and profiles
  • user_profiles_extended: Extended user profile information
  • user_interest_hierarchy: User interests with hierarchy
  • papers: Paper metadata
  • paper_embeddings_minilm: MiniLM embeddings (384-dim)
  • paper_embeddings_specter: SPECTER embeddings (768-dim)
  • ground_truth_papers: Papers used for evaluation
  • ground_truth_relationships: Citation relationships for ground truth
  • user_interactions: User interaction history
  • user_recommendation_state: User recommendation state

See documentation/db_schema_guide.md for detailed schema documentation.

Scripts

Model Development Scripts

These scripts implement the Model Development Pipeline requirements:

1. Data Validation

  • validate_data.py: Validates data quality and completeness from data pipeline 
    docker-compose exec api python scripts/validate_data.py
    • Validates paper data quality
    • Checks embedding coverage and dimensions
    • Verifies ground truth readiness
    • Output: Validation report with data quality metrics

2. Model Validation & Experiment Tracking

  • run_experiment.py: Runs MLflow experiments with model validation 
    docker-compose exec api python scripts/run_experiment.py
    • Loads data from data pipeline
    • Validates models on test users
    • Logs metrics to MLflow (Precision@10, Recall@10, MRR)
    • Tracks hyperparameters and model versions
    • Output: MLflow experiment run with metrics

3. Bias Detection

  • offline_evaluation.py: Runs offline evaluation with bias detection 
    docker-compose exec api python scripts/offline_evaluation.py
    • Evaluates recommendations across user segments
    • Detects bias using data slicing (domain, research stage, etc.)
    • Generates bias reports and visualizations
    • Output: bias_reports.json with detected biases

4. Ground Truth Initialization

  • initialize_ground_truth.py: Initializes ground truth papers for evaluation 
    docker-compose exec api python scripts/initialize_ground_truth.py
    • Identifies high-quality papers for evaluation
    • Computes citation relationships
    • Creates ground truth dataset
    • Output: Ground truth data in database

Data Management Scripts

  • seed_data.py: Seeds database with sample data

    docker-compose exec api python scripts/seed_data.py

  • seed_papers_from_pickle.py: Loads papers from pickle files

Utility Scripts

  • inspect_pickle.py: Inspects pickle files
  • fix_user_embeddings.py: Fixes user embedding issues
  • check_citation_overlap.py: Checks citation network overlap
  • test_recommendations.py: Tests recommendation generation

Testing

Run Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=app --cov-report=html

# Run specific test file
pytest tests/test_api/test_users.py

# Run with verbose output
pytest -v

Test Structure

  • tests/test_api/: API endpoint tests
  • tests/test_services/: Service layer tests
  • tests/test_utils/: Utility function tests

Deployment

Docker Deployment

The application is containerized using Docker:

# Build image
docker-compose build

# Start services
docker-compose up -d

# View logs
docker-compose logs -f api

# Stop services
docker-compose down

Production Considerations

  1. Environment Variables: Ensure all production secrets are set
  2. Database: Use managed PostgreSQL (Supabase) with connection pooling
  3. Caching: Configure Redis for production workloads
  4. Monitoring: Set up logging and monitoring (MLflow, Prometheus)
  5. Scaling: Use Docker Swarm or Kubernetes for horizontal scaling

Development

Code Style

The project uses:

  • Black: Code formatting
  • Ruff: Linting
  • MyPy: Type checking
  • Pre-commit: Git hooks
# Format code
black app/

# Lint code
ruff check app/

# Type check
mypy app/

Database Migrations

# Create new migration
alembic revision --autogenerate -m "description"

# Apply migrations
alembic upgrade head

# Rollback migration
alembic downgrade -1

Adding New Features

  1. Create feature branch
  2. Implement feature with tests
  3. Run tests and linting
  4. Create migration if needed
  5. Update documentation
  6. Submit pull request

Documentation

Additional documentation:

  • Database Schema: documentation/db_schema_guide.md
  • Low-Level Design: documentation/citeconnect_lld.md
  • Deployment Guide: documentation_backend_root/citeconnect_deployment_guide.txt
  • Testing Guide: documentation_backend_root/citeconnect_testing_guide.txt

Support

For issues and questions, please open an issue on the repository.

CI Test

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages