Learning-to-Rank (LTR) Two-Stage Search Engine

This repository contains a production-grade, two-stage retrieval and ranking search engine powered by Learning-to-Rank (LTR), complete with interactive telemetry feedback loops, online A/B testing infrastructure, and a web search dashboard.

---

💡 Why This Project?

Traditional Learning-to-Rank (LTR) tutorials teach how to train a ranking model on static offline datasets, but they completely omit how a search system generates candidates, tracks real-time clicks, runs A/B testing cohorts, or corrects position bias in production.

This project was built to fill that gap. It is a production-grade, closed-loop LTR search engine that runs entirely locally. Key educational and architectural highlights include:

• True Two-Stage Retrieval: Implements Stage 1 Hybrid retrieval (SQLite FTS5 BM25 keyword search + all-MiniLM-L6-v2 dense vector semantic search blended via Reciprocal Rank Fusion (RRF)) and Stage 2 LTR reranking (LightGBM LambdaMART).
• Closed-Loop Feedback: Features a live SQLite telemetry loop that registers user interactions (clicks, carts, and purchases), allowing you to trigger model retraining directly from the UI dashboard.
• Position Bias Correction: Implements Inverse Propensity Scoring (IPS) in the retraining pipeline to mathematically correct position bias (users clicking top results simply because they are visible).
• Statistical A/B Testing: Includes a live conversion dashboard running a pure-Python two-proportion z-test to calculate CTR/CR uplifts and exact p-value statistical significance.
• Zero-Config & CPU-Friendly: Designed to run entirely locally on CPU without external databases (like Solr, Elasticsearch, Pinecone, or Redis).

---

🚀 System Architecture

In modern high-scale search engines, sorting millions of catalog products using a complex machine learning model for every query is computationally prohibitive. This project implements a standard Two-Stage Search Architecture with Hybrid Retrieval:

                [ User Query ]
                      │
       ┌──────────────┴──────────────┐
       ▼                             ▼
┌──────────────┐              ┌──────────────┐
│ Text Search  │              │Semantic Search│
│(SQLite FTS5) │              │(all-MiniLM-L6)│
└──────────────┘              └──────────────┘
       │                             │
       └──────────────┬──────────────┘
                      ▼
          ┌──────────────────────┐
          │Reciprocal Rank Fusion│ ──► Combined top N candidates (High Recall)
          └──────────────────────┘
                      │
                      ▼
          ┌──────────────────────┐
          │  Feature Hydration   │ ──► Fetch ratings, CTRs, and compute affinities
          └──────────────────────┘
                      │
                      ▼
          ┌──────────────────────┐
          │  LTR Model Reranker  │ ──► LightGBM LambdaMART ranking (High Precision)
          └──────────────────────┘
                      │
                      ▼
              [ Reranked Results ]

1. Stage 1 (Hybrid Candidate Retrieval):
   • Text Search: Evaluates keyword matching using SQLite FTS5 Full-Text Search (BM25).
   • Semantic Search: Computes dense vector representations of queries and product titles using a lightweight transformer model (SentenceTransformers all-MiniLM-L6-v2), calculating similarity via cosine dot products.
   • Reciprocal Rank Fusion (RRF): Merges text and semantic candidate lists using standard rank reciprocal blending: $$RRF_Score(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$ This retrieves the top $N=30$ candidate products, guaranteeing semantic recall and keyword accuracy.
2. Stage 2 (LambdaMART Reranking):
   • Hydrates static catalog features (average rating, brand popularity, historical click-through rate, historical conversion rate) and computes dynamic query-dependent features (exact keyword match, category affinity, Gaussian price affinity, and semantic similarity score).
   • Scores candidates using a trained LightGBM LambdaMART LTR ranking model to optimize relative list ordering (NDCG).
   • Sorts results descending by score and serves them.

---

🖥️ Dashboard Preview

Below is a preview of the interactive LTR Search Dashboard, showing the A/B Testing Conversion metrics, the Live Telemetry log stream, and a comparison of Stage 1 Hybrid Retrieval (RRF) versus Stage 2 LTR Reranked results:

---

📈 Closed-Loop Learning & Telemetry

To ensure continuous improvement, this search engine implements an online logging feedback loop:

• Telemetry logging: Logs search session queries, shown product impressions (positions/layout context), and user actions (Clicks, Add to Carts, and Purchases).
• Position Bias Correction (Inverse Propensity Scoring - IPS): To prevent self-reinforcing ranking bias, retraining instances are weighted using IPS based on display position. Click, cart, and purchase actions at position $pos$ are weighted by $W(pos) = \sqrt{pos + 1}$. Unclicked items are weighted at $1.0$.
• Continuous Retraining: An automated retraining pipeline joins session impressions with user actions to compute dynamic training labels:
  • 3 (Purchase) | 2 (Cart) | 1 (Click) | 0 (Impression Only)
  • Trains a fresh LightGBM model from the log database, applying IPS sample weights, and hot-reloads it into the serving layer.

---

🧪 Online A/B Testing Infrastructure

The serving layer features a built-in A/B testing framework:

• Stable Cohort Partitioning: Search requests are assigned to control (BM25 search results only) or treatment (LTR reranked search results) cohorts using a hash function over the session ID.
• Conversion Dashboard: Compiles live Click-Through Rates (CTR) and Conversion Rates (CR) for Control vs. Treatment.
• Statistical Significance: Performs a pure-Python two-proportion z-test on click and purchase ratios to calculate the exact p-value and determine if LTR yields a statistically significant uplift ($p < 0.05$).

---

📁 Repository Structure

L2R-RANKING-ENGINE/
│
├── core_ml/
│   ├── __init__.py
│   ├── ab_testing.py           # Two-proportion z-test and A/B metric aggregations
│   ├── database.py             # SQLite setup, FTS5 virtual index, telemetry logging
│   ├── dataset_generator.py    # Synthetic product catalog & search session simulation
│   ├── embedder.py             # SentenceTransformer model wrapper for dense embeddings
│   ├── evaluate.py             # Offline exact mathematical NDCG@K validation
│   ├── features.py             # Standardized model feature names
│   ├── hybrid_retriever.py     # Semantic lookup & RRF candidate fusion logic
│   ├── inference_features.py   # Query intent classification & inference feature hydration
│   ├── reranker.py             # LightGBM booster wrappers for model scoring
│   ├── retrain.py              # Teleprocessing pipeline to retrain model from click logs
│   └── train.py                # LambdaMART model training with early stopping
│
├── serving/
│   ├── main.py                 # FastAPI application serving search APIs and dashboard
│   └── templates/
│       └── index.html          # Clean light-themed minimal A/B search UI
│
├── tests/                      # Pytest unit & integration test suites
│   ├── test_ab_testing.py      # Tests cohort allocation and Z-test metrics
│   ├── test_debias.py          # Tests position bias correction (IPS) calculations and retraining
│   ├── test_hybrid.py          # Tests embedder, semantic index, and RRF calculations
│   ├── test_search.py          # Tests database, features, and FastAPI client routing
│   └── test_telemetry.py       # Tests feedback endpoints and log-based retraining
│
├── requirements.txt            # Project dependencies
├── run_server.py               # Launcher script for FastAPI app
└── README.md                   # Project overview (this file)

---

⚙️ Installation & Setup

1. Clone the repository:

   git clone https://github.com/kanikakapoorcode/L2R-RANKING-ENGINE.git
   cd L2R-RANKING-ENGINE

2. Initialize a Python virtual environment:

   python -m venv venv
   # On Windows:
   .\venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

---

🏃 Running the Application

Step 1: Generate Catalog & Train Base Model

Generate the synthetic product catalog and training files, then train the base LambdaMART ranker:

# Set PYTHONPATH to project root
$env:PYTHONPATH="."  # On Windows PowerShell
export PYTHONPATH="." # On Linux/macOS

# Generate datasets
python core_ml/dataset_generator.py

# Train the base ranking model
python core_ml/train.py

Step 2: Run Offline Evaluations

Verify LTR model accuracy (NDCG) compared to standard BM25 and popularity baselines:

python core_ml/evaluate.py

Step 3: Run Unit & Integration Tests

Ensure the code, database operations, and A/B testing logic function without errors:

pytest tests/

Step 4: Boot Up Serving Dashboard

Start the online search engine application:

python run_server.py

Open your browser and navigate to http://127.0.0.1:8000 to search, log mock user interactions, and retrain the model!