Robust Retrieval-Augmented Generation with Page Indexing for Scalable Document QA
PRODUCTION-RAG-WITH-PAGEINDEX is a sophisticated retrieval-augmented generation system designed to handle ultra-long documents by indexing pages and leveraging hybrid retrieval methods. It enables efficient and accurate question answering over large, complex documents with a scalable and modular architecture.
Large documents with complex structures pose significant challenges for retrieval and question answering systems, especially when the context length exceeds typical model limits. PRODUCTION-RAG-WITH-PAGEINDEX solves this by indexing documents at the page level and combining multiple retrieval strategies to ensure relevant, precise, and efficient information retrieval.
This project benefits AI researchers, data scientists, and developers building scalable QA systems over large text corpora, technical reports, or knowledge bases.
| Feature | PRODUCTION-RAG-WITH-PAGEINDEX | Alternative A | Alternative B |
|---|---|---|---|
| Page-level Document Indexing | ✅ | ❌ | ❌ |
| Hybrid Retrieval (BM25 + Dense) | ✅ | ✅ (only BM25 or dense) | ❌ |
| LRU Caching for Speed | ✅ | ❌ | ✅ |
| Modular Chunking & Embedding | ✅ | Limited | Limited |
| Multi-Device Embedding Support | ✅ | CPU/GPU only | CPU only |
| Built-in Duplicate Detection | ✅ | ❌ | ❌ |
| Open Source & Easy Deployment | ✅ | Proprietary | Open source but complex |
- 🗂️ Page-Level Document Indexing: Supports fine-grained retrieval by indexing documents at the page granularity.
- 🔍 Hybrid Retriever: Combines BM25 keyword search with dense vector retrieval for robust relevance.
- 🧠 Context Assembly: Dynamically assembles prompts with token budget management for LLM inputs.
- 🔄 Duplicate Detection: Uses MinHash LSH for efficient duplicate chunk detection.
- ⚙️ Modular Design: Clear separation of concerns with cache, chunker, retrievers, and vector store components.
- 🧪 Extensive Validation: Document schema validation and metadata extraction ensure data quality.
- 📦 Pre-configured Models: Includes optimized embedding and reranker models with configurable device support.
- 🚀 LRU Cache with TTL: Improves performance by caching recent queries and embeddings with expiration.
- 🖥️ Configurable for CPU/GPU: Flexible device assignment for embedding and reranking to suit deployment environments.
flowchart LR
U[User Query] --> A[Frontend Interface]
A --> B[Query Processing]
B --> C{Hybrid Retriever}
C --> D[BM25 Retriever]
C --> E[Dense Retriever]
D --> F[Vector Store]
E --> F
F --> G[Duplicate Detector]
G --> H[Context Manager]
H --> I[Prompt Assembly]
I --> J[LLM API]
J --> K[Answer Output]
K --> A
| Component | Role | Technology / Library |
|---|---|---|
| Frontend Interface | User interaction layer | Gradio |
| Query Processing | Validates and prepares queries | Python |
| BM25 Retriever | Keyword-based retrieval | rank_bm25 |
| Dense Retriever | Semantic vector retrieval | Faiss / Chroma (vector store) |
| Vector Store | Stores embeddings and document chunks | Chroma DB / Custom VectorStore |
| Duplicate Detector | Removes near-duplicate chunks | datasketch MinHash LSH |
| Context Manager | Assembles prompt within token limits | Custom Python logic |
| LLM API | Large language model for response generation | OpenAI / HuggingFace APIs |
sequenceDiagram
actor User
participant Frontend
participant QueryProcessor
participant HybridRetriever
participant BM25Retriever
participant DenseRetriever
participant VectorStore
participant DuplicateDetector
participant ContextManager
participant LLM
User->>Frontend: Submit query
Frontend->>QueryProcessor: Validate and preprocess query
QueryProcessor->>HybridRetriever: Pass processed query
HybridRetriever->>BM25Retriever: Keyword search
HybridRetriever->>DenseRetriever: Semantic search
BM25Retriever->>VectorStore: Retrieve BM25 results
DenseRetriever->>VectorStore: Retrieve dense results
VectorStore->>DuplicateDetector: Filter duplicates
DuplicateDetector->>ContextManager: Assemble relevant docs
ContextManager->>LLM: Create prompt and request answer
LLM-->>ContextManager: Return answer
ContextManager-->>Frontend: Display answer
Frontend-->>User: Show response
Step-by-step:
- User submits a natural language query via the frontend interface.
- The query processor validates and preprocesses the query text.
- The hybrid retriever splits the retrieval task into BM25 keyword and dense vector searches.
- BM25 and dense retrievers query the vector store for relevant chunks/pages.
- Duplicate detector removes redundant or near-duplicate chunks to reduce noise.
- Context manager assembles the selected document chunks into a prompt, respecting token limits.
- The prompt is sent to the LLM API for answer generation.
- The generated answer is returned and displayed back to the user.
| Layer | Technology | Purpose |
|---|---|---|
| Frontend | Gradio | User interface for query submission |
| Backend Processing | Python | Core logic, retrieval, chunking |
| Retrieval | rank_bm25, Faiss/Chroma | Keyword and vector search |
| Caching | Python threading | LRU cache for performance |
| Embedding Models | HuggingFace Transformers | Text embeddings |
| Duplicate Detection | datasketch MinHash LSH | Near-duplicate chunk detection |
| Environment Config | dotenv | Configuration management |
- Python 3.8 or higher
- pip package manager
- Git
git clone https://github.com/Tharanika-R-Git/PRODUCTION-RAG-WITH-PAGEINDEX.git
cd PRODUCTION-RAG-WITH-PAGEINDEX
pip install -r requirements.txtcp .env.example .env
# Edit .env file to add your API keys and configuration variablesPRODUCTION-RAG-WITH-PAGEINDEX/
├── app.py # Main application entrypoint with Gradio UI
├── bm25_retriever.py # BM25 keyword retrieval implementation
├── cache.py # LRU cache with TTL for caching results
├── chunker.py # Document chunking and token estimation
├── config.py # Configuration and environment variables
├── context_manager.py # Prompt assembly and token budget management
├── data_loader.py # JSON document loading and validation
├── dense_retriever.py # Dense vector retrieval logic
├── duplicate_detector.py # Duplicate chunk detection with MinHash LSH
├── embedder.py # Embedding model wrapper
├── hybrid_retriever.py # Combines BM25 and dense retrievers
├── metadata_extractor.py # Metadata extraction from documents
├── vector_store.py # Vector storage and search backend
├── DeepSeek_v4.json # Sample large technical document dataset
├── requirements.txt # Python dependencies
├── .env.example # Sample environment file
└── README.md # Project documentation
Start the app and ask a question against the loaded document:
import app
if __name__ == "__main__":
app.run()Then open the Gradio UI in your browser, enter a query, and get answers based on the document index.
Programmatically run a query with custom prompt and retrieve top 5 answers:
from embedder import Embedder
from hybrid_retriever import HybridRetriever
from context_manager import ContextManager
from data_loader import load_documents
# Load and validate documents
data = load_documents("DeepSeek_v4.json")
# Initialize components
embedder = Embedder(model_name="BAAI/bge-small-en-v1.5", device="cpu")
retriever = HybridRetriever(data, embedder)
context_manager = ContextManager(max_tokens=8192)
# Query embedding
query_text = "Explain the hybrid attention architecture in DeepSeek V4"
query_emb = embedder.embed_text(query_text)
# Retrieve docs
results = retriever.search(query_emb, top_k=5)
# Assemble prompt
prompt = context_manager.assemble_prompt(query_text, results)
print(prompt)This example shows how to embed a query, retrieve relevant document chunks, and assemble a prompt for downstream LLM consumption.
Thank you for exploring PRODUCTION-RAG-WITH-PAGEINDEX! Contributions and feedback are warmly welcome.
This project is licensed under the MIT License.
🔗 GitHub Repo: https://github.com/Tharanika-R-Git/PRODUCTION-RAG-WITH-PAGEINDEX