A Retrieval-Augmented Generation (RAG) system for searching and querying Computer Vision research papers. Ask questions in natural language and get AI-synthesized answers with sources and architecture figures.
- Features
- Demo
- How It Works
- Project Structure
- Installation
- Complete Workflow
- Usage
- Downloading Papers from arXiv
- Maintenance & Debugging
- Configuration
- Tech Stack
- Performance & Costs
- Troubleshooting
- Future Improvements
- Author
- Automatic Paper Download: Fetch papers from arXiv by topic
- Semantic Search: Find relevant papers using natural language queries (not just keywords)
- AI-Powered Answers: Get synthesized responses from retrieved passages using GPT-4o
- Architecture Figures: Search and display architecture diagrams from papers
- Multimodal Search: Combine text and figure search for comprehensive results
- Source Citations: Every answer includes paper sources with similarity scores
- Modern Web Interface: Clean UI with Ask AI and Search modes
- REST API: Full API with Swagger documentation
- Smart Filtering: Automatically filters out references and related work sections
Ask a question and get a synthesized answer:
Q: "How does person re-identification work with gait features?"
A: "Person re-identification using gait features combines appearance
and motion patterns. GAF-Net extracts gait features from skeletal
data using Graph Convolutional Networks (GCN) and fuses them with
RGB appearance features. This multimodal approach achieves 89.6%
Rank-1 accuracy on iLIDS-VID dataset..."
Sources:
- VISAPP_2024_157_CR.pdf (68.4% match)
- algorithms-17-00352-v2.pdf (58.5% match)
Related Figures:
- [GAF-Net Architecture Diagram]
# Interactive RAG
$ python src/rag.py
RAG System for Computer Vision Papers
Your question: How does attention work in transformers?
ANSWER: [Generated answer with sources]
# Multimodal search
$ python src/multimodal_rag.py
Search: transformer encoder decoder
[Results with text chunks + architecture diagrams]+------------------------------------------------------------------+
| USER QUESTION |
| "How does person re-identification work?" |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| 1. RETRIEVAL (Local) |
| +-------------+ +-------------+ +-------------+ |
| | Embedding | -> | ChromaDB | -> | Top-K | |
| | (MiniLM) | | Search | | Passages | |
| +-------------+ +-------------+ +-------------+ |
| FREE FREE FREE |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| 2. GENERATION (LLM) |
| +-------------+ +-------------+ +-------------+ |
| | Passages | -> | GPT-4o | -> | Synthesized | |
| | + Question | | mini | | Answer | |
| +-------------+ +-------------+ +-------------+ |
| ~$0.001/request |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| RESPONSE: Answer + Sources + Figures |
+------------------------------------------------------------------+
| Component | What it does | Technology |
|---|---|---|
| Embeddings | Convert text to vectors for similarity search | all-MiniLM-L6-v2 (local, free) |
| Vector DB | Store and search embeddings efficiently | ChromaDB |
| Retrieval | Find relevant passages from papers | Cosine similarity |
| Generation | Synthesize answer from passages | GPT-4o-mini |
| Figures | Extract and describe architecture diagrams | PyMuPDF + GPT-4o Vision |
rag-cv-papers/
│
├── src/ # Core application modules
│ ├── api.py # FastAPI REST API (/ask, /search endpoints)
│ ├── arxiv_fetcher.py # Download papers from arXiv automatically
│ ├── embeddings.py # Text chunking and indexing into ChromaDB
│ ├── document_processor.py # PDF processing, intelligent section filtering
│ ├── figure_extractor.py # Extract architecture figures from PDFs
│ ├── index_figures.py # Index figure descriptions into ChromaDB
│ ├── rag.py # Core RAG pipeline (retrieval + generation)
│ ├── multimodal_rag.py # Multimodal search (text + figures)
│ └── test_llm.py # Test OpenAI API connection
│
├── check_desc.py # Validate figure descriptions quality
├── debug_figures.py # Debug ChromaDB figure collection
├── debug_p5.py # Debug specific figure extraction issues
├── fix_descriptions.py # Re-generate invalid figure descriptions
│
├── static/
│ └── index.html # Web interface
│
├── data/
│ ├── papers/ # Your PDF papers (input)
│ ├── figures/ # Extracted figures (generated)
│ └── figure_descriptions.json # Figure metadata (generated)
│
├── chroma_db/ # Vector database (generated)
│ ├── cv_papers/ # Text chunks collection
│ └── cv_figures/ # Figure descriptions collection
│
├── .env # API keys (OPENAI_API_KEY)
├── .env.example # Template for environment variables
├── .gitignore # Git ignore rules
├── requirements.txt # Python dependencies
└── README.md # This file
- Python 3.10 or higher
- OpenAI API key (for Ask mode and figure descriptions)
git clone https://github.com/Moncef-Bj/rag-cv-papers.git
cd rag-cv-papers# Windows
python -m venv .venv
.venv\Scripts\activate
# Linux/Mac
python -m venv .venv
source .venv/bin/activatepip install -r requirements.txtCreate a .env file (copy from .env.example):
cp .env.example .envEdit .env and add your OpenAI API key:
OPENAI_API_KEY=sk-your-api-key-here
# Test OpenAI connection
python src/test_llm.pyIf you see "API key loaded successfully" and a response from GPT, you're ready!
Here's a complete example starting from scratch with "person re-identification" topic:
# Download 10 papers on person re-identification
python src/arxiv_fetcher.py "person re-identification"
# Or download papers on multiple topics (default behavior)
python src/arxiv_fetcher.pyOutput:
Searching arXiv for: person re-identification
Download folder: data/papers
--------------------------------------------------
Deep Learning for Person Re-identification: A Survey...
Authors: Liang Zheng, Yi Yang, Alexander G. Hauptmann
Published: 2016-10-13
Downloaded: 1610.02984_Deep_Learning_for_Person_Re-identification.pdf
... (9 more papers)
==================================================
Fetched 10 papers
Next: Run 'python src/embeddings.py' to index new papers
python src/embeddings.pyWhat happens:
- Loads all PDFs from
data/papers/ - Extracts text and filters out references, related work
- Chunks documents into ~1000 character pieces
- Generates embeddings (vectors) for each chunk
- Stores everything in ChromaDB
Output:
============================================================
STEP 1: Loading and chunking PDFs
============================================================
Found 10 PDF files
--------------------------------------------------
Processing: 1610.02984_Deep_Learning_for_Person_Re...
Pages: 23
Chunks: 47 -> 31 (filtered 16)
...
PROCESSING SUMMARY
============================================================
Total chunks processed: 458
Chunks kept: 312
Filtered (references): 89
Filtered (related work): 38
Filtered (low quality): 19
Filter rate: 31.9%
============================================================
STEP 2: Creating vector store
============================================================
Vector store created!
Total vectors: 312
# Extract architecture diagrams from PDFs
python src/figure_extractor.pyWhat happens:
- Scans PDFs for figures (pages 1-10)
- Filters by size, aspect ratio
- Saves to
data/figures/ - Generates descriptions using GPT-4o Vision
- Saves metadata to
data/figure_descriptions.json
Cost: ~$0.01 per figure
Output:
============================================================
EXTRACTING ARCHITECTURE FIGURES FROM CV PAPERS
============================================================
Processing: 1610.02984_Deep_Learning_for_Person_Re...
Found 3 candidate figures
Saved: 1610.02984_Deep_Learning_p3_img1.png
Description: "A CNN architecture showing ResNet-50 backbone..."
...
SUMMARY
============================================================
Papers processed: 10
Figures extracted: 27
Descriptions generated: 27
python src/index_figures.pyOutput:
============================================================
INDEXING FIGURES INTO CHROMADB
============================================================
Total figures: 27
Valid descriptions: 25
Indexed 25 figures!
Collection: cv_figures
# Check figure descriptions quality
python check_desc.py
# Verify what's in ChromaDB
python debug_figures.pyOption A: Web Interface
python -m uvicorn src.api:app --reload --port 8000
# Open http://localhost:8000Option B: Command Line RAG
python src/rag.pyOption C: Multimodal Search
python src/multimodal_rag.py- Start the server:
python -m uvicorn src.api:app --reload --port 8000-
Open http://localhost:8000 in your browser
-
Choose mode:
- Ask AI: Get synthesized answers with sources
- Search: Get raw search results
-
Example questions:
- "How does person re-identification work?"
- "What are the main challenges in multi-camera tracking?"
- "Explain the difference between CNN and Transformer architectures"
- "What loss functions are used for metric learning?"
python src/rag.pyExample session:
============================================================
RAG System for Computer Vision Papers
============================================================
Ask questions about the papers in your database.
Type 'quit' to exit.
Your question: How does person re-identification work?
Retrieving context for: How does person re-identification work?
Context retrieved (3421 chars)
Generating answer...
ANSWER:
Person re-identification (Re-ID) aims to match pedestrian images across
non-overlapping camera views. Modern approaches use deep learning with
metric learning objectives. The typical pipeline involves:
1. Feature Extraction: CNN or Transformer backbones extract appearance features
2. Metric Learning: Triplet loss or contrastive loss learn discriminative features
3. Matching: Compute similarity between query and gallery features
According to the papers, state-of-the-art methods achieve 89-95% Rank-1
accuracy on standard benchmarks like Market-1501 and DukeMTMC-reID.
[Source: 1610.02984_Deep_Learning_for_Person_Re-identification.pdf, Page 4-6]
Your question: quit
Goodbye!
python src/multimodal_rag.pyExample session:
======================================================================
ARCHITECTURE SEARCH ENGINE
======================================================================
Search for CV architectures using natural language.
Example queries:
- transformer encoder decoder
- object detection CNN backbone
- multi-camera tracking
- attention mechanism for video
Search: person re-identification CNN architecture
======================================================================
SEARCH: "person re-identification CNN architecture"
======================================================================
Found: 5 text chunks, 3 figures
----------------------------------------------------------------------
COMBINED RESULTS (sorted by relevance)
----------------------------------------------------------------------
[1] Text | Similarity: 0.842
Source: 1610.02984_Deep_Learning_for_Person_Re-identificatio...
Page: 5
Content: Deep convolutional networks for person Re-ID typically use
ResNet or DenseNet backbones. The network learns a metric...
[2] Figure | Similarity: 0.791
Source: algorithms-17-00352-v2.pdf
Page: 8
Title: GAF-Net Architecture Overview
Description: The architecture shows a dual-branch network with CNN
for appearance and GCN for gait features...
File: algorithms-17-00352-v2_p8_img2.png
[3] Text | Similarity: 0.756
Source: VISAPP_2024_157_CR.pdf
Page: 3
Content: The proposed architecture uses a ResNet-50 backbone
pretrained on ImageNet, followed by a global average...
----------------------------------------------------------------------
ARCHITECTURE FIGURES
----------------------------------------------------------------------
[1] algorithms-17-00352-v2_p8_img2.png
Paper: algorithms-17-00352-v2.pdf
Path: data/figures/algorithms-17-00352-v2_p8_img2.png
Title: GAF-Net Architecture Overview
[2] VISAPP_2024_157_CR_p5_img4.png
Paper: VISAPP_2024_157_CR.pdf
Path: data/figures/VISAPP_2024_157_CR_p5_img4.png
Title: Feature Fusion Module
[3] 1610.02984_Deep_Learning_p7_img1.png
Paper: 1610.02984_Deep_Learning_for_Person_Re-identification.pdf
Path: data/figures/1610.02984_Deep_Learning_p7_img1.png
Single topic:
python src/arxiv_fetcher.py "person re-identification"Multiple topics (default):
python src/arxiv_fetcher.pySuggested Topics for CV Research:
# Person Re-identification & Tracking
"person re-identification"
"multi-camera tracking"
"pedestrian detection"
"gait recognition"
# Human Pose & Activity
"human pose estimation"
"action recognition"
"3D human reconstruction"
# Object Detection & Segmentation
"object detection transformer"
"instance segmentation"
"semantic segmentation"
# Video Understanding
"video object tracking"
"temporal action detection"
"video captioning"
# 3D Vision
"visual SLAM"
"3D object detection"
"depth estimation"
# Transformers & Attention
"vision transformer"
"self-attention mechanism"
"cross-attention fusion"
# Few-shot & Meta-learning
"few-shot learning"
"meta-learning computer vision"
# Domain Adaptation
"domain adaptation"
"transfer learning"Custom download example:
# Download 20 papers on transformers for CV
python src/arxiv_fetcher.py "vision transformer" --max-results 20| Endpoint | Method | Description |
|---|---|---|
/ |
GET | Web interface |
/health |
GET | Health check |
/stats |
GET | Collection statistics |
/search |
GET | Semantic search (retrieval only) |
/ask |
GET | Ask AI (RAG with LLM synthesis) |
/figures/{name} |
GET | Serve figure images |
/docs |
GET | Swagger API documentation |
Health check:
curl http://localhost:8000/healthGet stats:
curl http://localhost:8000/statsResponse:
{
"text_chunks": 312,
"figures": 25,
"papers": 10
}Search (retrieval only):
curl "http://localhost:8000/search?q=person+re-identification&n_text=5&n_figures=3"Ask AI (full RAG):
curl "http://localhost:8000/ask?q=How+does+person+re-identification+work"Response:
{
"question": "How does person re-identification work?",
"answer": "Person re-identification (Re-ID) is the task of matching...",
"sources": [
{
"paper": "1610.02984_Deep_Learning_for_Person_Re-identification.pdf",
"content": "Person re-identification aims to match pedestrians...",
"similarity": 0.842,
"page": 5
}
],
"figures": [
{
"filename": "algorithms-17-00352-v2_p8_img2.png",
"title": "GAF-Net Architecture",
"description": "Dual-branch network combining appearance and gait..."
}
]
}The arxiv_fetcher.py tool automatically downloads research papers from arXiv.
# Single topic
python src/arxiv_fetcher.py "person re-identification"
# Multiple topics (default config)
python src/arxiv_fetcher.pyWhen run without arguments, it downloads papers on these topics:
1. person re-identification (15 papers)
2. multi-camera tracking (15 papers)
3. pedestrian detection (15 papers)
4. gait recognition (15 papers)
5. human pose estimation (15 papers)
6. object detection transformer (15 papers)
7. video object tracking (15 papers)
8. action recognition (15 papers)
9. 3D human reconstruction (15 papers)
10. visual SLAM (15 papers)
Total: 150 papers
Edit src/arxiv_fetcher.py to customize:
# Change topics
topics = [
"your topic 1",
"your topic 2",
]
# Change papers per topic
papers = fetch_multiple_topics(topics=topics, papers_per_topic=20)Papers are saved to data/papers/ with filenames like:
2312.12345_Paper_Title_Here.pdf
python check_desc.pyValidates that all figure descriptions are valid (not GPT refusals).
Output:
============================================================
CHECKING VISAPP AND ALGORITHM FIGURES
============================================================
Filename: VISAPP_2024_157_CR_p5_img4.png
Source: VISAPP_2024_157_CR.pdf
Description length: 234 chars
Description: The architecture diagram shows a feature fusion module
that combines appearance and gait features using...
STATUS: Description looks OK!
python debug_figures.pyShows what's actually indexed in ChromaDB.
Output:
============================================================
FIGURES IN CHROMADB
============================================================
algorithms-17-00352-v2.pdf -> algorithms-17-00352-v2_p8_img2.png
VISAPP_2024_157_CR.pdf -> VISAPP_2024_157_CR_p5_img4.png
1610.02984_Deep_Learning_for_Per... -> 1610.02984_Deep_Learning_p7_img1.png
============================================================
SEARCHING FOR VISAPP AND ALGORITHM FIGURES
============================================================
FOUND:
source_pdf: VISAPP_2024_157_CR.pdf
filename: VISAPP_2024_157_CR_p5_img4.png
title: Feature Fusion Architecture
python debug_p5.pyDetailed debugging for why a specific figure wasn't extracted/indexed.
python fix_descriptions.pyRe-generates descriptions for figures that have invalid descriptions (GPT refusals).
Output:
============================================================
RE-DESCRIBING INVALID FIGURES
============================================================
Re-describing: paper_p3_img2.png
Old description: I cannot analyze this image...
New description: The diagram shows a ResNet architecture with skip...
STATUS: Success!
============================================================
Updated 3 figures
Saved to data/figure_descriptions.json
Now run: python src/index_figures.py
python src/test_llm.pyVerifies that your OpenAI API key works correctly.
The system intelligently filters paper sections:
| Kept | Filtered |
|---|---|
| Abstract | References |
| Introduction | Related Work |
| Method/Approach | Acknowledgments |
| Architecture | Appendix |
| Experiments | Bibliography |
| Results | |
| Conclusion |
This filtering reduces noise and improves search quality by ~30%.
In src/figure_extractor.py:
MIN_IMAGE_SIZE = 20000 # Minimum 20KB (filters out small icons)
MIN_WIDTH = 300 # Minimum width in pixels
MIN_HEIGHT = 200 # Minimum height in pixels
MIN_ASPECT_RATIO = 0.8 # Accept vertical/square figures
TARGET_PAGES = (1, 10) # Extract from pages 1-10 only
MAX_FIGURES_PER_PAPER = 3 # Max figures per paper
MAX_TOTAL_FIGURES = 100 # Total limit across all papersIn src/embeddings.py and src/index_figures.py:
USE_OPENAI = False # Set to True to use OpenAI embeddings (paid)
# False = use local all-MiniLM-L6-v2 (free)Trade-offs:
| Model | Cost | Quality | Speed |
|---|---|---|---|
| all-MiniLM-L6-v2 (local) | FREE | Good | Fast |
| OpenAI text-embedding-ada-002 | ~$0.10/1M tokens | Better | Slower |
For most use cases, the free local model is sufficient.
| Component | Technology | Purpose |
|---|---|---|
| Backend | FastAPI | REST API framework |
| Vector DB | ChromaDB | Store and search embeddings |
| Embeddings | all-MiniLM-L6-v2 | Convert text to vectors (local, free) |
| LLM | OpenAI GPT-4o-mini | Answer synthesis |
| Vision LLM | OpenAI GPT-4o | Figure descriptions |
| PDF Processing | PyMuPDF (fitz) | Extract text and images |
| Text Splitting | LangChain | Chunk documents |
| arXiv API | arxiv (Python) | Download papers |
| Frontend | HTML/CSS/JavaScript | Web interface |
- ChromaDB: Simple, embedded vector DB - no server needed
- MiniLM: Free, local embeddings - no API costs for search
- GPT-4o-mini: Good quality at low cost (~$0.15/1M tokens)
- FastAPI: Modern, fast, automatic API docs
- PyMuPDF: Fast PDF processing with image extraction
| Metric | Value |
|---|---|
| Indexing speed | ~100 papers/minute |
| Search latency | < 100ms |
| Ask latency | 2-5 seconds (LLM dependent) |
| Storage | ~10MB per 100 papers |
| RAM usage | ~500MB with 100 papers indexed |
| Operation | Cost | Frequency |
|---|---|---|
| Text indexing | FREE | One-time per paper |
| Search | FREE | Every search |
| Ask AI | ~$0.001 | Per question |
| Figure extraction | ~$0.01 | One-time per figure |
| Figure description | ~$0.01 | One-time per figure |
Example Cost for 100 papers:
- Text indexing: $0 (free, local)
- 30 figures @ $0.01 each: $0.30
- 100 questions @ $0.001 each: $0.10
- Total: < $1 for complete setup + 100 questions
- Use local embeddings (already default) - saves 100% on search costs
- Limit figure extraction to key papers - reduce figure description costs
- Use GPT-4o-mini (not GPT-4o) for answers - 10x cheaper
- Cache results - avoid re-asking same questions
pip install -r requirements.txtCreate .env file with your API key:
echo "OPENAI_API_KEY=sk-your-key-here" > .envDownload papers first:
python src/arxiv_fetcher.py "your topic"Or manually add PDFs to data/papers/.
Index your papers first:
python src/embeddings.pyRun figure extraction and indexing:
python src/figure_extractor.py
python src/index_figures.pyRe-generate descriptions:
python fix_descriptions.py
python src/index_figures.pyChromaDB is persistent. If it's empty, you need to re-run:
python src/embeddings.py
python src/index_figures.py- Set
USE_OPENAI = Falseinembeddings.py(use free local embeddings) - Use GPT-4o-mini instead of GPT-4o
- Reduce number of retrieved chunks:
TOP_K = 3inrag.py
-
Check if paper sections are properly filtered:
python src/document_processor.py
-
Increase chunk overlap for better context:
# In document_processor.py chunk_overlap = 300 # Increase from 200
-
Use more specific queries
- LaTeX equation rendering - Proper mathematical notation in answers
- Docker containerization - One-command deployment
- PDF upload via web interface - No command line needed
- Paper summarization - Auto-generate TL;DR for each paper
- CLIP embeddings - Visual similarity search for figures
- Citation graph visualization - Explore paper relationships
- Automatic paper recommendations - Based on reading history
- Streamlit dashboard - Advanced analytics
- Export to Notion/Obsidian - Note-taking integrations
- Collaborative annotations - Highlight and share insights
Contributions are welcome! Please:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
- LangChain for document processing utilities
- ChromaDB for simple and powerful vector storage
- FastAPI for the excellent API framework
- OpenAI for GPT models and embeddings
- Sentence Transformers for free local embeddings
- arXiv for open access to research papers
If you use this project in your research, please cite:
@software{boujou2025cvrag,
author = {Boujou, Moncef},
title = {CV Papers RAG: Multimodal Research Assistant for Computer Vision},
year = {2025},
url = {https://github.com/yourusername/rag-cv-papers}
}If you find this project useful, please consider giving it a star!
Built with care by a CV researcher, for CV researchers