Hands-On RAG System

A progressive, hands-on implementation of a Retrieval-Augmented Generation (RAG) system — built from scratch to production-ready, one phase at a time.

Stack: Python · LangChain · OpenAI · ChromaDB

---

What is RAG?

RAG gives an LLM access to your private documents at query time — no fine-tuning required. The system retrieves the most relevant chunks from your documents and injects them into the LLM prompt, grounding answers in real content.

Your documents → chunk → embed → vector DB
User question  → embed → retrieve top-k chunks → LLM → grounded answer

---

Roadmap

✅ Phase 1 — Naive RAG (done)

A working end-to-end pipeline as fast as possible.

Step	What was built
Document loading	PDF, .txt, .md support via PyPDFLoader / TextLoader
Chunking	RecursiveCharacterTextSplitter with configurable size & overlap
Embeddings	OpenAI text-embedding-3-small (1536 dims)
Vector store	ChromaDB — fully local, persisted to disk
RAG chain	LangChain LCEL: retriever → prompt → GPT-4o-mini → answer
CLI	index, ask, chat commands via argparse + rich

Usage:

# 1. Set your API key
echo "OPENAI_API_KEY=sk-..." > .env

# 2. Install dependencies
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 3. Drop documents into data/
# 4. Index them (pick a chunking strategy + embedding model)
python main.py index --path data/
python main.py index --path data/ --strategy structure --embedding openai-small

# 5. Ask a question (pick a retriever, stream, cite)
python main.py ask "What is this document about?" --retriever hybrid --cite
python main.py ask "How long should passwords be?" --stream

# 6. Or start a chat with conversational memory
python main.py chat --memory --stream --cite

# 7. Compare chunking strategies / evaluate quality
python main.py benchmark --path data/handbook.md
python main.py eval --retriever hybrid

# 8. Or launch the web UI
streamlit run app.py

A sample document (data/handbook.md) and golden eval set (data/eval/golden.json) are included so every command runs out of the box.

---

✅ Phase 2 — Chunking Strategies (done)

Understand how chunking directly affects retrieval quality. Five strategies, selectable via --strategy, plus a benchmark command to compare them.

• Fixed-size vs. sentence-aware vs. semantic chunking
• SentenceTransformersTokenTextSplitter — never breaks mid-sentence
• SemanticChunker — splits at topic boundaries, not character counts
• Document-structure-aware chunking (structure — by Markdown heading, keeps the heading path in metadata)
• Benchmarking strategies on the same document (python main.py benchmark)

---

✅ Phase 3 — Embeddings Deep Dive (done)

Swap embedding models with a single --embedding flag. Each model gets its own Chroma collection so vector spaces never mix.

• OpenAI text-embedding-3-small vs text-embedding-3-large
• Local embeddings with BAAI/bge-m3 via HuggingFace (free, private)
• nomic-embed-text and all-MiniLM-L6-v2 — open-source alternatives
• Model registry records dim + cost note for quality/cost tradeoffs (src/embedder.py)

Keys: openai-small, openai-large, bge-m3, nomic, minilm.

---

✅ Phase 4 — Advanced Retrieval (done)

Six retrievers, selectable via --retriever. See src/retrieval.py.

• Multi-query retrieval — LLM rephrases the question N ways, results unioned
• Parent-document retriever — embed small chunks, return their larger parent
• BM25 (sparse retrieval) — keyword search, reuses the persisted corpus
• Hybrid retrieval — dense + sparse fused with Reciprocal Rank Fusion (RRF)
• Reranking — over-fetch top-20, CrossEncoder re-scores to top-4

---

✅ Phase 5 — Generation & Prompting (done)

RAGPipeline in src/rag_chain.py.

• Hardened RAG prompt (context-only, explicit rules)
• Graceful "no answer" refusal when context is insufficient
• Multi-turn conversation with memory + LLM question-condensing (chat --memory)
• Streaming responses to the terminal in real time (--stream)
• Inline citations — chunks numbered [n], sources footer (--cite)

---

✅ Phase 6 — Evaluation (done)

src/evaluate.py + python main.py eval.

• Golden test set (question + expected answer + source) — data/eval/golden.json
• Built-in LLM-as-judge (faithfulness, correctness) + retrieval hit-rate — no extra deps
• RAGAS metrics (faithfulness, answer_relevancy, context_precision, context_recall) via --ragas
• Compare retrievers/embeddings on the same golden set

---

✅ Phase 7 — Advanced Topics (done)

src/advanced.py + app.py.

• HyDE (Hypothetical Document Embeddings) — embed a generated answer, not the question
• Agentic RAG — LLM writes a search query, self-grades the context, re-retrieves if weak
• Streamlit UI — streamlit run app.py
• RAPTOR / GraphRAG — documented as extensions (need a summarisation tree / graph store)

---

Project Structure

RAG/
├── .env                  # API keys (never committed)
├── requirements.txt      # Python dependencies
├── main.py               # CLI: index / ask / chat / benchmark / eval
├── app.py                # Phase 7: Streamlit web UI
├── IMPLEMENTATION.md     # Deep technical reference
├── data/
│   ├── handbook.md       # Sample document
│   └── eval/golden.json  # Phase 6: golden test set
├── notebooks/            # Exploratory notebooks (Phase 2 comparison)
├── chroma_db/            # Auto-created vector store (local, gitignored)
└── src/
    ├── loader.py         # Phase 1:     Load files → Document objects
    ├── chunker.py        # Phase 1+2:   5 chunking strategies
    ├── embedder.py       # Phase 1+3:   Pluggable embedding models → ChromaDB
    ├── retrieval.py      # Phase 4:     Multi-query / BM25 / hybrid / rerank / parent
    ├── rag_chain.py      # Phase 1+5:   Generation, streaming, citations, memory
    ├── evaluate.py       # Phase 6:     Golden-set eval (built-in judge + RAGAS)
    └── advanced.py       # Phase 7:     HyDE + agentic RAG

---

Key Parameters

Parameter	Default	Flag / File	Effect
chunk_size	1000	--chunk-size	Characters per chunk
chunk_overlap	200	--chunk-overlap	Overlap between chunks
strategy	recursive	--strategy	Chunking: recursive / sentence / token / semantic / structure
embedding	openai-small	--embedding	openai-small / openai-large / bge-m3 / nomic / minilm
retriever	vector	--retriever	vector / multi_query / bm25 / hybrid / rerank / hyde / agentic
k	4	-k	Chunks retrieved per query
LLM_MODEL	gpt-4o-mini	rag_chain.py	OpenAI generation model

---

Resources

• LangChain Docs
• ChromaDB Docs
• OpenAI Embeddings Guide
• RAGAS — RAG Evaluation
• Original RAG Paper (Lewis et al., 2020)