🔌 MCP Docs Assistant

Production-grade Retrieval-Augmented Generation (RAG) pipeline over the official Model Context Protocol documentation — served over REST, MCP tools, and Docker.

Ask natural-language questions about MCP (architecture, building servers/clients, tools/resources/prompts, security) and get answers grounded in the real docs, with guardrails, PII masking, reranking, semantic caching, and hallucination checking built in.

---

✨ Features

Capability	Implementation
🔀 Multi-key LLM gateway	Portkey-routed, load-balanced across 2 Gemini keys + 2 Groq keys, with automatic provider fallback
📚 Grounded retrieval	Official MCP docs, chunked + embedded into a persistent Qdrant vector store
🎯 Reranking	Cross-encoder (ms-marco-MiniLM-L-6-v2) narrows a wide candidate pool down to the most relevant chunks
🛡️ Guardrails	NeMo Guardrails (Colang 2.x) — input/output safety checks, jailbreak + instruction-leak detection
🕵️ PII masking	Microsoft Presidio — masks emails, phone numbers, credit cards in both input and output
🧮 Token budgeting	Retrieved context is greedily fit to a fixed token budget before hitting the LLM
⚡ Semantic cache	Embedding-similarity cache (not exact-match) with TTL + size cap
💬 Multi-turn conversations	LangGraph checkpointer + follow-up query condensation ("show a Python example of that")
🔍 Hallucination check	Runtime LLM-as-judge verdict (GROUNDED / HALLUCINATED) on every generated answer
📊 Offline evaluation	RAGAS metrics (faithfulness, relevancy, context precision/recall) against 25 reference Q&A pairs
🔌 MCP-native	Exposes itself as MCP tools (ask_mcp_docs, search_mcp_docs, …) — usable directly from Claude Desktop, Claude Code, or any MCP host
🌐 REST API	FastAPI endpoints for any regular HTTP client
🐳 Dockerized	One-command deploy with docker compose up

---

🏗️ Architecture

flowchart TD
    A[User Question] --> B[Guard Input<br/>NeMo Guardrails]
    B -->|blocked| Z[Refusal message]
    B -->|allowed| C[Mask Input PII<br/>Presidio]
    C --> D[Condense Follow-up<br/>into standalone question]
    D --> E{Semantic<br/>Cache Hit?}
    E -->|yes| F[Return cached answer]
    E -->|no| G[Retrieve Top-15<br/>Qdrant Vector Store]
    G --> H[Rerank Top-5<br/>Cross-Encoder]
    H --> I[Fit to Token Budget]
    I --> J[Generate Answer<br/>Portkey: Gemini / Groq]
    J --> K[Guard Output<br/>leak / PII pattern check]
    K --> L[Hallucination Check<br/>LLM-as-judge]
    L --> M[Mask Output PII]
    M --> N[Cache + Store History]
    N --> O[Return Answer]

Loading

Every node above is a module in rag_pipeline/, wired together as a LangGraph StateGraph in rag_pipeline/graph.py. rag_core.py builds every dependency once (as a singleton) and exposes a small stable API — chat(), search(), get_history(), cache_stats() — consumed identically by both the REST layer (main.py) and the MCP layer (mcp_server.py), so a single vector store / cache / conversation history is shared no matter which interface a request comes through.

---

📁 Project Structure

mcp-docs-rag-assistant/
├── main.py                    # FastAPI app — REST endpoints + mounts MCP at /mcp
├── mcp_server.py               # MCP tools (stdio standalone, or mounted in main.py)
├── rag_core.py                 # Singleton facade wiring the whole pipeline together
├── rag_pipeline/
│   ├── config.py                 # Env vars / secrets (single source of truth)
│   ├── logging_setup.py          # Logging + Logfire
│   ├── gateway.py                 # Portkey multi-key LLM gateway
│   ├── errors.py                   # Retry + safe-node error handling
│   ├── ingestion.py                 # MCP docs loader + splitter
│   ├── vectorstore.py                # Embeddings + persistent Qdrant store
│   ├── reranker.py                    # Cross-encoder reranking
│   ├── pii_masking.py                  # Presidio PII masking
│   ├── guardrails.py                    # NeMo Guardrails (Colang 2.x)
│   ├── token_management.py               # Context window budgeting
│   ├── semantic_cache.py                  # Embedding-similarity cache
│   ├── query_condensation.py               # Follow-up question rewriting
│   ├── hallucination.py                     # Runtime hallucination judge
│   └── graph.py                              # LangGraph StateGraph — full pipeline
├── scripts/
│   └── evaluate_ragas.py        # Offline RAGAS evaluation (25 reference Q&A)
├── tests/
│   └── test_pipeline.py         # Fast smoke tests (no API keys needed)
├── configs/guardrails/           # Colang rail files (generated at first run)
├── data/                          # Persisted Qdrant vector store (gitignored)
├── notebooks/                      # Original development notebook
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env.example

---

🚀 Getting Started

Prerequisites

• Python 3.11+
• API keys: Google AI Studio (Gemini, ×2), Groq (×2), Portkey (gateway + config id)

1. Clone and set up a virtual environment

git clone https://github.com/<your-username>/mcp-docs-rag-assistant.git
cd mcp-docs-rag-assistant
python -m venv venv
venv\Scripts\activate          # Windows
# source venv/bin/activate     # macOS/Linux

2. Install dependencies

pip install -r requirements.txt
python -m spacy download en_core_web_sm   # required by Presidio for PII detection

3. Configure environment variables

cp .env.example .env

Open .env and fill in your real keys (GEMINI_API_KEY_1/2, GROQ_API_KEY_1/2, PORTKEY_API_KEY, PORTKEY_CONFIG_ID).

4. Run the server

uvicorn main:app --reload

⏳ First run only: the vector store doesn't exist yet, so the server ingests the MCP docs and embeds them in rate-limited batches — this can take 5–10 minutes. On every subsequent run it loads the persisted store from data/qdrant_mcp_db/ instantly.

Once you see Startup: RAG pipeline ready., open:

• http://localhost:8000/docs — interactive Swagger UI, try POST /chat
• http://localhost:8000/health — health check

---

📡 REST API

Method	Endpoint	Description
POST	/chat	Ask a question. Body: {"question": "...", "thread_id": "optional"}
POST	/search	Retrieve + rerank raw context, no generation. Body: {"query": "...", "top_n": 5}
GET	/history/{thread_id}	Get conversation history for a thread
GET	/cache/stats	Semantic cache observability
GET	/health	Health check

---

🔌 Using it as an MCP Server

Standalone (stdio) — for Claude Desktop

Run directly:

python mcp_server.py

Or point a local MCP host at it, e.g. in Claude Desktop's config (claude_desktop_config.json):

{
  "mcpServers": {
    "mcp-docs-assistant": {
      "command": "python",
      "args": ["E:\\mcp-docs-rag-assistant\\mcp_server.py"]
    }
  }
}

Remote (streamable-http) — via FastAPI

When main.py is running, the same MCP tools are reachable at:

http://localhost:8000/mcp

Available tools: ask_mcp_docs, search_mcp_docs, get_conversation_history, cache_stats.

---

🐳 Docker

The only prerequisite is Docker Desktop (which bundles Docker Compose) — you do not need to separately install Python, the pip dependencies, or spacy's model on your machine. All of that happens automatically inside the image when you build it (see the Dockerfile — it runs pip install -r requirements.txt and python -m spacy download en_core_web_sm as build steps).

# 1. Make sure .env exists (same as the local setup, step 3 above)
cp .env.example .env   # then fill in real keys

# 2. Build and run
docker compose up --build

That single command builds the image, installs everything inside it, and starts the container. The data/ folder is mounted as a volume (see docker-compose.yml), so the vector store persists across container restarts — you only pay the slow first-run ingestion cost once, even with Docker.

Server is reachable the same way as running locally: http://localhost:8000/docs.

To stop:

docker compose down

To rebuild after changing code or dependencies:

docker compose up --build

---

🧪 Testing

Fast smoke tests — no API keys or network calls needed (uses fake embeddings):

pip install pytest
pytest tests/ -v

---

📊 Offline Evaluation (RAGAS)

Scores the pipeline against 25 hand-written MCP questions with reference answers, using RAGAS:

python scripts/evaluate_ragas.py

This does not need the server running — it builds the pipeline itself (same singleton as main.py/mcp_server.py) and prints a metrics table:

• Faithfulness — is the answer grounded in the retrieved context?
• Response Relevancy — does the answer actually address the question?
• Context Precision — is the retrieved context relevant?
• Context Recall — does retrieved context cover what the reference answer needs?

⏱️ Takes a few minutes: each of the 25 questions runs a real retrieval + generation pass, then every metric is itself scored by an LLM-as-judge call.

---

⚙️ Configuration Reference

All configuration lives in .env (see .env.example). Key variables:

Variable	Purpose
GEMINI_API_KEY_1/2, GROQ_API_KEY_1/2	Provider keys, load-balanced by Portkey
PORTKEY_API_KEY, PORTKEY_CONFIG_ID	Portkey gateway credentials + routing config
QDRANT_PATH, QDRANT_COLLECTION	Vector store location/name
LOGFIRE_TOKEN	Optional — omit to fall back to console-only logging
HOST, PORT	Server bind address

---

🛠️ Tech Stack

FastAPI · LangChain · LangGraph · Qdrant · Portkey · Sentence-Transformers · Presidio · NeMo Guardrails · RAGAS · MCP Python SDK · Docker

---

📄 License

MIT — see LICENSE.