Epsilon RAG

A 2024 local-first RAG experiment for Discord and the command line.

Epsilon RAG is an older personal project originally built in June 2024 as an early exploration of retrieval-augmented generation. It turns PDF documents into a searchable local knowledge base, retrieves relevant passages with Chroma, and generates answers through a local Ollama model for Discord or the command line.

The repository has since been reorganized and documented for archival and portfolio purposes. The core concept and demo remain representative of the original 2024 prototype rather than a newly launched production system.

Project background

This project began as a practical experiment to learn how document retrieval, vector embeddings, local language models, and Discord bots could work together. The prototype indexed a personal profile document and allowed questions to be asked directly inside a Discord channel using the !ask command.

The screenshot below was captured during the original prototype testing on 25 June 2024.

Demo

The screenshot shows the original Epsilon prototype answering questions from an indexed personal profile document directly inside Discord. Users invoke the assistant with !ask, while the bot retrieves relevant context and generates a natural-language response.

Example interaction

User: !ask what are Mohammad Raffy's interests?

Epsilon: Mohammad Raffy has a great interest in Data Science and is also
very interested in Artificial Intelligence.

Highlights

• Local-first generation with Ollama; document context stays on your machine.
• Incremental PDF indexing with stable chunk identifiers.
• Semantic retrieval powered by Hugging Face embeddings and Chroma.
• Discord !ask command plus a standalone query CLI.
• Environment-based secrets and centralized configuration.
• Reproducible HotpotQA reranker quality-latency benchmark with Pareto selection.

How it works

1. PDF files from data/ are loaded and split into overlapping text chunks.
2. Each chunk is converted into a semantic embedding with Sentence Transformers.
3. Chroma stores the embeddings and retrieves the chunks closest to a question.
4. The retrieved text and question are passed to Mistral through Ollama.
5. The answer and its document sources are returned through Discord or the CLI.

Architecture

PDF files → text chunks → embeddings → Chroma
                                      ↓
Discord / CLI → semantic search → Ollama → grounded answer

Tech stack

Layer	Technology
Language	Python 3.11+
RAG orchestration	LangChain
Local LLM	Ollama + Mistral
Embeddings	sentence-transformers/all-MiniLM-L6-v2
Vector database	Chroma
Interface	Discord.py and CLI

Getting started

Prerequisites

• Python 3.11 or newer
• Ollama installed and running
• A Discord bot token if you want to use the Discord interface

Installation

git clone https://github.com/zzeiidann/Epsilon-RAG.git
cd Epsilon-RAG
python -m venv .venv
source .venv/bin/activate
pip install -e .
cp .env.example .env
ollama pull mistral

Place one or more PDF files inside data/. This directory is intentionally ignored by Git so private documents are not published.

Build the knowledge base

epsilon-index

Use epsilon-index --reset to rebuild the vector store from scratch.

Ask from the terminal

epsilon-query "What are the key points in the document?"

Run the Discord bot

Set DISCORD_TOKEN in .env, enable the Message Content Intent in the Discord Developer Portal, then run:

epsilon-bot

In a channel where the bot is present:

!ask What does the document say about this topic?

Project structure

.
├── benchmarks/
│   ├── reranking/              # Independent retrieval and reranking benchmark
│   └── rag_inference/          # HotpotQA ranking-to-Ollama benchmark
├── data/                       # Local PDF documents (not tracked)
├── docs/assets/                # Published reranking and Ollama benchmark plots
├── src/epsilon_rag/
│   ├── bot.py                  # Discord interface
│   ├── cli.py                  # Terminal interface
│   ├── config.py               # Environment and paths
│   ├── embeddings.py           # Embedding model factory
│   ├── rag.py                  # Retrieval and generation
│   └── store.py                # PDF indexing pipeline
├── tests/                      # Ranking metric, feature, and dataset tests
├── .env.example
├── pyproject.toml
└── README.md

Configuration

Variable	Default	Purpose
DISCORD_TOKEN	Not set	Discord bot authentication token
OLLAMA_MODEL	mistral	Ollama model used for generation
EMBEDDING_MODEL	sentence-transformers/all-MiniLM-L6-v2	Hugging Face embedding model
RAG_TOP_K	7	Number of document chunks retrieved

Use cases

• Personal knowledge assistants built from notes, profiles, or reference documents
• Internal question-answering bots for teams and communities
• Document exploration without sending source material to a hosted LLM API
• Prototyping local retrieval-augmented generation workflows

Limitations

• The indexing pipeline currently accepts PDF files only.
• Answer quality depends on the source documents, selected model, and retrieved context.
• Ollama must remain running while queries are processed.
• Discord messages are truncated to the platform's 2,000-character limit.
• This project does not yet include authentication or per-server knowledge bases.

Reranker Quality-Latency Benchmark

The repository includes an independent retrieval experiment under benchmarks/reranking/. It asks whether a low-cost LambdaMART reranker can retain much of a neural Cross-Encoder's ranking quality while reducing production latency. It does not invoke Discord, Chroma, Ollama, or answer generation, and its ranking metrics do not measure final answer correctness.

Protocol

The benchmark uses the Hugging Face hotpotqa/hotpot_qa dataset's distractor configuration. Each question has a constrained set of candidate Wikipedia paragraphs. One document is constructed per normalized context title as [TITLE] ...\n[TEXT] ...; unique titles in supporting_facts provide binary document relevance labels. Duplicate supporting sentences for the same title therefore cannot inflate relevance.

By default, exactly 4,000 complete training-query groups are sampled deterministically with seed 42. Those query IDs are split into 3,200 development-training and 800 internal-development queries. Optuna sees only these two partitions. After selection, the chosen configuration is retrained on all 4,000 sampled queries and evaluated on an untouched prefix of 1,000 official validation queries. Set --validation-limit 7405 to evaluate the complete distractor validation split. The 4,000-query choice makes repeatable local training practical; it is not claimed to be the globally optimal training size.

The compared methods are:

• dense cosine retrieval with the application's default sentence-transformers/all-MiniLM-L6-v2 model;
• an untuned, fixed-parameter LightGBM LGBMRanker(objective="lambdarank");
• a tuned LambdaMART model selected from a multi-objective Optuna Pareto frontier; and
• configurable batched cross-encoder/ms-marco-MiniLM-L4-v2 reranking, on CUDA when available and CPU otherwise.

LambdaMART uses dense similarity, BM25, lexical overlap and coverage, title overlap, query and document lengths, and dense/BM25 ranks. Relevance, supporting identities, and answer text are not features. Distractor paragraphs remain in their original query groups and act as hard negatives. Optional --feature-ablation compares dense-only, dense+BM25, lexical, and complete feature sets on internal development data.

Quality reporting includes Recall@k, Hit@k, first-relevant MRR, binary NDCG@k, unique supporting-document recall, and Full-Support Hit@k at 1, 3, 5, and 10. Full-Support Hit is one only when every gold document is present. A k larger than a query's candidate count is omitted from that query's aggregate rather than silently changing k.

Latency and production selection

Models are loaded and warmed before timing. Dataset download, model loading, initialization, and cached document embedding computation are excluded. time.perf_counter() measures:

• dense: query embedding, cosine scoring, and sorting (document vectors are cached);
• LambdaMART reranker-only: BM25/lexical feature computation, prediction, and sorting;
• Cross-Encoder reranker-only: pair preparation, batched neural inference, and sorting; and
• end-to-end ranking-stage latency: dense latency plus reranker-only latency.

Reports include mean, p50, p95, and p99 end-to-end latency, component means/p95s, and QPS from the actual timed query batch. These boundaries matter: reranker-only and end-to-end values are not interchangeable. CUDA is synchronized around Cross-Encoder inference. Cross-Encoder batching may improve throughput while changing per-query latency, so both are retained.

Optuna maximizes internal-development NDCG@5 and minimizes P95 total reranking latency. All Pareto trials are saved. Production selection chooses the highest-NDCG Pareto model satisfying --latency-sla-ms (5 ms by default); when none qualifies, it explicitly records the lowest-latency Pareto model and sla_met=false. Latency is hardware-dependent, so every run also records CPU, CUDA/GPU, Python, and package information.

Published benchmark results

The following results are from the reproducible standard run completed on 4 September 2026. All four methods were evaluated on the same HotpotQA validation subset. These are retrieval and reranking measurements only; no LLM answers were generated.

Setting	Value
Dataset	hotpotqa/hotpot_qa, distractor
Sampled training queries	4,000
Development train / internal development	3,200 / 800 query groups
Official validation queries	1,000, excluded from model development
Random seed	42
Candidate count	10 or the number actually available
Evaluation cutoffs	1, 3, 5, 10
Dense model	sentence-transformers/all-MiniLM-L6-v2
Cross-Encoder	cross-encoder/ms-marco-MiniLM-L4-v2
Cross-Encoder batch size	32
Optuna trials	30, multi-objective
Reranker P95 SLA	5 ms
Host	Apple ARM64 CPU, no CUDA; Python 3.11.14

The initially considered MiniLM-L6 Cross-Encoder returned non-finite logits with the tested macOS runtime. It was not used for the published comparison. The lighter configurable L4 checkpoint produced finite scores for every evaluated pair. The benchmark now fails fast if a Cross-Encoder returns a missing, NaN, or infinite score.

Ranking quality

Method	NDCG@1	NDCG@3	NDCG@5	NDCG@10	MRR
Dense	0.8260	0.7271	0.7796	0.8478	0.8921
LambdaMART untuned	0.9040	0.8427	0.8815	0.9132	0.9460
LambdaMART tuned	0.9140	0.8488	0.8896	0.9177	0.9517
Cross-Encoder	0.8940	0.8060	0.8392	0.8920	0.9395

Method	Recall@1	Recall@3	Recall@5	Recall@10
Dense	0.4130	0.7200	0.8241	1.0000
LambdaMART untuned	0.4520	0.8449	0.9214	1.0000
LambdaMART tuned	0.4570	0.8499	0.9304	1.0000
Cross-Encoder	0.4470	0.7981	0.8634	1.0000

Method	Hit@1	Hit@3	Hit@5	Hit@10
Dense	0.8260	0.9507	0.9798	1.0000
LambdaMART untuned	0.9040	0.9879	0.9950	1.0000
LambdaMART tuned	0.9140	0.9889	0.9960	1.0000
Cross-Encoder	0.8940	0.9869	0.9960	1.0000

Method	Full-Support Hit@1	Full-Support Hit@3	Full-Support Hit@5	Full-Support Hit@10
Dense	0.0000	0.4894	0.6683	1.0000
LambdaMART untuned	0.0000	0.7019	0.8478	1.0000
LambdaMART tuned	0.0000	0.7110	0.8649	1.0000
Cross-Encoder	0.0000	0.6093	0.7308	1.0000

Supporting-document Recall@k equals Recall@k here because relevance is defined at the unique normalized supporting-title level. Full-Support Hit@1 is necessarily zero for the evaluated two-document multi-hop questions. Values at k=10 reflect the constrained distractor candidate set and must not be interpreted as large-corpus retrieval performance.

Latency and throughput

Method	Mean	P50	P95	P99	QPS
Dense	6.09 ms	5.90 ms	7.44 ms	8.52 ms	160.87
LambdaMART untuned	9.90 ms	9.69 ms	11.90 ms	12.92 ms	99.12
LambdaMART tuned	10.60 ms	10.22 ms	12.67 ms	14.06 ms	92.54
Cross-Encoder	256.91 ms	229.13 ms	480.37 ms	604.07 ms	3.89

These values are end-to-end ranking-stage measurements. The component breakdown explains the additional cost after dense retrieval:

Method	Mean retrieval	Mean features/preparation	Mean model	Mean reranker	P95 reranker
Dense	6.09 ms	0.00 ms	0.00 ms	0.00 ms	0.00 ms
LambdaMART untuned	8.49 ms	1.21 ms	0.15 ms	1.41 ms	1.82 ms
LambdaMART tuned	9.09 ms	1.28 ms	0.19 ms	1.52 ms	2.00 ms
Cross-Encoder	10.83 ms	0.01 ms	245.96 ms	246.08 ms	470.07 ms

The tuned LambdaMART model improves NDCG@5 by 14.1% relative to dense and raises Full-Support Hit@5 from 0.6683 to 0.8649. Compared with this Cross-Encoder checkpoint, it achieves 6.0% higher NDCG@5, approximately 38 times lower end-to-end P95 latency, 235 times lower reranker-only P95 latency, and 23.8 times higher throughput. Dense remains the fastest choice when minimum latency matters more than multi-document recall.

Tuning adds 0.0081 absolute NDCG@5 and 0.0171 Full-Support Hit@5 over the untuned ranker at a small latency cost. Its final-validation reranker P95 is 2.00 ms, safely inside the 5 ms SLA.

Pareto selection

Trial	Internal-dev NDCG@5	P95 reranker	Trees	Leaves	Depth	Selected
3	0.89755	1.01349 ms	282	69	5	No
5	0.90265	1.01392 ms	305	15	5	Yes

Both Pareto configurations met the 5 ms SLA. Trial 5 was selected because it had the highest NDCG@5 among SLA-compliant models. Its learning rate is 0.10080, minimum child samples is 34, feature fraction is 0.8550, bagging fraction is 0.9549, bagging frequency is 3, alpha regularization is 0.00000845, and lambda regularization is 0.03685.

Visualizations

The three subplots use a logarithmic end-to-end P95 ranking-stage latency axis so both lightweight and neural methods remain readable. The Pareto plot uses reranker-only P95 measured on the internal-development queries, which is the Optuna latency objective.

The result supports the benchmark's research hypothesis for this constrained HotpotQA setting: the inexpensive tuned ranker provides a better observed quality-latency operating point than the tested neural reranker. It does not establish that LambdaMART will outperform all Cross-Encoders, hardware configurations, candidate depths, or large-corpus retrievers.

Ollama retrieval-to-generation experiment

An additional local experiment passes the same HotpotQA distractor candidates through dense, tuned LambdaMART, or Cross-Encoder ranking and then sends the top five documents to Ollama. It does not require Discord or Chroma. This measures the user-visible path from query ranking to a completed generated answer while retaining HotpotQA reference answers for Exact Match (EM) and token F1 evaluation.

The published run used the first 30 untouched official-validation questions, candidate k=10, context k=5, seed 42, temperature 0, and at most 32 generated tokens. Ollama 0.33.3 ran qwen2.5:1.5b locally on the same 8 GB Apple ARM64 machine. The small model was selected to fit the available memory and disk; these results should not be interpreted as the quality ceiling of Ollama or of larger language models.

Method	EM	Token F1	Mean ranking	Mean TTFT	P50 total	P95 total	Output tok/s	QPS
Dense	0.3667	0.4961	8.68 ms	1,322.86 ms	1,347.11 ms	1,850.44 ms	60.67	0.710
LambdaMART tuned	0.4333	0.5837	10.26 ms	1,034.97 ms	1,161.60 ms	1,642.99 ms	57.96	0.882
Cross-Encoder	0.3667	0.5202	320.02 ms	1,061.14 ms	1,402.16 ms	2,306.29 ms	58.83	0.679

On this 30-query run, tuned LambdaMART improves EM by 6.7 percentage points and token F1 by 8.8 points over dense ranking. Its P95 retrieval-to-generation latency is also 11.2% lower. This total-latency difference is not solely reranker speed: different rankings produce contexts with different token lengths, which changes Ollama prompt-evaluation time. Cross-Encoder adds a much larger ranking cost and has the highest observed P95 total latency.

Model loading, dataset download, and document-embedding cache preparation are excluded. TTFT is measured from the local Ollama request until the first non-empty streamed token. Total latency runs from query embedding through the completed response. QPS is measured from each method's sequential 30-query batch, not derived from P95 and not a concurrent-server load test. Thirty queries are enough for a reproducible local comparison but too few for a definitive answer- quality claim.

Install and start Ollama on macOS, then download the benchmark model:

brew install --cask ollama-app
open -a Ollama
ollama pull qwen2.5:1.5b

The official macOS guide documents installation and storage. The benchmark uses the local streaming /api/generate endpoint and its native token/duration fields as described in the Ollama API usage guide.

Reproduce the published generation experiment:

python -m benchmarks.rag_inference.benchmark \
  --validation-limit 30 \
  --methods dense lambdamart cross_encoder \
  --candidate-k 10 \
  --context-k 5 \
  --ollama-model qwen2.5:1.5b \
  --num-predict 32

Outputs are written to results/rag_inference/: method summary and per-query CSVs, configuration/hardware JSON, and the generated comparison plot. These run artifacts remain Git-ignored; the representative plot and documented measurements above are versioned.

Install and run

Install the isolated benchmark dependencies without changing the runtime behavior of the existing app:

pip install -e '.[benchmark,test]'

Fast real-data smoke test (first use downloads HotpotQA and the embedding model):

python -m benchmarks.reranking.benchmark \
  --train-limit 100 \
  --validation-limit 50 \
  --optuna-trials 2 \
  --methods dense lambdamart

An offline implementation smoke test is also available and is clearly marked as synthetic; its values are not research results:

python -m benchmarks.reranking.benchmark \
  --dataset synthetic \
  --embedding-model hashing \
  --train-limit 20 \
  --validation-limit 5 \
  --optuna-trials 2 \
  --methods dense lambdamart

Standard benchmark:

python -m benchmarks.reranking.benchmark \
  --dataset hotpotqa \
  --config distractor \
  --train-limit 4000 \
  --validation-limit 1000 \
  --seed 42 \
  --candidate-k 10 \
  --eval-k 1 3 5 10 \
  --optuna-trials 30 \
  --latency-sla-ms 5

Use --methods dense, --methods dense lambdamart, or --methods dense cross_encoder to run selected systems. Pass multiple values such as --candidate-k 5 10 for a candidate-count sweep. For complete validation, use the standard command with --validation-limit 7405.

Outputs are written to results/reranking/: summary and per-query CSVs, all Optuna trials, the Pareto table, optional feature ablations, configuration/hardware JSON, and two PNG figures. Selected models are stored in artifacts/rerankers/; generated results, caches, and models are Git-ignored.

Methodological limitations

HotpotQA distractor is a constrained ten-paragraph candidate task, not large-corpus retrieval. Its binary supporting-document labels do not express graded relevance. Timings are hardware- and batching-dependent. The intentionally limited training sample may leave quality on the table, and ranking gains do not guarantee that a downstream LLM would answer correctly.

Roadmap

• Support additional document formats such as Markdown and DOCX
• Add conversational memory and follow-up questions
• Stream responses in Discord
• Add automated tests and continuous integration
• Support separate collections for multiple Discord servers

Troubleshooting

The bot starts but does not respond to !ask. Ensure Message Content Intent is enabled both in the Discord Developer Portal and in the bot configuration.

Ollama cannot connect. Run ollama serve, verify that it is available locally, and download the configured model with ollama pull mistral.

No relevant information is returned. Add PDFs to data/ and run epsilon-index --reset before querying again.

The first indexing run is slow. The embedding model is downloaded on first use and cached for subsequent runs.

Privacy

PDF files, the generated Chroma database, environment secrets, and model artifacts are excluded from version control. Before publishing a previously private repository, inspect its Git history as ignored files may still exist in earlier commits.

License

Distributed under the MIT License.