FinLLM RAG Evaluation

This repository contains a modular RAG evaluation pipeline for LLMs for finance related QA systems. This was part of my thesis work.

Capabilities:

• Fetch and preprocess SEC 10K filings
• Chunk documents for retrieval
• Build a FAISS vectorstore for RAG
• Run generation experiments (QA)
• Evaluate retriever, generator, and faithfulness (RAGAS) performance

---

System Architecture Overview

---

Project Setup

1. Clone the repository

git clone https://github.com/0xBuro/FinLLM-RAG-Eval
cd FinLLM_RAG_Eval

2. Create a virtual environment

python -m venv venv
source venv/bin/activate  # Linux/macOS
venv\Scripts\activate     # Windows

3. Install dependencies

pip install -r requirements.txt

4. Add credentials in .env

SEC_API_KEY=your_sec_api_key
EDU_MAIL_ACC=your_edu_email
OPENAI_API_KEY=your_openai_api_key

---

Directory Structure

FinLLM_RAG_Eval/
├── config/
│   ├── __init__.py
│   └── constants.py         # All constants, paths, tickers, and model settings
├── data/
│   ├── api/                 # fetcher
│   ├── corpus/              # Raw + chunked documents
├── eval/
│   ├── eval_faith.py        # Evaluates faithfulness
│   ├── eval_generation.py   # Evaluates generator performance
│   ├── eval_retrieval.py    # Evaluates retriever performance
│   ├── faith_generation.py  # Generates json for faithfulness
│   └── qa_generation.py     # Generates QA responses for RAG evaluation
├── helper/
│   ├── data_loader.py       # Helper functions for sample data loading
│   ├── models.py            # Pydantic models for data validation
│   ├── nlg_metrics.py       # NLG calculation metrics
│   └── prompts.py           # Zero-Shot Prompt templates
├── scripts/
│   ├── create_corpus.py     # Fetch filings
│   ├── chunk_sections.py    # Chunk documents
│   └── build_index.py       # Build FAISS vectorstore
├── results/                 # Output JSONs from generation/evaluation
├── main.py                  # Pipeline orchestrator
├── requirements.txt
└── README.md

---

Data Pipeline

Fetch filings

python -m scripts.create_corpus

• Fetches 10-K filings for all tickers in config/constants.py
• Saves raw filings to data/corpus/

Chunk documents

python -m scripts.chunk_sections

• Chunks raw filings into smaller sections
• Saves chunks to data/corpus/i.e AAPL_2025

Build RAG index

python -m scripts.build_index

• Loads chunked documents
• Computes embeddings (EMBEDDING_MODEL in config/constants.py)
• Builds a FAISS vectorstore
• Saves index to data/vectorstore/

---

Generation Pipeline

QA generation

python -m eval.qa_generation

• Runs local inference to generate QA responses
• Saves results to results/qa_generation_results.json

Faithfulness evaluation generation

python -m eval.faith_generation

• Uses GPT-3.5 as a judge to generate JSON for RAGAS faithfulness evaluation
• Saves results to results/faithfulness_eval_results.json

These scripts do not need to be rerun if the JSON outputs already exist.

---

Evaluation Pipeline

Run evaluations

python -m main --eval

• Evaluates:

  • Retriever quality (eval/eval_retrieval.py)
  • Generator performance (eval/eval_generation.py)
  • Faithfulness (eval/eval_faith.py)

• Uses previously generated JSON files

• Saves evaluation outputs in results/

---

Full Pipeline Orchestration

python -m main

By default, main.py will:

1. Preprocess the corpus
2. Build embeddings / RAG index
3. Run generation experiments
4. Run evaluation scripts

You can also run pipelines individually:

# Only data preprocessing
python -m main --data

# Only generation
python -m main --gen

# Only evaluation
python -m main --eval

---