diff --git a/RESULTS.md b/RESULTS.md index f4adca3c..3c08dbe4 100644 --- a/RESULTS.md +++ b/RESULTS.md @@ -16,12 +16,46 @@ The counts above describe the tracked repository and CI matrix. They do not incl ## 1. Verified Measured Runs -### RAG Assistant: embedding retrieval versus cross-encoder re-ranking +### RAG Assistant: versioned deterministic retrieval benchmark + +- **Run date:** 2026-07-27 +- **Environment:** local CPU; deterministic network-free `HashEmbedder`, 512 dimensions +- **Corpus:** 10 checked-in documents, 31 chunks +- **Cases:** 40 checked-in labeled retrieval cases +- **Chunking:** 400 characters with 80-character overlap +- **Metric cutoff:** fixed recall@1 and recall@3; MRR over the first relevant source + +| Metric | Baseline | +|---|---:| +| recall@1 | 0.500 | +| recall@3 | 0.775 | +| MRR | 0.629 | + +| Required category | Cases | recall@1 | recall@3 | MRR | +|---|---:|---:|---:|---:| +| Direct lookup | 8 | 0.250 | 0.750 | 0.479 | +| Paraphrase | 8 | 0.625 | 0.750 | 0.688 | +| Terminology | 8 | 0.500 | 0.875 | 0.688 | +| Cross-chunk | 8 | 0.750 | 0.750 | 0.750 | +| Hard negative | 8 | 0.375 | 0.750 | 0.542 | + +This result is a project regression baseline, not a leaderboard or production-quality retrieval claim. HashEmbedder is intentionally weak and deterministic. It makes CI sensitive to changes in corpus loading, chunking, labels, metric aggregation, and ranking behavior without downloading model weights. + +The checked-in comparison rule rejects a change when recall@1 falls below 0.450, recall@3 falls below 0.725, MRR falls below 0.579, or any required category loses more than one of its eight top-three hits. Comparisons must keep the corpus, labels, chunk settings, and embedder fixed. + +Regenerate the deterministic report: + +```bash +cd ai_engineering/rag_assistant +python -m smoke.run_smoke +``` + +### RAG Assistant: historical five-case MiniLM re-ranking smoke run - **Run date:** 2026-07-26 - **Environment:** CPU; `sentence-transformers` 5.6.1; `transformers` 5.14.1; `torch` 2.13.0 -- **Corpus:** three checked-in sample documents -- **Cases:** five checked-in retrieval cases +- **Corpus:** three checked-in sample documents at the time of the run +- **Cases:** five retrieval cases at the time of the run - **Metric cutoff:** k = 3 | Configuration | recall@3 | MRR | @@ -30,9 +64,9 @@ The counts above describe the tracked repository and CI matrix. They do not incl | Cross-encoder re-ranked | 1.000 | 1.000 | | Observed delta | +0.000 | +0.100 | -The re-ranker moved one relevant result from rank 2 to rank 1. This is a scoped smoke result, not evidence of a general quality lift. [Issue 18](https://github.com/lmdixon23/my_dev_projects/issues/18) tracks the required discriminative benchmark. +The re-ranker moved one relevant result from rank 2 to rank 1. This result predates the expanded benchmark and is retained as historical smoke evidence. It is not directly comparable with the 40-case baseline and does not establish a general quality lift. -Regenerate the comparison: +Run the current MiniLM comparison on the expanded benchmark: ```bash cd ai_engineering/rag_assistant @@ -91,7 +125,7 @@ These tracked projects provide a `smoke/run_smoke.py` entry point. A smoke run e | Project | Command | Evidence produced | |---|---|---| -| [RAG Assistant](./ai_engineering/rag_assistant/) | `cd ai_engineering/rag_assistant && python -m smoke.run_smoke` | Ingest, retrieve, and retrieval metrics on checked-in documents | +| [RAG Assistant](./ai_engineering/rag_assistant/) | `cd ai_engineering/rag_assistant && python -m smoke.run_smoke` | Versioned 40-case deterministic retrieval benchmark with aggregate, category, and per-case metrics | | [Agent Toolkit](./ai_engineering/agent_toolkit/) | `cd ai_engineering/agent_toolkit && python -m smoke.run_smoke` | Deterministic agent/tool execution and trace output | | [LLM Eval Harness](./ai_engineering/llm_eval_harness/) | `cd ai_engineering/llm_eval_harness && python -m smoke.run_smoke` | Evaluation-suite execution and report generation | | [Regularized Operator Zoo](./ai_engineering/rlvr/regularized_operator_zoo/) | `cd ai_engineering/rlvr/regularized_operator_zoo && python -m smoke.run_smoke` | Operator identity checks and beta-sweep artifacts | @@ -112,7 +146,9 @@ These projects are validated primarily through deterministic tests, compilation, |---|---|---| | [NLP Text Summarization CLI](./ai_engineering/nlp_text_summarization_api/) | `cd ai_engineering/nlp_text_summarization_api && python -m unittest discover tests` | Network-free HTTP-path, persistence, concurrency, and retry tests; no retained summary-quality benchmark | -The RAG-to-evaluation bridge is also enforced separately: +The RAG Assistant suite contains **34 tests across 6 files**, including metric aggregation, multi-source labels, missing hits, category slicing, benchmark validation, and deterministic regression thresholds. + +The RAG-to-evaluation bridge is enforced separately: ```bash cd ai_engineering/llm_eval_harness @@ -200,4 +236,4 @@ for path in smoke_scripts: PY_AUDIT ``` -The release gate also checks relative links, stale claims, whitespace, RAG tests, the RAG-to-eval bridge, and the exact three-file change boundary. +The v1.1.0 release gate checked links, stale claims, whitespace, RAG tests, the RAG-to-evaluation bridge, and exact release-documentation boundaries. Issue 18 adds the versioned retrieval benchmark and its separate regression gate. diff --git a/ai_engineering/rag_assistant/README.md b/ai_engineering/rag_assistant/README.md index 62af7f7f..3faaf601 100644 --- a/ai_engineering/rag_assistant/README.md +++ b/ai_engineering/rag_assistant/README.md @@ -4,32 +4,68 @@ RAG Assistant is a modular Retrieval-Augmented Generation system written in Python. It chunks documents, embeds them, stores normalized vectors in FAISS or a NumPy fallback, retrieves relevant context, optionally re-ranks a larger candidate pool with a cross-encoder, and generates cited answers through the OpenAI Chat Completions API. -The repository includes a CLI, a Flask service, retrieval metrics, deterministic network-free tests, a basic smoke pipeline, and a baseline-versus-re-ranked comparison command. +The repository includes a CLI, Flask service, versioned retrieval benchmark, deterministic regression baseline, network-free tests, smoke report, and baseline-versus-re-ranked comparison command. -## Verified Snapshot +## Verified Retrieval Benchmark -A CPU run completed on 2026-07-26 with `sentence-transformers` 5.6.1, `transformers` 5.14.1, and `torch` 2.13.0: +The checked-in `rag-retrieval-v1` benchmark contains **40 labeled cases over 10 documents and 31 chunks**. It includes eight cases in each required category: -| Configuration | recall@3 | MRR | -|---|---:|---:| -| Embedding retrieval | 1.000 | 0.900 | -| Cross-encoder re-ranked | 1.000 | 1.000 | -| Observed delta | +0.000 | +0.100 | +- direct lookup; +- paraphrase; +- terminology and acronym variation; +- cross-chunk wording; +- hard negatives with plausible distractors. -The re-ranker moved one relevant result from rank 2 to rank 1. This is a five-case smoke measurement over three small documents. It demonstrates that the comparison path works; it does not establish a general retrieval-quality lift. [Issue 18](https://github.com/lmdixon23/my_dev_projects/issues/18) tracks the required 30-50 case discriminative benchmark. +The deterministic HashEmbedder baseline uses 512 dimensions with 400-character chunks and 80-character overlap: + +| Metric | Baseline | +|---|---:| +| recall@1 | 0.500 | +| recall@3 | 0.775 | +| MRR | 0.629 | + +| Category | Cases | recall@1 | recall@3 | MRR | +|---|---:|---:|---:|---:| +| Direct lookup | 8 | 0.250 | 0.750 | 0.479 | +| Paraphrase | 8 | 0.625 | 0.750 | 0.688 | +| Terminology | 8 | 0.500 | 0.875 | 0.688 | +| Cross-chunk | 8 | 0.750 | 0.750 | 0.750 | +| Hard negative | 8 | 0.375 | 0.750 | 0.542 | + +This is a project regression baseline, not a leaderboard or production-quality retrieval claim. HashEmbedder is intentionally weak and deterministic. Its purpose is to make CI sensitive to changes in corpus loading, chunking, labels, metric aggregation, and ranking behavior without downloading model weights. + +The earlier five-case MiniLM comparison from 2026-07-26 produced recall@3 `1.000` for both configurations and MRR `0.900` versus `1.000` after re-ranking. That result is historical and not directly comparable with the expanded benchmark. Run the current MiniLM comparison before making a new re-ranking claim. + +## Metric Definitions and Regression Rule + +- **recall@1**: fraction of cases with an accepted source at rank 1. +- **recall@3**: fraction with an accepted source in the top three. +- **MRR**: mean reciprocal rank of the first accepted source; a miss contributes zero. +- **Multi-source label**: any listed relevant source can satisfy the case. +- **Category slice**: the same metrics aggregated only over cases carrying a given tag. + +The checked-in regression thresholds allow a small bounded change while rejecting material losses: + +- recall@1 must remain at least `0.450`; +- recall@3 must remain at least `0.725`; +- MRR must remain at least `0.579`; +- no required category may lose more than one of its eight top-three hits. + +Comparisons are valid only when the corpus, labels, chunk settings, and embedder remain fixed. A changed benchmark version requires a new recorded baseline. ## Key Features - **Paragraph-first chunking**: Long paragraphs fall back to sentence-level splitting, with configurable character overlap. -- **Pluggable embeddings**: `SentenceTransformerEmbedder` is the default; `HashEmbedder` provides deterministic, network-free tests. +- **Pluggable embeddings**: `SentenceTransformerEmbedder` is the production-like default; `HashEmbedder` provides deterministic network-free evaluation. - **FAISS with NumPy fallback**: Normalized vectors use inner-product search, equivalent to cosine similarity after normalization. -- **Optional cross-encoder re-ranking**: Embedding retrieval remains the default. An opt-in cross-encoder re-scores a configurable candidate pool before the final top-k is returned. -- **Score provenance**: Re-ranked responses expose both the final cross-encoder `score` and the original embedding `retrieval_score`. -- **Cited generation with an empty-context guard**: The generator requests source tags and refuses to produce a grounded answer when retrieval returns no context. -- **Retrieval evaluation**: `eval_retrieval` reports recall@k, MRR, and per-case results. -- **CLI, Flask, and Docker surfaces**: Ingestion, asking, evaluation, serving, and container execution use the same pipeline components. -- **Reproducible comparison**: `python -m smoke.run_reranker_eval` records aggregate and per-case changes between embedding-only and re-ranked retrieval. -- **Network-free validation**: The RAG suite contains 19 tests across 4 files. A separate two-test bridge verifies compatibility with the sibling LLM Eval Harness. +- **Optional cross-encoder re-ranking**: Embedding retrieval remains the default. An opt-in cross-encoder re-scores a configurable candidate pool. +- **Score provenance**: Re-ranked responses expose the final cross-encoder score and original embedding retrieval score. +- **Cited generation with an empty-context guard**: The generator requests source tags and refuses a grounded answer when retrieval returns no context. +- **Versioned evaluation schema**: Stable case IDs and tags extend the legacy `question` and `relevant_sources` fields. +- **Dataset validation**: Tests enforce unique IDs, non-empty labels, existing sources, and required category coverage. +- **Retrieval metrics**: `eval_retrieval` reports recall@1, recall@3, configurable recall@k, MRR, per-case ranks, and tag slices. +- **Reproducible comparison**: `python -m smoke.run_reranker_eval` compares embedding-only and re-ranked retrieval on the same benchmark. +- **Network-free validation**: The RAG suite contains 34 tests across 6 files. A separate two-test bridge verifies compatibility with the sibling LLM Eval Harness. ## Architecture @@ -52,9 +88,10 @@ flowchart TD end subgraph evaluation["Evaluation"] - T["Labeled cases"] --> R - R --> M["recall@k and MRR"] - B["Baseline versus re-ranked runner"] --> M + T["40 labeled cases"] --> R + R --> M["recall@1, recall@3, MRR"] + M --> S["Per-case and category slices"] + B["Versioned Hash baseline"] --> S end ``` @@ -62,25 +99,29 @@ flowchart TD ```text rag/ - chunker.py Document and chunk types; paragraph-first splitting - embedder.py Embedder protocol, HashEmbedder, SentenceTransformerEmbedder - vector_store.py FAISS IndexFlatIP with NumPy fallback and persistence - retriever.py Embedding retrieval and optional candidate-pool expansion - reranker.py Reranker protocol and CrossEncoderReranker - generator.py OpenAI generation, source formatting, empty-context guard - eval.py EvalCase, recall@k, MRR, and per-case results - pipeline.py End-to-end ingest, retrieve, optional re-rank, and generate -cli.py ingest, ask, serve, and eval commands -app.py Flask /ask and /health endpoints -sample_docs/ Three sample documents and five labeled cases -smoke/run_smoke.py Embedding-only end-to-end smoke evaluation -smoke/run_reranker_eval.py Baseline-versus-re-ranked CPU comparison -reports/ Generated local reports; values may vary by environment + chunker.py Document and chunk types; paragraph-first splitting + embedder.py HashEmbedder and SentenceTransformerEmbedder + vector_store.py FAISS IndexFlatIP with NumPy fallback and persistence + retriever.py Embedding retrieval and optional candidate expansion + reranker.py Reranker protocol and CrossEncoderReranker + generator.py OpenAI generation, source formatting, empty-context guard + eval.py Case loading, validation, ranking metrics, and slices + pipeline.py End-to-end ingest, retrieve, optional re-rank, generate +cli.py ingest, ask, serve, and eval commands +sample_docs/ + *.md Ten overlapping and confusable benchmark documents + eval_cases.json Forty stable-ID cases with source labels and tags + hash_baseline_v1.json Deterministic baseline and regression thresholds +smoke/run_smoke.py Hash baseline by default; optional local MiniLM run +smoke/run_reranker_eval.py MiniLM baseline-versus-re-ranked comparison +reports/ Generated local reports tests/ test_chunker.py test_generator.py test_pipeline.py test_reranker.py + test_eval.py + test_benchmark.py Dockerfile requirements.txt .env.example @@ -92,7 +133,7 @@ requirements.txt - Python 3.10+ - An OpenAI API key only for answer generation -- Downloadable sentence-transformers model weights for the default embedder and real cross-encoder comparison +- Downloadable sentence-transformers weights only for MiniLM or real cross-encoder runs ### Installation @@ -125,28 +166,46 @@ python cli.py ask \ -k 5 ``` -### Evaluate Retrieval +### Run the Deterministic Benchmark ```bash -# Embedding-only evaluation -python cli.py eval \ - --store ./index \ - --cases ./sample_docs/eval_cases.json \ - -k 3 +python -m smoke.run_smoke +``` + +The default command uses `HashEmbedder`, writes `reports/smoke_eval.md`, and reproduces the checked-in regression baseline without network access. + +Run a local MiniLM benchmark against the same corpus and labels: -# Re-ranked evaluation +```bash +RAG_EVAL_EMBEDDER=minilm python -m smoke.run_smoke +``` + +Windows PowerShell: + +```powershell +$env:RAG_EVAL_EMBEDDER = "minilm" +python -m smoke.run_smoke +Remove-Item Env:RAG_EVAL_EMBEDDER +``` + +### Evaluate a Saved Store + +```bash python cli.py eval \ --store ./index \ --cases ./sample_docs/eval_cases.json \ - --rerank \ - --candidate-k 20 \ -k 3 +``` + +The CLI prints aggregate metrics, per-case first relevant ranks, and all tag slices. Legacy case files containing only `question` and `relevant_sources` still load, although the versioned benchmark requires IDs and tags. -# Reproducible baseline comparison +### Compare the Re-ranker + +```bash python -m smoke.run_reranker_eval ``` -The comparison writes `reports/reranker_eval.md` with aggregate metrics, deltas, and per-case reciprocal-rank changes. +This local-model command writes `reports/reranker_eval.md` with recall@1, recall@3, MRR, required category slices, and per-case rank changes. It requires sentence-transformers model weights and does not require an OpenAI API key. ### Serve over HTTP @@ -177,7 +236,7 @@ curl -X POST http://localhost:8080/ask \ python -m pytest tests/ -v ``` -The current RAG suite contains **19 tests across 4 files**. The tests use deterministic embeddings, an injected fake cross-encoder, and `httpx.MockTransport`, so they require no model download, network call, or API key. +The RAG suite contains **34 tests across 6 files**. It covers chunking, generation, pipeline persistence, re-ranking, metric aggregation, multi-source labels, missing hits, category slices, benchmark validation, and deterministic regression thresholds. Tests use deterministic embeddings, an injected fake cross-encoder, and `httpx.MockTransport`, so they require no model download, network call, or API key. The cross-project bridge is run from the sibling harness: @@ -191,31 +250,28 @@ The bridge contains **2 tests** and checks both a successful grading contract an ## Result Interpretation -The five-case result is useful for release verification because it confirms that: - -1. the baseline and re-ranked paths execute against the same corpus and labels; -2. the candidate-pool and score-provenance logic are wired correctly; -3. aggregate and per-case deltas are reported reproducibly. +The expanded benchmark is designed to compare retrieval changes on a fixed project corpus. It is materially more discriminative than the previous five-case smoke suite because it includes confusable documents, hard negatives, multiple acceptable sources, stable case IDs, and category slices. -It is not large or difficult enough to support a broad comparative claim. The next evidence gate is [issue 18](https://github.com/lmdixon23/my_dev_projects/issues/18), which requires confusable sources, hard negatives, category slices, recall@1, recall@3, MRR, deterministic and real-model baselines, and regression checks. +The deterministic HashEmbedder score should not be read as model quality. It establishes that the benchmark and metric path are reproducible in CI. MiniLM, re-ranking, token-aware chunking, or hybrid retrieval should be evaluated on the same benchmark and reported as a before-and-after comparison. ## Scope and Limitations -- The checked-in benchmark has five cases over three small documents and is intentionally described as a smoke evaluation. +- The benchmark is a small synthetic technical corpus, not a public retrieval leaderboard. +- HashEmbedder is a deterministic test instrument rather than a semantic model. - The chunker is character-based rather than token-aware. -- Cross-encoder re-ranking is optional and requires external model weights for a real-model run; CI tests the control flow with an injected deterministic model. -- Retrieval is dense-only. BM25, hybrid retrieval, and reciprocal-rank fusion are not implemented. +- Cross-encoder re-ranking is optional and requires external model weights for a real-model run. +- Retrieval is dense-only. BM25, hybrid retrieval, and reciprocal-rank fusion are documented benchmark topics, not implemented production features. - Generation is single-shot; the Flask service does not expose streaming responses. -- The default embedding model is English-oriented. Multilingual retrieval has not been benchmarked in this repository. +- The default embedding model is English-oriented. Multilingual retrieval is represented conceptually but not evaluated with a multilingual corpus. - The project is a compact reference system, not a multi-tenant production service. ## Future Enhancements -1. **Complete issue 18**: Build the 30-50 case discriminative benchmark before changing retrieval algorithms. -2. **Token-aware chunking ablation**: Compare the current splitter against a tokenizer-aware alternative on the expanded benchmark. -3. **Hybrid retrieval**: Add BM25 and reciprocal-rank fusion only after the benchmark can measure gains and regressions. -4. **Streaming responses**: Add a Server-Sent Events path for generation. -5. **Namespace isolation**: Add explicit index namespaces after retrieval quality and evaluation coverage are stable. +1. **Token-aware chunking ablation**: Compare the current splitter with a tokenizer-aware alternative on `rag-retrieval-v1`. +2. **Expanded MiniLM and re-ranker run**: Record real-model aggregate and category results on the new benchmark. +3. **Hybrid retrieval**: Add BM25 and reciprocal-rank fusion only as a separate measured change. +4. **Streaming responses**: Add a Server-Sent Events generation path. +5. **Namespace isolation**: Add explicit index namespaces and authorization tests. ## References diff --git a/ai_engineering/rag_assistant/cli.py b/ai_engineering/rag_assistant/cli.py index 4c757ac3..55220fb0 100644 --- a/ai_engineering/rag_assistant/cli.py +++ b/ai_engineering/rag_assistant/cli.py @@ -4,21 +4,12 @@ ingest Index a directory of .md / .txt / .pdf files into a store on disk. ask Query a saved store and print the model's answer. serve Run the Flask API. - eval Run a retrieval-quality eval from a YAML/JSON cases file. - -Usage: - python cli.py ingest --docs-dir ./sample_docs --store ./index - python cli.py ask --store ./index --question "What is RAG?" - python cli.py ask --store ./index --question "What is RAG?" --rerank - python cli.py serve --store ./index --port 8080 - python cli.py eval --store ./index --cases ./sample_docs/eval_cases.json -k 3 - python cli.py eval --store ./index --cases ./sample_docs/eval_cases.json -k 3 --rerank + eval Run retrieval evaluation from a JSON cases file. """ from __future__ import annotations import argparse -import json import os import sys from pathlib import Path @@ -29,10 +20,10 @@ CrossEncoderReranker, DEFAULT_RERANKER_MODEL, Document, - EvalCase, RAGPipeline, Reranker, eval_retrieval, + load_eval_cases, ) from rag.embedder import make_default_embedder from rag.vector_store import VectorStore @@ -52,7 +43,9 @@ def load_docs_from_dir(dir_path: str) -> List[Document]: except ImportError: print(f"skipping {path}: pypdf not installed", file=sys.stderr) continue - text = "\n\n".join(p.extract_text() or "" for p in PdfReader(str(path)).pages) + text = "\n\n".join( + page.extract_text() or "" for page in PdfReader(str(path)).pages + ) else: text = path.read_text(encoding="utf-8", errors="ignore") docs.append(Document(source=str(path), text=text)) @@ -89,10 +82,13 @@ def cmd_ingest(args: argparse.Namespace) -> None: if not docs: sys.exit(f"no .md/.txt/.pdf files found under {args.docs_dir}") pipeline = RAGPipeline.from_env() - pipeline.chunker = Chunker(chunk_size=args.chunk_size, chunk_overlap=args.chunk_overlap) - n = pipeline.ingest(docs) + pipeline.chunker = Chunker( + chunk_size=args.chunk_size, + chunk_overlap=args.chunk_overlap, + ) + n_chunks = pipeline.ingest(docs) pipeline.save(args.store) - print(f"Indexed {len(docs)} docs -> {n} chunks; saved to {args.store}") + print(f"Indexed {len(docs)} docs -> {n_chunks} chunks; saved to {args.store}") def cmd_ask(args: argparse.Namespace) -> None: @@ -109,19 +105,20 @@ def cmd_ask(args: argparse.Namespace) -> None: def cmd_serve(args: argparse.Namespace) -> None: - from app import build_app # local import to avoid forcing Flask for non-serve users + from app import build_app + build_app(args.store).run(host="0.0.0.0", port=args.port) def cmd_eval(args: argparse.Namespace) -> None: - with open(args.cases, "r", encoding="utf-8") as fh: - raw = json.load(fh) - cases = [EvalCase(question=c["question"], relevant_sources=c["relevant_sources"]) for c in raw] + cases = load_eval_cases(args.cases) embedder = make_default_embedder() store = VectorStore.load(args.store) if embedder.dim != store.dim: sys.exit(f"embedder dim {embedder.dim} != store dim {store.dim}; reindex.") + from rag.retriever import Retriever + result = eval_retrieval( Retriever( embedder, @@ -134,7 +131,22 @@ def cmd_eval(args: argparse.Namespace) -> None: ) print(result) for row in result.per_case: - print(f" - {row['question'][:60]:60s} recall={row['recall']:.0f} rr={row['reciprocal_rank']:.3f}") + case_id = row["id"] or "legacy" + rank = row["first_relevant_rank"] or "miss" + print( + f" - {case_id:12s} rank={str(rank):4s} " + f"rr={row['reciprocal_rank']:.3f} {row['question'][:72]}" + ) + + if result.by_tag: + print("\nTag slices:") + for tag, metrics in result.by_tag.items(): + print( + f" - {tag:18s} n={metrics['n_cases']:2d} " + f"recall@1={metrics['recall_at_1']:.3f} " + f"recall@3={metrics['recall_at_3']:.3f} " + f"MRR={metrics['mrr']:.3f}" + ) def main() -> None: @@ -163,7 +175,7 @@ def main() -> None: p_eval = sub.add_parser("eval") p_eval.add_argument("--store", required=True) p_eval.add_argument("--cases", required=True) - p_eval.add_argument("-k", type=int, default=5) + p_eval.add_argument("-k", type=int, default=3) _add_reranker_args(p_eval) p_eval.set_defaults(func=cmd_eval) diff --git a/ai_engineering/rag_assistant/rag/__init__.py b/ai_engineering/rag_assistant/rag/__init__.py index 3dc1da9f..4c3c6bc1 100644 --- a/ai_engineering/rag_assistant/rag/__init__.py +++ b/ai_engineering/rag_assistant/rag/__init__.py @@ -1,17 +1,7 @@ """RAG (Retrieval-Augmented Generation) toolkit. -Exposes: - Chunker - split documents into overlapping chunks - Embedder - convert chunks to dense vectors - VectorStore - FAISS-backed nearest-neighbor index with metadata - Retriever - embedding retrieval with optional cross-encoder re-ranking - Reranker - protocol for re-ordering an initial candidate pool - Generator - calls the OpenAI Chat Completions API with retrieved context - RAGPipeline - end-to-end: ingest -> retrieve -> generate - eval_retrieval - retrieval-quality metrics (recall@k, MRR) - -`RAGPipeline.from_env()` reads `OPENAI_API_KEY`, `OPENAI_MODEL`, and -`EMBEDDING_MODEL` so test fixtures and the CLI agree on configuration. +Exposes chunking, embedding, storage, retrieval, optional re-ranking, +generation, the end-to-end pipeline, and versioned retrieval evaluation. """ from .chunker import Chunker, Document, Chunk @@ -25,7 +15,14 @@ from .retriever import Retriever, RetrievedChunk from .generator import Generator from .pipeline import RAGPipeline -from .eval import eval_retrieval, EvalResult, EvalCase +from .eval import ( + REQUIRED_BENCHMARK_TAGS, + EvalCase, + EvalResult, + eval_retrieval, + load_eval_cases, + validate_eval_cases, +) __all__ = [ "Chunker", "Document", "Chunk", @@ -35,5 +32,7 @@ "Retriever", "RetrievedChunk", "Generator", "RAGPipeline", - "eval_retrieval", "EvalResult", "EvalCase", + "REQUIRED_BENCHMARK_TAGS", + "eval_retrieval", "load_eval_cases", "validate_eval_cases", + "EvalResult", "EvalCase", ] diff --git a/ai_engineering/rag_assistant/rag/eval.py b/ai_engineering/rag_assistant/rag/eval.py index 77e6262f..7851f2a2 100644 --- a/ai_engineering/rag_assistant/rag/eval.py +++ b/ai_engineering/rag_assistant/rag/eval.py @@ -1,84 +1,253 @@ -"""Retrieval-quality evaluation. +"""Deterministic retrieval-quality evaluation. -The metric set is the small-but-honest one most RAG systems publish: +The evaluator reports three complementary ranking metrics: - * **recall@k**: did the retriever surface at least one expected chunk - among the top-k results? - * **MRR (mean reciprocal rank)**: how high up was the first relevant - chunk on average? 1.0 == always first; 0.0 == never retrieved. +* ``recall@1``: fraction of cases with a relevant source at rank 1. +* ``recall@3``: fraction of cases with a relevant source in the top 3. +* ``MRR``: mean reciprocal rank of the first relevant source. -`EvalCase.relevant_sources` is the *set* of acceptable source IDs (any -substring match against `Chunk.source` counts), so it tolerates the -ground-truth file being chunked into multiple pieces. - -For generation-quality eval, see the separate `llm_eval_harness` project -in this repo — keeping retrieval and generation eval in separate places -is the standard pattern. +``EvalCase.relevant_sources`` remains a set of acceptable source markers. A +marker matches when it is contained in ``RetrievedChunk.chunk.source``, so a +single labeled source can still be split into multiple chunks. Stable case IDs +and tags are optional for backward compatibility, but the checked-in benchmark +validates and requires them. """ from __future__ import annotations -from dataclasses import dataclass -from typing import List, Sequence +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import Dict, Iterable, List, Mapping, Sequence from .retriever import RetrievedChunk, Retriever +REQUIRED_BENCHMARK_TAGS = ( + "direct_lookup", + "paraphrase", + "terminology", + "cross_chunk", + "hard_negative", +) + + @dataclass(frozen=True) class EvalCase: question: str relevant_sources: List[str] + id: str = "" + tags: List[str] = field(default_factory=list) @dataclass class EvalResult: n_cases: int + k: int + recall_at_1: float + recall_at_3: float recall_at_k: float mrr: float per_case: List[dict] + by_tag: Dict[str, dict] def __str__(self) -> str: - return ( - f"n={self.n_cases} " - f"recall@k={self.recall_at_k:.3f} " - f"MRR={self.mrr:.3f}" + parts = [ + f"n={self.n_cases}", + f"recall@1={self.recall_at_1:.3f}", + f"recall@3={self.recall_at_3:.3f}", + ] + if self.k not in (1, 3): + parts.append(f"recall@{self.k}={self.recall_at_k:.3f}") + parts.append(f"MRR={self.mrr:.3f}") + return " ".join(parts) + + +def load_eval_cases(path: str | Path) -> List[EvalCase]: + """Load JSON cases while preserving the legacy two-field schema.""" + with open(path, "r", encoding="utf-8") as handle: + raw_cases = json.load(handle) + + if not isinstance(raw_cases, list): + raise ValueError("evaluation cases must be a JSON list") + + cases: List[EvalCase] = [] + for index, raw in enumerate(raw_cases, start=1): + if not isinstance(raw, Mapping): + raise ValueError("each evaluation case must be an object") + + question = raw.get("question", "") + case_id = raw.get("id", "") + sources = raw.get("relevant_sources", []) + tags = raw.get("tags", []) + if not isinstance(question, str) or not isinstance(case_id, str): + raise ValueError(f"case #{index} has non-string id or question") + if not isinstance(sources, list) or any( + not isinstance(source, str) for source in sources + ): + raise ValueError(f"case #{index} relevant_sources must be a string list") + if not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags): + raise ValueError(f"case #{index} tags must be a string list") + + cases.append( + EvalCase( + question=question, + relevant_sources=list(sources), + id=case_id, + tags=list(tags), + ) + ) + return cases + + +def validate_eval_cases( + cases: Sequence[EvalCase], + available_sources: Iterable[str] | None = None, + required_tags: Sequence[str] = REQUIRED_BENCHMARK_TAGS, +) -> None: + """Raise ``ValueError`` when a benchmark dataset is malformed.""" + source_set = set(available_sources) if available_sources is not None else None + seen_ids: set[str] = set() + seen_questions: set[str] = set() + represented_tags: set[str] = set() + + for index, case in enumerate(cases, start=1): + label = case.id or f"case #{index}" + if not case.id.strip(): + raise ValueError(f"{label} has an empty id") + if case.id in seen_ids: + raise ValueError(f"duplicate case id: {case.id}") + seen_ids.add(case.id) + + question = case.question.strip() + if not question: + raise ValueError(f"{label} has an empty question") + if question in seen_questions: + raise ValueError(f"duplicate question: {question}") + seen_questions.add(question) + + sources = [source.strip() for source in case.relevant_sources] + if not sources or any(not source for source in sources): + raise ValueError(f"{label} has empty relevant-source labels") + if len(sources) != len(set(sources)): + raise ValueError(f"{label} has duplicate relevant-source labels") + if source_set is not None: + missing = sorted(set(sources) - source_set) + if missing: + raise ValueError( + f"{label} references missing sources: {', '.join(missing)}" + ) + + tags = [tag.strip() for tag in case.tags] + if not tags or any(not tag for tag in tags): + raise ValueError(f"{label} has empty tags") + if len(tags) != len(set(tags)): + raise ValueError(f"{label} has duplicate tags") + represented_tags.update(tags) + + missing_tags = sorted(set(required_tags) - represented_tags) + if missing_tags: + raise ValueError( + "benchmark is missing required tags: " + ", ".join(missing_tags) ) -def _hits(retrieved: Sequence[RetrievedChunk], relevant: Sequence[str]) -> List[int]: - """Return 1-based ranks of retrieved chunks whose source contains any - of the relevant source markers.""" - out = [] - for rank, r in enumerate(retrieved, start=1): - if any(marker in r.chunk.source for marker in relevant): - out.append(rank) - return out - - -def eval_retrieval(retriever: Retriever, cases: Sequence[EvalCase], k: int = 5) -> EvalResult: +def _hits( + retrieved: Sequence[RetrievedChunk], relevant: Sequence[str] +) -> List[int]: + """Return 1-based ranks whose source contains an accepted marker.""" + return [ + rank + for rank, result in enumerate(retrieved, start=1) + if any(marker in result.chunk.source for marker in relevant) + ] + + +def _aggregate(rows: Sequence[dict], k: int) -> dict: + if not rows: + return { + "n_cases": 0, + "recall_at_1": 0.0, + "recall_at_3": 0.0, + "recall_at_k": 0.0, + "mrr": 0.0, + } + + count = len(rows) + return { + "n_cases": count, + "recall_at_1": sum(row["recall_at_1"] for row in rows) / count, + "recall_at_3": sum(row["recall_at_3"] for row in rows) / count, + "recall_at_k": sum(row["recall_at_k"] for row in rows) / count, + "mrr": sum(row["reciprocal_rank"] for row in rows) / count, + } + + +def eval_retrieval( + retriever: Retriever, + cases: Sequence[EvalCase], + k: int = 5, +) -> EvalResult: + if k < 1: + raise ValueError("k must be >= 1") if not cases: - return EvalResult(n_cases=0, recall_at_k=0.0, mrr=0.0, per_case=[]) + return EvalResult( + n_cases=0, + k=k, + recall_at_1=0.0, + recall_at_3=0.0, + recall_at_k=0.0, + mrr=0.0, + per_case=[], + by_tag={}, + ) + + retrieval_depth = max(3, k) + per_case: List[dict] = [] - rec_sum = 0.0 - rr_sum = 0.0 - per_case = [] for case in cases: - retrieved = retriever.retrieve(case.question, k=k) + retrieved = retriever.retrieve(case.question, k=retrieval_depth) hits = _hits(retrieved, case.relevant_sources) - recall = 1.0 if hits else 0.0 - rr = 1.0 / hits[0] if hits else 0.0 - rec_sum += recall - rr_sum += rr - per_case.append({ - "question": case.question, - "recall": recall, - "reciprocal_rank": rr, - "top_source": retrieved[0].chunk.source if retrieved else None, - }) + first_rank = hits[0] if hits else None + recall_at_1 = 1.0 if first_rank is not None and first_rank <= 1 else 0.0 + recall_at_3 = 1.0 if first_rank is not None and first_rank <= 3 else 0.0 + recall_at_k = 1.0 if first_rank is not None and first_rank <= k else 0.0 + reciprocal_rank = 1.0 / first_rank if first_rank is not None else 0.0 + + per_case.append( + { + "id": case.id, + "question": case.question, + "tags": list(case.tags), + "relevant_sources": list(case.relevant_sources), + "hit_ranks": hits, + "first_relevant_rank": first_rank, + "recall_at_1": recall_at_1, + "recall_at_3": recall_at_3, + "recall_at_k": recall_at_k, + "recall": recall_at_k, + "reciprocal_rank": reciprocal_rank, + "top_source": retrieved[0].chunk.source if retrieved else None, + } + ) + + aggregate = _aggregate(per_case, k) + rows_by_tag: Dict[str, List[dict]] = {} + for row in per_case: + for tag in row["tags"]: + rows_by_tag.setdefault(tag, []).append(row) + by_tag = { + tag: _aggregate(rows, k) + for tag, rows in sorted(rows_by_tag.items()) + } return EvalResult( n_cases=len(cases), - recall_at_k=rec_sum / len(cases), - mrr=rr_sum / len(cases), + k=k, + recall_at_1=aggregate["recall_at_1"], + recall_at_3=aggregate["recall_at_3"], + recall_at_k=aggregate["recall_at_k"], + mrr=aggregate["mrr"], per_case=per_case, + by_tag=by_tag, ) diff --git a/ai_engineering/rag_assistant/sample_docs/chunking_strategies.md b/ai_engineering/rag_assistant/sample_docs/chunking_strategies.md new file mode 100644 index 00000000..3e9593f4 --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/chunking_strategies.md @@ -0,0 +1,7 @@ +# Chunking Strategies + +Chunking determines the text units that retrieval can return. Paragraph-first splitting preserves local structure, while sentence fallback prevents a single long paragraph from becoming an oversized chunk. Character-based limits are model-agnostic; token-aware limits align more directly with model context windows. + +Chunks that are too small lose surrounding definitions and qualifiers. Chunks that are too large mix unrelated topics and dilute the embedding signal. Overlap carries a boundary region into the next chunk so a statement split near an edge can still be retrieved with its nearby context. + +Cross-chunk questions expose a common failure mode. One paragraph may define a concept while the next paragraph states an exception or operating condition. Increasing overlap can help, but excessive overlap creates near-duplicate candidates and wastes index space. Chunk size and overlap should therefore be versioned and compared on the same labeled benchmark. diff --git a/ai_engineering/rag_assistant/sample_docs/embeddings_explained.md b/ai_engineering/rag_assistant/sample_docs/embeddings_explained.md index 1879d070..0bdd4c33 100644 --- a/ai_engineering/rag_assistant/sample_docs/embeddings_explained.md +++ b/ai_engineering/rag_assistant/sample_docs/embeddings_explained.md @@ -1,19 +1,7 @@ # Embeddings, Briefly -An embedding is a vector representation of a piece of text such that -semantically similar texts produce vectors that are close together -under a chosen distance metric, usually cosine similarity. Modern -sentence embedding models like `sentence-transformers/all-MiniLM-L6-v2` -produce 384-dimensional vectors that work well for retrieval out of the -box. +An embedding is a vector representation of text. Semantically related passages should produce nearby vectors even when their wording differs. Sentence-transformers/all-MiniLM-L6-v2 is a common English retrieval baseline and produces 384-dimensional vectors. -When using cosine similarity it is convenient to L2-normalize the -vectors at encoding time so that the inner product equals the cosine. -This lets you use a FAISS `IndexFlatIP` (inner product) index directly -without computing norms at query time. +Cosine similarity compares vector direction. L2-normalizing vectors at encoding time makes inner product equal cosine similarity, which allows a normalized corpus to use FAISS IndexFlatIP without recomputing vector norms for every query. -Embedding model choice matters most for retrieval quality. A 384-d -MiniLM model is a strong default. For multilingual corpora, the -`paraphrase-multilingual-MiniLM-L12-v2` model is the natural choice. -For code search, `text-embedding-3-small` from OpenAI handles syntax -better than general-purpose sentence models. +Model choice affects retrieval behavior. paraphrase-multilingual-MiniLM-L12-v2 is intended for multilingual corpora. Code-oriented search may benefit from an embedding model trained on source code and technical syntax. Embedding dimensions, preprocessing, and model version should be recorded with the index because changing any of them can make a saved store incompatible or invalidate a baseline. diff --git a/ai_engineering/rag_assistant/sample_docs/eval_cases.json b/ai_engineering/rag_assistant/sample_docs/eval_cases.json index 326dc488..f2dbbe85 100644 --- a/ai_engineering/rag_assistant/sample_docs/eval_cases.json +++ b/ai_engineering/rag_assistant/sample_docs/eval_cases.json @@ -1,22 +1,445 @@ [ { + "id": "rag-001", "question": "What is retrieval-augmented generation?", - "relevant_sources": ["rag_overview.md"] + "relevant_sources": [ + "rag_overview.md" + ], + "tags": [ + "direct_lookup", + "rag" + ] }, { - "question": "Why L2-normalize embedding vectors?", - "relevant_sources": ["embeddings_explained.md"] + "id": "rag-002", + "question": "What are the four components of a typical RAG pipeline?", + "relevant_sources": [ + "rag_overview.md" + ], + "tags": [ + "direct_lookup", + "rag" + ] }, { - "question": "When should I use approximate nearest neighbor search?", - "relevant_sources": ["vector_stores.md"] + "id": "emb-001", + "question": "Why does L2 normalization make inner product equal cosine similarity?", + "relevant_sources": [ + "embeddings_explained.md", + "vector_stores.md" + ], + "tags": [ + "direct_lookup", + "embeddings" + ] }, { - "question": "What are the four components of a RAG pipeline?", - "relevant_sources": ["rag_overview.md"] + "id": "vec-001", + "question": "When is exact IndexFlatIP search often sufficient?", + "relevant_sources": [ + "vector_stores.md" + ], + "tags": [ + "direct_lookup", + "vector_store" + ] }, { - "question": "Which embedding model works for multilingual corpora?", - "relevant_sources": ["embeddings_explained.md"] + "id": "chk-001", + "question": "Why is overlap added between adjacent chunks?", + "relevant_sources": [ + "chunking_strategies.md" + ], + "tags": [ + "direct_lookup", + "chunking" + ] + }, + { + "id": "eval-001", + "question": "How is mean reciprocal rank calculated?", + "relevant_sources": [ + "retrieval_evaluation.md" + ], + "tags": [ + "direct_lookup", + "evaluation" + ] + }, + { + "id": "rank-001", + "question": "What does candidate-k control in a re-ranking pipeline?", + "relevant_sources": [ + "reranking_methods.md" + ], + "tags": [ + "direct_lookup", + "reranking" + ] + }, + { + "id": "hyb-001", + "question": "What does reciprocal rank fusion combine?", + "relevant_sources": [ + "hybrid_search.md" + ], + "tags": [ + "direct_lookup", + "hybrid_search" + ] + }, + { + "id": "rag-003", + "question": "How can a language model answer from external knowledge instead of only its parameters?", + "relevant_sources": [ + "rag_overview.md", + "grounding_and_citations.md" + ], + "tags": [ + "paraphrase", + "rag" + ] + }, + { + "id": "eval-002", + "question": "Which metric averages the inverse position of the first correct source?", + "relevant_sources": [ + "retrieval_evaluation.md" + ], + "tags": [ + "paraphrase", + "evaluation" + ] + }, + { + "id": "chk-002", + "question": "How can long documents be divided while retaining context near boundaries?", + "relevant_sources": [ + "chunking_strategies.md" + ], + "tags": [ + "paraphrase", + "chunking" + ] + }, + { + "id": "rank-002", + "question": "Which second-stage model jointly reads a query and passage before changing their order?", + "relevant_sources": [ + "reranking_methods.md" + ], + "tags": [ + "paraphrase", + "reranking" + ] + }, + { + "id": "hyb-002", + "question": "Why combine keyword matching with semantic vector retrieval?", + "relevant_sources": [ + "hybrid_search.md" + ], + "tags": [ + "paraphrase", + "hybrid_search" + ] + }, + { + "id": "grd-001", + "question": "What should the system do when retrieval supplies no usable evidence?", + "relevant_sources": [ + "grounding_and_citations.md" + ], + "tags": [ + "paraphrase", + "grounding" + ] + }, + { + "id": "ops-001", + "question": "Why can recently edited documents remain absent from answers?", + "relevant_sources": [ + "operations_and_monitoring.md" + ], + "tags": [ + "paraphrase", + "operations" + ] + }, + { + "id": "meta-001", + "question": "How can search be restricted by customer, language, or access attributes?", + "relevant_sources": [ + "metadata_and_filters.md" + ], + "tags": [ + "paraphrase", + "metadata" + ] + }, + { + "id": "vec-002", + "question": "What is ANN search and which index families implement it?", + "relevant_sources": [ + "vector_stores.md" + ], + "tags": [ + "terminology", + "vector_store" + ] + }, + { + "id": "rank-003", + "question": "How does a cross-encoder differ from a bi-encoder during retrieval?", + "relevant_sources": [ + "reranking_methods.md" + ], + "tags": [ + "terminology", + "reranking" + ] + }, + { + "id": "hyb-003", + "question": "What does the acronym RRF mean in search?", + "relevant_sources": [ + "hybrid_search.md" + ], + "tags": [ + "terminology", + "hybrid_search" + ] + }, + { + "id": "eval-003", + "question": "Is recall@k the same kind of hit-rate metric as recall@3?", + "relevant_sources": [ + "retrieval_evaluation.md" + ], + "tags": [ + "terminology", + "evaluation" + ] + }, + { + "id": "emb-002", + "question": "What is the relationship between L2 normalization, cosine similarity, and inner product?", + "relevant_sources": [ + "embeddings_explained.md", + "vector_stores.md" + ], + "tags": [ + "terminology", + "embeddings" + ] + }, + { + "id": "vec-003", + "question": "What are HNSW, IVF, and PQ used for?", + "relevant_sources": [ + "vector_stores.md" + ], + "tags": [ + "terminology", + "vector_store" + ] + }, + { + "id": "meta-002", + "question": "How do ACLs and namespaces support tenant isolation?", + "relevant_sources": [ + "metadata_and_filters.md" + ], + "tags": [ + "terminology", + "metadata" + ] + }, + { + "id": "grd-002", + "question": "What does source provenance mean for a generated answer?", + "relevant_sources": [ + "grounding_and_citations.md" + ], + "tags": [ + "terminology", + "grounding" + ] + }, + { + "id": "rag-004", + "question": "How can RAG update knowledge without retraining, and which four stages process a question?", + "relevant_sources": [ + "rag_overview.md" + ], + "tags": [ + "cross_chunk", + "rag" + ] + }, + { + "id": "emb-003", + "question": "Which embedding model is suggested for multilingual text, and why must model versions be stored with an index?", + "relevant_sources": [ + "embeddings_explained.md" + ], + "tags": [ + "cross_chunk", + "embeddings" + ] + }, + { + "id": "vec-004", + "question": "Why might exact search work below ten million chunks, while HNSW or IVF becomes attractive later?", + "relevant_sources": [ + "vector_stores.md" + ], + "tags": [ + "cross_chunk", + "vector_store" + ] + }, + { + "id": "chk-003", + "question": "How do chunk-size tradeoffs interact with overlap and near-duplicate candidates?", + "relevant_sources": [ + "chunking_strategies.md" + ], + "tags": [ + "cross_chunk", + "chunking" + ] + }, + { + "id": "eval-004", + "question": "How do recall@1 and MRR differ, and why should results also be sliced by category?", + "relevant_sources": [ + "retrieval_evaluation.md" + ], + "tags": [ + "cross_chunk", + "evaluation" + ] + }, + { + "id": "rank-004", + "question": "Why can a larger candidate pool help a re-ranker, and what relevant passage can it never recover?", + "relevant_sources": [ + "reranking_methods.md" + ], + "tags": [ + "cross_chunk", + "reranking" + ] + }, + { + "id": "hyb-004", + "question": "Why is BM25 useful for rare acronyms, and how does RRF merge its results with dense retrieval?", + "relevant_sources": [ + "hybrid_search.md" + ], + "tags": [ + "cross_chunk", + "hybrid_search" + ] + }, + { + "id": "grd-003", + "question": "Why can a cited claim still be unsupported, and what should happen when context is empty?", + "relevant_sources": [ + "grounding_and_citations.md" + ], + "tags": [ + "cross_chunk", + "grounding" + ] + }, + { + "id": "rank-005", + "question": "Which stage changes the order of already retrieved candidates without adding lexical matches?", + "relevant_sources": [ + "reranking_methods.md" + ], + "tags": [ + "hard_negative", + "reranking" + ] + }, + { + "id": "hyb-005", + "question": "Which method combines BM25 and dense result lists before an optional final re-ranker?", + "relevant_sources": [ + "hybrid_search.md" + ], + "tags": [ + "hard_negative", + "hybrid_search" + ] + }, + { + "id": "vec-005", + "question": "Which component chooses between exact IndexFlatIP and approximate HNSW based on scale and latency?", + "relevant_sources": [ + "vector_stores.md" + ], + "tags": [ + "hard_negative", + "vector_store" + ] + }, + { + "id": "eval-005", + "question": "Recall@3 is unchanged, but relevant sources moved from rank one to rank three. Which metric reveals the regression?", + "relevant_sources": [ + "retrieval_evaluation.md" + ], + "tags": [ + "hard_negative", + "evaluation" + ] + }, + { + "id": "meta-003", + "question": "Which document explains the difference between pre-filtering and removing disallowed results after retrieval?", + "relevant_sources": [ + "metadata_and_filters.md" + ], + "tags": [ + "hard_negative", + "metadata" + ] + }, + { + "id": "grd-004", + "question": "Which evaluation concern asks whether a cited passage actually supports the attached claim?", + "relevant_sources": [ + "grounding_and_citations.md" + ], + "tags": [ + "hard_negative", + "grounding" + ] + }, + { + "id": "vec-006", + "question": "A query contains the rare acronym ANN and asks about HNSW rather than embedding-model selection. Which source is relevant?", + "relevant_sources": [ + "vector_stores.md" + ], + "tags": [ + "hard_negative", + "vector_store" + ] + }, + { + "id": "ops-002", + "question": "Which operational record helps diagnose stale indexes and embedding-version mismatches?", + "relevant_sources": [ + "operations_and_monitoring.md" + ], + "tags": [ + "hard_negative", + "operations" + ] } ] diff --git a/ai_engineering/rag_assistant/sample_docs/grounding_and_citations.md b/ai_engineering/rag_assistant/sample_docs/grounding_and_citations.md new file mode 100644 index 00000000..438ae2bc --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/grounding_and_citations.md @@ -0,0 +1,7 @@ +# Grounding and Citations + +Grounding means constraining an answer to retrieved evidence. The prompt should identify each passage with a source label and tell the generator to distinguish supported statements from uncertainty. When retrieval returns no usable context, an empty-context guard should refuse a grounded answer instead of encouraging invention. + +Citation presence is not the same as citation correctness. A response can attach a source label to a claim that the cited passage does not support. Evaluation should therefore separate retrieval quality, citation attribution, and answer faithfulness. + +Source provenance helps a reader inspect the evidence and helps an operator trace failures. Good provenance includes the document source and, when useful, chunk identifiers or offsets. It does not compensate for a stale corpus, an access-control error, or a relevant passage that never entered the retrieved context. diff --git a/ai_engineering/rag_assistant/sample_docs/hash_baseline_v1.json b/ai_engineering/rag_assistant/sample_docs/hash_baseline_v1.json new file mode 100644 index 00000000..48155416 --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/hash_baseline_v1.json @@ -0,0 +1,79 @@ +{ + "benchmark_version": "rag-retrieval-v1", + "embedder": { + "class": "HashEmbedder", + "dim": 512 + }, + "chunker": { + "chunk_size": 400, + "chunk_overlap": 80 + }, + "corpus": { + "documents": 10, + "chunks": 31, + "sources": [ + "chunking_strategies.md", + "embeddings_explained.md", + "grounding_and_citations.md", + "hybrid_search.md", + "metadata_and_filters.md", + "operations_and_monitoring.md", + "rag_overview.md", + "reranking_methods.md", + "retrieval_evaluation.md", + "vector_stores.md" + ] + }, + "cases": 40, + "observed": { + "recall_at_1": 0.5, + "recall_at_3": 0.775, + "mrr": 0.629167, + "required_tag_slices": { + "direct_lookup": { + "n_cases": 8, + "recall_at_1": 0.25, + "recall_at_3": 0.75, + "mrr": 0.479167 + }, + "paraphrase": { + "n_cases": 8, + "recall_at_1": 0.625, + "recall_at_3": 0.75, + "mrr": 0.6875 + }, + "terminology": { + "n_cases": 8, + "recall_at_1": 0.5, + "recall_at_3": 0.875, + "mrr": 0.6875 + }, + "cross_chunk": { + "n_cases": 8, + "recall_at_1": 0.75, + "recall_at_3": 0.75, + "mrr": 0.75 + }, + "hard_negative": { + "n_cases": 8, + "recall_at_1": 0.375, + "recall_at_3": 0.75, + "mrr": 0.541667 + } + } + }, + "regression_thresholds": { + "min_recall_at_1": 0.45, + "min_recall_at_3": 0.725, + "min_mrr": 0.579167, + "min_required_tag_recall_at_3": { + "direct_lookup": 0.625, + "paraphrase": 0.625, + "terminology": 0.75, + "cross_chunk": 0.625, + "hard_negative": 0.625 + } + }, + "comparison_rule": "Reject a retrieval change when aggregate recall@1, recall@3, or MRR falls below its minimum, or when any required category loses more than one of its eight top-3 hits. Improvements must be reported on the same corpus, labels, chunk settings, and embedder.", + "claim_boundary": "This deterministic HashEmbedder result is a project regression baseline, not a leaderboard or production-quality retrieval claim." +} diff --git a/ai_engineering/rag_assistant/sample_docs/hybrid_search.md b/ai_engineering/rag_assistant/sample_docs/hybrid_search.md new file mode 100644 index 00000000..eabc7ad7 --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/hybrid_search.md @@ -0,0 +1,7 @@ +# Hybrid Search + +Hybrid search combines dense semantic retrieval with lexical retrieval such as BM25. Dense vectors handle paraphrases and conceptual similarity. BM25 is often stronger for exact identifiers, rare names, acronyms, error codes, and other terms whose spelling matters. + +Reciprocal rank fusion (RRF) merges ranked lists without requiring their raw scores to share a scale. Each document receives a contribution based on its rank in each list, and the fused score determines the combined ordering. Weighted score normalization is another option, but it requires more calibration. + +Hybrid retrieval can add candidates that dense search missed. A later re-ranker may then reorder the fused pool. These are separate stages: fusion expands or combines retrieval evidence, while re-ranking performs a more expensive second-stage comparison over an existing pool. diff --git a/ai_engineering/rag_assistant/sample_docs/metadata_and_filters.md b/ai_engineering/rag_assistant/sample_docs/metadata_and_filters.md new file mode 100644 index 00000000..d9ccce7c --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/metadata_and_filters.md @@ -0,0 +1,7 @@ +# Metadata, Filters, and Isolation + +Metadata can describe tenant, document type, language, publication date, access level, or product area. Pre-filtering restricts the searchable set before vector ranking. Post-filtering retrieves broadly and removes disallowed results afterward, which can leave too few candidates if the initial top-k is small. + +Tenant isolation is an authorization boundary, not merely a relevance preference. A user must never retrieve another tenant's private chunks even when those chunks are semantically similar. Namespaces, access-control lists (ACLs), and mandatory metadata predicates can enforce that boundary. + +Filters can improve precision by removing ineligible sources, but an incorrect or overly narrow predicate can destroy recall. Filter behavior should be tested with positive and negative cases, and the active filter expression should be recorded with retrieval traces. diff --git a/ai_engineering/rag_assistant/sample_docs/operations_and_monitoring.md b/ai_engineering/rag_assistant/sample_docs/operations_and_monitoring.md new file mode 100644 index 00000000..52f2d8a9 --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/operations_and_monitoring.md @@ -0,0 +1,7 @@ +# RAG Operations and Monitoring + +A deployed retrieval system should record corpus version, embedding model, chunk settings, and index build time. Index freshness matters because updated documents do not affect answers until they are ingested. Changing an embedding model without rebuilding the stored vectors creates a dimension or representation mismatch. + +Operational monitoring includes query latency, candidate counts, zero-hit rates, source distribution, and retrieval-quality checks on a fixed benchmark. Recall@3 can remain stable while MRR falls, indicating that relevant sources still appear but are ranked lower. Aggregate production clicks are not a substitute for labeled regression cases. + +Reproducible incident analysis requires the exact query, configuration, index version, and retrieved source order. Access-control failures and stale data should be tracked separately from semantic retrieval failures because they require different remedies. diff --git a/ai_engineering/rag_assistant/sample_docs/rag_overview.md b/ai_engineering/rag_assistant/sample_docs/rag_overview.md index 3f473d9a..54661c73 100644 --- a/ai_engineering/rag_assistant/sample_docs/rag_overview.md +++ b/ai_engineering/rag_assistant/sample_docs/rag_overview.md @@ -1,18 +1,7 @@ # Retrieval-Augmented Generation -Retrieval-Augmented Generation (RAG) is a technique that improves the -quality of large language model output by grounding it in retrieved -documents from a knowledge base. Instead of relying only on the model's -parametric memory, the system first searches an index of documents, -selects the most relevant passages, and inserts them into the prompt -before generating an answer. +Retrieval-Augmented Generation (RAG) grounds a language model in external knowledge. The system searches an indexed corpus, selects relevant passages, inserts those passages into the prompt, and then asks the generator to answer from that context. Retrieval changes the evidence available to the model; generation turns that evidence into a response. -The core advantages of RAG are that it can cite sources, that it can be -updated by re-indexing rather than re-training, and that it dramatically -reduces hallucination on factual questions. +A typical RAG pipeline has four components. A chunker divides documents into retrieval units, an embedder maps each chunk and query to vectors, a vector store performs nearest-neighbor search, and a generator writes the final answer. Optional re-ranking can reorder retrieved candidates, but it does not replace the core four-stage path. -A typical RAG pipeline has four components: a document chunker that -splits long documents into overlapping windows, an embedder that turns -each chunk into a dense vector, a vector store that supports fast -nearest-neighbor search, and a generator that writes the final answer -conditioned on the retrieved context. +RAG can cite sources, reduce unsupported factual claims, and incorporate updated information by re-indexing documents rather than retraining the model. Those advantages depend on retrieval quality and index freshness. A citation shows where context came from, while separate evaluation is still needed to determine whether the answer is actually supported by that context. diff --git a/ai_engineering/rag_assistant/sample_docs/reranking_methods.md b/ai_engineering/rag_assistant/sample_docs/reranking_methods.md new file mode 100644 index 00000000..ac8f8930 --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/reranking_methods.md @@ -0,0 +1,7 @@ +# Re-ranking Methods + +A re-ranker receives a candidate pool from an initial retriever and assigns new relevance scores. A cross-encoder reads the query and candidate passage together, which is slower than a bi-encoder but can model detailed interactions between their words. + +Re-ranking changes the order of candidates already present. It cannot recover a relevant passage that the embedding retriever omitted from the candidate pool. Candidate-k therefore controls a quality-latency tradeoff: a larger pool gives the re-ranker more opportunities, while a smaller pool reduces scoring cost. + +The final top-k should preserve score provenance. Recording both the original retrieval score and the re-ranker score makes debugging possible. Re-ranking is different from hybrid search: hybrid retrieval can introduce candidates from a lexical system, whereas a re-ranker only reorders the candidates it receives. diff --git a/ai_engineering/rag_assistant/sample_docs/retrieval_evaluation.md b/ai_engineering/rag_assistant/sample_docs/retrieval_evaluation.md new file mode 100644 index 00000000..21b02d28 --- /dev/null +++ b/ai_engineering/rag_assistant/sample_docs/retrieval_evaluation.md @@ -0,0 +1,7 @@ +# Retrieval Evaluation + +Retrieval evaluation compares ranked results with labeled relevant sources. Recall@1 is the fraction of questions with an acceptable source in the first position. Recall@3 asks whether an acceptable source appears anywhere in the first three results. Mean reciprocal rank (MRR) averages 1 divided by the rank of the first relevant result, assigning zero when no relevant source is retrieved. + +A case may name more than one acceptable source when several documents contain sufficient evidence. Per-case output is needed to diagnose individual failures. Category slices reveal whether an aggregate score hides regressions on paraphrases, terminology variants, cross-chunk questions, or hard negatives. + +A deterministic baseline is a comparison instrument, not a leaderboard claim. Future changes should run against the same versioned corpus, case labels, chunk settings, and embedder. A practical regression rule allows small numerical variation while rejecting material losses in recall@3, MRR, or a required category slice. diff --git a/ai_engineering/rag_assistant/sample_docs/vector_stores.md b/ai_engineering/rag_assistant/sample_docs/vector_stores.md index 7a77b31d..81685236 100644 --- a/ai_engineering/rag_assistant/sample_docs/vector_stores.md +++ b/ai_engineering/rag_assistant/sample_docs/vector_stores.md @@ -1,16 +1,7 @@ # Vector Stores -A vector store indexes embeddings and answers nearest-neighbor queries. -FAISS is the most widely used in-process library; it offers exact search -(`IndexFlatIP`, `IndexFlatL2`) and several approximate indexes (HNSW, -IVF, PQ) with tunable speed-vs-recall tradeoffs. +A vector store indexes embeddings and answers nearest-neighbor queries. With L2-normalized vectors, an inner-product index ranks by cosine similarity. Exact indexes such as IndexFlatIP compare the query with every stored vector and avoid approximation error. -For most projects under ten million chunks, an exact `IndexFlatIP` -index is fast enough and removes a whole class of tuning concerns. The -approximate indexes become attractive when memory or latency becomes -the constraint. +For many corpora below roughly ten million chunks, exact search can be fast enough and easier to reason about. Approximate nearest-neighbor (ANN) structures such as HNSW, IVF, and product quantization (PQ) trade some recall for lower latency or memory use. Their parameters should be tuned against a retrieval benchmark rather than selected only from throughput measurements. -Hosted alternatives include Pinecone, Weaviate, Qdrant, and Chroma. -The right choice depends on whether you need multi-tenant isolation -(Pinecone), hybrid filtering (Weaviate), or a single-node experience -that just works (Chroma). +Hosted systems such as Pinecone, Weaviate, Qdrant, and Chroma add operational features. Metadata filters, tenant isolation, replication, and hybrid-search support are separate concerns from the vector similarity algorithm itself. A filter that excludes the relevant document will reduce recall no matter how strong the embedding model is. diff --git a/ai_engineering/rag_assistant/smoke/run_reranker_eval.py b/ai_engineering/rag_assistant/smoke/run_reranker_eval.py index 394d5471..6708f230 100644 --- a/ai_engineering/rag_assistant/smoke/run_reranker_eval.py +++ b/ai_engineering/rag_assistant/smoke/run_reranker_eval.py @@ -1,8 +1,7 @@ -"""Compare embedding-only retrieval against optional cross-encoder re-ranking. +"""Compare MiniLM embedding retrieval with optional cross-encoder re-ranking. -This script requires sentence-transformers model weights. It does not require -an OpenAI API key. The checked-in five-case suite is saturated, so the report -is a provisional measurement until the expanded benchmark in issue #18 lands. +This command uses the versioned 40-case benchmark. It requires local model +weights but no OpenAI API key. Outputs: reports/reranker_eval.md @@ -13,105 +12,138 @@ import json import os from datetime import datetime, timezone +from pathlib import Path from rag.chunker import Chunker, Document -from rag.embedder import make_default_embedder -from rag.eval import EvalCase, eval_retrieval +from rag.embedder import SentenceTransformerEmbedder +from rag.eval import ( + REQUIRED_BENCHMARK_TAGS, + eval_retrieval, + load_eval_cases, + validate_eval_cases, +) from rag.reranker import CrossEncoderReranker, DEFAULT_RERANKER_MODEL from rag.retriever import Retriever from rag.vector_store import VectorStore -SAMPLE_DIR = os.path.join(os.path.dirname(__file__), "..", "sample_docs") -REPORT_PATH = "reports/reranker_eval.md" -K = int(os.environ.get("RERANKER_EVAL_K", "3")) +PROJECT_ROOT = Path(__file__).resolve().parents[1] +SAMPLE_DIR = PROJECT_ROOT / "sample_docs" +BASELINE_PATH = SAMPLE_DIR / "hash_baseline_v1.json" +REPORT_PATH = Path(os.environ.get("RERANKER_EVAL_REPORT", "reports/reranker_eval.md")) CANDIDATE_K = int(os.environ.get("RERANKER_CANDIDATES", "20")) -MODEL_NAME = os.environ.get("RERANKER_MODEL", DEFAULT_RERANKER_MODEL) +EMBEDDING_MODEL = os.environ.get( + "RAG_EVAL_MODEL", + "sentence-transformers/all-MiniLM-L6-v2", +) +RERANKER_MODEL = os.environ.get("RERANKER_MODEL", DEFAULT_RERANKER_MODEL) def _load_docs() -> list[Document]: - docs = [] - for name in sorted(os.listdir(SAMPLE_DIR)): - if not name.endswith(".md"): - continue - with open(os.path.join(SAMPLE_DIR, name), "r", encoding="utf-8") as fh: - docs.append(Document(source=name, text=fh.read())) - return docs + return [ + Document(source=path.name, text=path.read_text(encoding="utf-8")) + for path in sorted(SAMPLE_DIR.glob("*.md")) + ] -def main() -> None: - os.makedirs("reports", exist_ok=True) - docs = _load_docs() - embedder = make_default_embedder() +def _rank(row: dict) -> str: + value = row["first_relevant_rank"] + return str(value) if value is not None else "miss" + - chunker = Chunker(chunk_size=400, chunk_overlap=80) - chunks = chunker.chunk_corpus(docs) +def main() -> None: + baseline_spec = json.loads(BASELINE_PATH.read_text(encoding="utf-8")) + documents = _load_docs() + cases = load_eval_cases(SAMPLE_DIR / "eval_cases.json") + validate_eval_cases(cases, [document.source for document in documents]) + + embedder = SentenceTransformerEmbedder(EMBEDDING_MODEL) + chunker = Chunker(**baseline_spec["chunker"]) + chunks = chunker.chunk_corpus(documents) store = VectorStore(dim=embedder.dim) store.add(chunks, embedder.embed([chunk.text for chunk in chunks])) - with open( - os.path.join(SAMPLE_DIR, "eval_cases.json"), - "r", - encoding="utf-8", - ) as fh: - cases = [EvalCase(**case) for case in json.load(fh)] - - baseline = eval_retrieval( + embedding_only = eval_retrieval( Retriever(embedder, store), cases, - k=K, + k=3, ) reranked = eval_retrieval( Retriever( embedder, store, - reranker=CrossEncoderReranker(model_name=MODEL_NAME), + reranker=CrossEncoderReranker(model_name=RERANKER_MODEL), candidate_k=CANDIDATE_K, ), cases, - k=K, + k=3, ) - recall_delta = reranked.recall_at_k - baseline.recall_at_k - mrr_delta = reranked.mrr - baseline.mrr - rows = "\n".join( + aggregate_rows = ( + f"| Embedding only | {embedding_only.recall_at_1:.3f} | " + f"{embedding_only.recall_at_3:.3f} | {embedding_only.mrr:.3f} |\n" + f"| Cross-encoder re-ranked | {reranked.recall_at_1:.3f} | " + f"{reranked.recall_at_3:.3f} | {reranked.mrr:.3f} |\n" + f"| Delta | {reranked.recall_at_1 - embedding_only.recall_at_1:+.3f} | " + f"{reranked.recall_at_3 - embedding_only.recall_at_3:+.3f} | " + f"{reranked.mrr - embedding_only.mrr:+.3f} |" + ) + category_rows = "\n".join( ( - f"| {base['question']} | {base['reciprocal_rank']:.3f} | " + f"| `{tag}` | {embedding_only.by_tag[tag]['recall_at_3']:.3f} | " + f"{reranked.by_tag[tag]['recall_at_3']:.3f} | " + f"{reranked.by_tag[tag]['recall_at_3'] - embedding_only.by_tag[tag]['recall_at_3']:+.3f} | " + f"{embedding_only.by_tag[tag]['mrr']:.3f} | " + f"{reranked.by_tag[tag]['mrr']:.3f} |" + ) + for tag in REQUIRED_BENCHMARK_TAGS + ) + case_rows = "\n".join( + ( + f"| `{base['id']}` | {', '.join(base['tags'])} | {_rank(base)} | " + f"{_rank(rerank)} | {base['reciprocal_rank']:.3f} | " f"{rerank['reciprocal_rank']:.3f} | " f"{rerank['reciprocal_rank'] - base['reciprocal_rank']:+.3f} |" ) - for base, rerank in zip(baseline.per_case, reranked.per_case) + for base, rerank in zip(embedding_only.per_case, reranked.per_case) ) - with open(REPORT_PATH, "w", encoding="utf-8") as fh: - fh.write( - "# RAG Assistant - Re-ranker Evaluation\n\n" - f"_Generated: {datetime.now(timezone.utc).replace(tzinfo=None).isoformat(timespec='seconds')}Z_\n\n" - f"- **Embedder**: `{type(embedder).__name__}`\n" - f"- **Re-ranker**: `{MODEL_NAME}`\n" - f"- **Cases**: {len(cases)}\n" - f"- **k**: {K}\n" - f"- **Candidate pool**: {CANDIDATE_K}\n\n" - "## Aggregate comparison\n\n" - "| Configuration | recall@k | MRR |\n" - "|---|---:|---:|\n" - f"| Embedding only | {baseline.recall_at_k:.3f} | {baseline.mrr:.3f} |\n" - f"| Cross-encoder re-ranked | {reranked.recall_at_k:.3f} | {reranked.mrr:.3f} |\n" - f"| Delta | {recall_delta:+.3f} | {mrr_delta:+.3f} |\n\n" - "## Per-case reciprocal-rank comparison\n\n" - "| Question | Baseline RR | Re-ranked RR | Delta |\n" - "|---|---:|---:|---:|\n" - f"{rows}\n\n" - "## Interpretation boundary\n\n" - "This is a five-case smoke evaluation over three small documents. " - "It records the observed delta but is not large enough to establish " - "a reliable quality lift. Re-run against the expanded benchmark from " - "issue #18 before making a comparative performance claim.\n" - ) + REPORT_PATH.parent.mkdir(parents=True, exist_ok=True) + REPORT_PATH.write_text( + "# RAG Assistant - Re-ranker Evaluation\n\n" + f"_Generated: {datetime.now(timezone.utc).replace(tzinfo=None).isoformat(timespec='seconds')}Z_\n\n" + f"- **Benchmark**: `{baseline_spec['benchmark_version']}`\n" + f"- **Embedding model**: `{EMBEDDING_MODEL}`\n" + f"- **Re-ranker**: `{RERANKER_MODEL}`\n" + f"- **Documents**: {len(documents)}\n" + f"- **Chunks**: {len(chunks)}\n" + f"- **Cases**: {len(cases)}\n" + f"- **Candidate pool**: {CANDIDATE_K}\n\n" + "## Aggregate comparison\n\n" + "| Configuration | recall@1 | recall@3 | MRR |\n" + "|---|---:|---:|---:|\n" + f"{aggregate_rows}\n\n" + "## Required category slices\n\n" + "| Category | Base recall@3 | Re-ranked recall@3 | Delta | Base MRR | Re-ranked MRR |\n" + "|---|---:|---:|---:|---:|---:|\n" + f"{category_rows}\n\n" + "## Per-case comparison\n\n" + "| ID | Tags | Base rank | Re-ranked rank | Base RR | Re-ranked RR | Delta |\n" + "|---|---|---:|---:|---:|---:|---:|\n" + f"{case_rows}\n\n" + "## Interpretation boundary\n\n" + "This comparison is reproducible on the checked-in project benchmark. " + "It is more discriminative than the earlier five-case smoke suite, but it " + "remains a small synthetic corpus and is not a leaderboard or production " + "quality claim.\n", + encoding="utf-8", + ) print( f"Wrote {REPORT_PATH}: " - f"recall delta={recall_delta:+.3f}, MRR delta={mrr_delta:+.3f}" + f"recall@1 delta={reranked.recall_at_1 - embedding_only.recall_at_1:+.3f}, " + f"recall@3 delta={reranked.recall_at_3 - embedding_only.recall_at_3:+.3f}, " + f"MRR delta={reranked.mrr - embedding_only.mrr:+.3f}" ) diff --git a/ai_engineering/rag_assistant/smoke/run_smoke.py b/ai_engineering/rag_assistant/smoke/run_smoke.py index d2bb5d26..71956bd9 100644 --- a/ai_engineering/rag_assistant/smoke/run_smoke.py +++ b/ai_engineering/rag_assistant/smoke/run_smoke.py @@ -1,13 +1,11 @@ -"""End-to-end smoke run: ingest sample_docs, retrieve, evaluate. +"""Run the versioned retrieval benchmark and write a Markdown report. -No OpenAI API key required — generation is skipped; retrieval-quality -eval (recall@k, MRR) is what gets reported. With sentence-transformers -installed, this uses the real MiniLM embedder; otherwise it falls back -to HashEmbedder and the numbers will be low. +The default is the deterministic, network-free HashEmbedder baseline. Set +``RAG_EVAL_EMBEDDER=minilm`` for a local all-MiniLM-L6-v2 run. Outputs: - index/ persisted vector store - reports/smoke_eval.md recall@k + MRR + per-case results + index/ persisted vector store + reports/smoke_eval.md aggregate, category, and per-case results """ from __future__ import annotations @@ -15,60 +13,140 @@ import json import os from datetime import datetime, timezone +from pathlib import Path from rag.chunker import Chunker, Document -from rag.embedder import make_default_embedder -from rag.eval import EvalCase, eval_retrieval +from rag.embedder import HashEmbedder, SentenceTransformerEmbedder +from rag.eval import ( + REQUIRED_BENCHMARK_TAGS, + eval_retrieval, + load_eval_cases, + validate_eval_cases, +) from rag.retriever import Retriever from rag.vector_store import VectorStore -SAMPLE_DIR = os.path.join(os.path.dirname(__file__), "..", "sample_docs") -INDEX_DIR = "index" -REPORT_PATH = "reports/smoke_eval.md" + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +SAMPLE_DIR = PROJECT_ROOT / "sample_docs" +BASELINE_PATH = SAMPLE_DIR / "hash_baseline_v1.json" +INDEX_DIR = Path(os.environ.get("RAG_EVAL_INDEX", "index")) +REPORT_PATH = Path(os.environ.get("RAG_EVAL_REPORT", "reports/smoke_eval.md")) +EMBEDDER_MODE = os.environ.get("RAG_EVAL_EMBEDDER", "hash").lower() +MINILM_MODEL = os.environ.get( + "RAG_EVAL_MODEL", + "sentence-transformers/all-MiniLM-L6-v2", +) def _load_docs() -> list[Document]: - docs = [] - for name in sorted(os.listdir(SAMPLE_DIR)): - if not name.endswith(".md"): - continue - with open(os.path.join(SAMPLE_DIR, name), "r", encoding="utf-8") as fh: - docs.append(Document(source=name, text=fh.read())) - return docs + return [ + Document(source=path.name, text=path.read_text(encoding="utf-8")) + for path in sorted(SAMPLE_DIR.glob("*.md")) + ] + + +def _load_baseline() -> dict: + return json.loads(BASELINE_PATH.read_text(encoding="utf-8")) + + +def _make_embedder(baseline: dict): + if EMBEDDER_MODE == "hash": + return HashEmbedder(dim=baseline["embedder"]["dim"]), "deterministic" + if EMBEDDER_MODE == "minilm": + return SentenceTransformerEmbedder(MINILM_MODEL), "local model" + raise ValueError("RAG_EVAL_EMBEDDER must be 'hash' or 'minilm'") + + +def _format_rank(value) -> str: + return str(value) if value is not None else "miss" def main() -> None: - os.makedirs("reports", exist_ok=True) - docs = _load_docs() - embedder = make_default_embedder() + baseline = _load_baseline() + documents = _load_docs() + cases = load_eval_cases(SAMPLE_DIR / "eval_cases.json") + validate_eval_cases(cases, [document.source for document in documents]) - chunker = Chunker(chunk_size=400, chunk_overlap=80) - chunks = chunker.chunk_corpus(docs) + embedder, mode_label = _make_embedder(baseline) + chunker = Chunker(**baseline["chunker"]) + chunks = chunker.chunk_corpus(documents) store = VectorStore(dim=embedder.dim) - store.add(chunks, embedder.embed([c.text for c in chunks])) - store.save(INDEX_DIR) - - with open(os.path.join(SAMPLE_DIR, "eval_cases.json"), "r", encoding="utf-8") as fh: - cases = [EvalCase(**c) for c in json.load(fh)] + store.add(chunks, embedder.embed([chunk.text for chunk in chunks])) + store.save(str(INDEX_DIR)) result = eval_retrieval(Retriever(embedder, store), cases, k=3) - rows = "\n".join( - f"| {r['question']} | {r['recall']:.0f} | {r['reciprocal_rank']:.3f} | {r['top_source']} |" - for r in result.per_case + + category_rows = "\n".join( + ( + f"| `{tag}` | {result.by_tag[tag]['n_cases']} | " + f"{result.by_tag[tag]['recall_at_1']:.3f} | " + f"{result.by_tag[tag]['recall_at_3']:.3f} | " + f"{result.by_tag[tag]['mrr']:.3f} |" + ) + for tag in REQUIRED_BENCHMARK_TAGS + ) + case_rows = "\n".join( + ( + f"| `{row['id']}` | {', '.join(row['tags'])} | " + f"{_format_rank(row['first_relevant_rank'])} | " + f"{row['reciprocal_rank']:.3f} | {row['top_source']} |" + ) + for row in result.per_case ) - with open(REPORT_PATH, "w", encoding="utf-8") as fh: - fh.write( - f"# RAG Assistant — Smoke Run\n\n" - f"_Generated: {datetime.now(timezone.utc).replace(tzinfo=None).isoformat(timespec='seconds')}Z_\n\n" - f"- **Embedder**: `{type(embedder).__name__}` (dim {embedder.dim})\n" - f"- **Docs**: {len(docs)} **Chunks**: {len(chunks)}\n" - f"- **k**: 3\n\n" - f"## Headline\n\n- **recall@3**: {result.recall_at_k:.3f}\n" - f"- **MRR**: {result.mrr:.3f}\n\n" - f"## Per-case results\n\n" - f"| Question | Recall | RR | Top source |\n|---|---|---|---|\n{rows}\n" + + baseline_note = "" + if EMBEDDER_MODE == "hash": + thresholds = baseline["regression_thresholds"] + baseline_note = ( + "## Regression gate\n\n" + f"- Minimum recall@1: {thresholds['min_recall_at_1']:.3f}\n" + f"- Minimum recall@3: {thresholds['min_recall_at_3']:.3f}\n" + f"- Minimum MRR: {thresholds['min_mrr']:.3f}\n" + "- Each required category may lose at most one of its eight " + "top-3 hits relative to the checked-in baseline.\n\n" + f"{baseline['comparison_rule']}\n\n" ) - print(f"Wrote {REPORT_PATH}: recall@3={result.recall_at_k:.3f}, MRR={result.mrr:.3f}") + + REPORT_PATH.parent.mkdir(parents=True, exist_ok=True) + REPORT_PATH.write_text( + "# RAG Assistant - Retrieval Benchmark\n\n" + f"_Generated: {datetime.now(timezone.utc).replace(tzinfo=None).isoformat(timespec='seconds')}Z_\n\n" + f"- **Benchmark**: `{baseline['benchmark_version']}`\n" + f"- **Embedder**: `{type(embedder).__name__}` ({mode_label}, dim {embedder.dim})\n" + f"- **Documents**: {len(documents)}\n" + f"- **Chunks**: {len(chunks)}\n" + f"- **Cases**: {len(cases)}\n" + f"- **Chunking**: {chunker.chunk_size} characters, {chunker.chunk_overlap} overlap\n\n" + "## Metric definitions\n\n" + "- **recall@1**: fraction of cases with an accepted source at rank 1.\n" + "- **recall@3**: fraction with an accepted source in the top three.\n" + "- **MRR**: mean reciprocal rank of the first accepted source; misses score zero.\n\n" + "## Headline\n\n" + f"- **recall@1**: {result.recall_at_1:.3f}\n" + f"- **recall@3**: {result.recall_at_3:.3f}\n" + f"- **MRR**: {result.mrr:.3f}\n\n" + "## Required category slices\n\n" + "| Category | Cases | recall@1 | recall@3 | MRR |\n" + "|---|---:|---:|---:|---:|\n" + f"{category_rows}\n\n" + f"{baseline_note}" + "## Per-case results\n\n" + "| ID | Tags | First relevant rank | RR | Top source |\n" + "|---|---|---:|---:|---|\n" + f"{case_rows}\n\n" + "## Claim boundary\n\n" + "This is a versioned project baseline for regression testing. The corpus is " + "small and synthetic, so the scores are not a leaderboard result or a claim " + "about production retrieval quality.\n", + encoding="utf-8", + ) + + print( + f"Wrote {REPORT_PATH}: cases={result.n_cases}, " + f"recall@1={result.recall_at_1:.3f}, " + f"recall@3={result.recall_at_3:.3f}, MRR={result.mrr:.3f}" + ) if __name__ == "__main__": diff --git a/ai_engineering/rag_assistant/tests/test_benchmark.py b/ai_engineering/rag_assistant/tests/test_benchmark.py new file mode 100644 index 00000000..fe60bee9 --- /dev/null +++ b/ai_engineering/rag_assistant/tests/test_benchmark.py @@ -0,0 +1,95 @@ +"""Validation and deterministic regression tests for the checked-in benchmark.""" + +import json +import unittest +from collections import Counter +from pathlib import Path + +from rag.chunker import Chunker, Document +from rag.embedder import HashEmbedder +from rag.eval import ( + REQUIRED_BENCHMARK_TAGS, + eval_retrieval, + load_eval_cases, + validate_eval_cases, +) +from rag.retriever import Retriever +from rag.vector_store import VectorStore + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +SAMPLE_DIR = PROJECT_ROOT / "sample_docs" +BASELINE_PATH = SAMPLE_DIR / "hash_baseline_v1.json" + + +def load_documents(): + return [ + Document(source=path.name, text=path.read_text(encoding="utf-8")) + for path in sorted(SAMPLE_DIR.glob("*.md")) + ] + + +def run_hash_baseline(): + baseline = json.loads(BASELINE_PATH.read_text(encoding="utf-8")) + documents = load_documents() + cases = load_eval_cases(SAMPLE_DIR / "eval_cases.json") + embedder = HashEmbedder(dim=baseline["embedder"]["dim"]) + chunker = Chunker(**baseline["chunker"]) + chunks = chunker.chunk_corpus(documents) + store = VectorStore(dim=embedder.dim) + store.add(chunks, embedder.embed([chunk.text for chunk in chunks])) + return baseline, documents, chunks, cases, eval_retrieval( + Retriever(embedder, store), cases, k=3 + ) + + +class TestBenchmarkDataset(unittest.TestCase): + def test_benchmark_has_40_cases_and_10_documents(self): + cases = load_eval_cases(SAMPLE_DIR / "eval_cases.json") + documents = load_documents() + + self.assertEqual(len(cases), 40) + self.assertEqual(len(documents), 10) + + def test_dataset_passes_schema_and_source_validation(self): + cases = load_eval_cases(SAMPLE_DIR / "eval_cases.json") + documents = load_documents() + + validate_eval_cases(cases, [document.source for document in documents]) + + def test_each_required_category_has_eight_cases(self): + cases = load_eval_cases(SAMPLE_DIR / "eval_cases.json") + counts = Counter(tag for case in cases for tag in case.tags) + + for tag in REQUIRED_BENCHMARK_TAGS: + self.assertEqual(counts[tag], 8, tag) + + def test_dataset_contains_multi_source_labels(self): + cases = load_eval_cases(SAMPLE_DIR / "eval_cases.json") + multi_source = [case for case in cases if len(case.relevant_sources) > 1] + + self.assertGreaterEqual(len(multi_source), 3) + + def test_baseline_metadata_matches_corpus_and_chunking(self): + baseline, documents, chunks, cases, _ = run_hash_baseline() + + self.assertEqual(baseline["benchmark_version"], "rag-retrieval-v1") + self.assertEqual(baseline["corpus"]["documents"], len(documents)) + self.assertEqual(baseline["corpus"]["chunks"], len(chunks)) + self.assertEqual(baseline["corpus"]["sources"], [doc.source for doc in documents]) + self.assertEqual(baseline["cases"], len(cases)) + + def test_hash_baseline_stays_above_regression_thresholds(self): + baseline, _, _, _, result = run_hash_baseline() + thresholds = baseline["regression_thresholds"] + + self.assertGreaterEqual(result.recall_at_1, thresholds["min_recall_at_1"]) + self.assertGreaterEqual(result.recall_at_3, thresholds["min_recall_at_3"]) + self.assertGreaterEqual(result.mrr, thresholds["min_mrr"]) + + for tag, minimum in thresholds["min_required_tag_recall_at_3"].items(): + self.assertGreaterEqual(result.by_tag[tag]["recall_at_3"], minimum, tag) + + +if __name__ == "__main__": + unittest.main() diff --git a/ai_engineering/rag_assistant/tests/test_eval.py b/ai_engineering/rag_assistant/tests/test_eval.py new file mode 100644 index 00000000..92f7bdcd --- /dev/null +++ b/ai_engineering/rag_assistant/tests/test_eval.py @@ -0,0 +1,186 @@ +"""Unit tests for retrieval metrics, labels, loading, and category slices.""" + +import json +import tempfile +import unittest +from pathlib import Path + +from rag.chunker import Chunk +from rag.eval import ( + EvalCase, + REQUIRED_BENCHMARK_TAGS, + eval_retrieval, + load_eval_cases, + validate_eval_cases, +) +from rag.retriever import RetrievedChunk + + +def retrieved(source: str, score: float = 1.0) -> RetrievedChunk: + chunk = Chunk( + chunk_id=f"{source}#0", + doc_id=source, + source=source, + chunk_index=0, + text=source, + ) + return RetrievedChunk(score=score, chunk=chunk) + + +class FakeRetriever: + def __init__(self, results_by_question): + self.results_by_question = results_by_question + self.requested_depths = [] + + def retrieve(self, question: str, k: int = 5): + self.requested_depths.append(k) + return list(self.results_by_question.get(question, []))[:k] + + +class TestRetrievalMetrics(unittest.TestCase): + def test_aggregate_recall_at_1_recall_at_3_recall_at_k_and_mrr(self): + cases = [ + EvalCase("rank one", ["a.md"], id="a", tags=["direct_lookup"]), + EvalCase("rank three", ["b.md"], id="b", tags=["paraphrase"]), + EvalCase("missing", ["c.md"], id="c", tags=["hard_negative"]), + ] + fake = FakeRetriever( + { + "rank one": [retrieved("a.md"), retrieved("x.md"), retrieved("y.md")], + "rank three": [retrieved("x.md"), retrieved("y.md"), retrieved("b.md")], + "missing": [retrieved("x.md"), retrieved("y.md"), retrieved("z.md")], + } + ) + + result = eval_retrieval(fake, cases, k=5) + + self.assertAlmostEqual(result.recall_at_1, 1 / 3) + self.assertAlmostEqual(result.recall_at_3, 2 / 3) + self.assertAlmostEqual(result.recall_at_k, 2 / 3) + self.assertAlmostEqual(result.mrr, (1.0 + 1 / 3) / 3) + self.assertEqual(result.k, 5) + + def test_multi_source_labels_accept_any_labeled_source(self): + case = EvalCase( + "shared answer", + ["primary.md", "alternate.md"], + id="multi", + tags=["direct_lookup"], + ) + fake = FakeRetriever( + {"shared answer": [retrieved("alternate.md#section"), retrieved("noise.md")]} + ) + + result = eval_retrieval(fake, [case], k=3) + + self.assertEqual(result.recall_at_1, 1.0) + self.assertEqual(result.per_case[0]["first_relevant_rank"], 1) + self.assertEqual(result.per_case[0]["hit_ranks"], [1]) + + def test_missing_hit_records_zero_metrics_and_no_rank(self): + case = EvalCase("none", ["missing.md"], id="none", tags=["hard_negative"]) + fake = FakeRetriever({"none": [retrieved("noise.md")]}) + + result = eval_retrieval(fake, [case], k=3) + row = result.per_case[0] + + self.assertEqual(row["recall_at_1"], 0.0) + self.assertEqual(row["recall_at_3"], 0.0) + self.assertEqual(row["reciprocal_rank"], 0.0) + self.assertIsNone(row["first_relevant_rank"]) + self.assertEqual(row["hit_ranks"], []) + + def test_tag_slices_aggregate_only_matching_cases(self): + cases = [ + EvalCase("a", ["a.md"], id="a", tags=["direct_lookup", "rag"]), + EvalCase("b", ["b.md"], id="b", tags=["paraphrase", "rag"]), + EvalCase("c", ["c.md"], id="c", tags=["paraphrase", "vector"]), + ] + fake = FakeRetriever( + { + "a": [retrieved("a.md")], + "b": [retrieved("x.md"), retrieved("b.md")], + "c": [retrieved("x.md")], + } + ) + + result = eval_retrieval(fake, cases, k=3) + + self.assertEqual(result.by_tag["rag"]["n_cases"], 2) + self.assertEqual(result.by_tag["rag"]["recall_at_3"], 1.0) + self.assertEqual(result.by_tag["paraphrase"]["n_cases"], 2) + self.assertEqual(result.by_tag["paraphrase"]["recall_at_3"], 0.5) + self.assertEqual(result.by_tag["vector"]["mrr"], 0.0) + + def test_k_one_still_retrieves_three_for_fixed_recall_at_3(self): + case = EvalCase("q", ["target.md"], id="q", tags=["direct_lookup"]) + fake = FakeRetriever( + {"q": [retrieved("noise.md"), retrieved("target.md"), retrieved("other.md")]} + ) + + result = eval_retrieval(fake, [case], k=1) + + self.assertEqual(fake.requested_depths, [3]) + self.assertEqual(result.recall_at_k, 0.0) + self.assertEqual(result.recall_at_3, 1.0) + + def test_empty_cases_return_zero_metrics(self): + result = eval_retrieval(FakeRetriever({}), [], k=3) + + self.assertEqual(result.n_cases, 0) + self.assertEqual(result.recall_at_1, 0.0) + self.assertEqual(result.recall_at_3, 0.0) + self.assertEqual(result.mrr, 0.0) + self.assertEqual(result.per_case, []) + self.assertEqual(result.by_tag, {}) + + def test_loader_preserves_legacy_and_enriched_cases(self): + payload = [ + {"question": "legacy", "relevant_sources": ["a.md"]}, + { + "id": "new-1", + "question": "enriched", + "relevant_sources": ["b.md"], + "tags": ["direct_lookup"], + }, + ] + with tempfile.TemporaryDirectory() as temp_dir: + path = Path(temp_dir) / "cases.json" + path.write_text(json.dumps(payload), encoding="utf-8") + cases = load_eval_cases(path) + + self.assertEqual(cases[0].question, "legacy") + self.assertEqual(cases[0].id, "") + self.assertEqual(cases[0].tags, []) + self.assertEqual(cases[1].id, "new-1") + self.assertEqual(cases[1].tags, ["direct_lookup"]) + + def test_validation_rejects_duplicate_ids(self): + cases = [ + EvalCase("a", ["a.md"], id="same", tags=list(REQUIRED_BENCHMARK_TAGS)), + EvalCase("b", ["a.md"], id="same", tags=["direct_lookup"]), + ] + with self.assertRaisesRegex(ValueError, "duplicate case id"): + validate_eval_cases(cases, ["a.md"]) + + def test_validation_rejects_missing_sources_and_required_tags(self): + missing_source = [ + EvalCase( + "a", + ["missing.md"], + id="a", + tags=list(REQUIRED_BENCHMARK_TAGS), + ) + ] + with self.assertRaisesRegex(ValueError, "missing sources"): + validate_eval_cases(missing_source, ["a.md"]) + + missing_tags = [ + EvalCase("a", ["a.md"], id="a", tags=["direct_lookup"]) + ] + with self.assertRaisesRegex(ValueError, "missing required tags"): + validate_eval_cases(missing_tags, ["a.md"]) + + +if __name__ == "__main__": + unittest.main()