Skip to content

Expand RAG retrieval benchmark - #23

Merged
lmdixon23 merged 1 commit into
mainfrom
issue-18-rag-benchmark-20260727
Jul 27, 2026
Merged

Expand RAG retrieval benchmark#23
lmdixon23 merged 1 commit into
mainfrom
issue-18-rag-benchmark-20260727

Conversation

@lmdixon23

Copy link
Copy Markdown
Owner

Summary

Expands the RAG Assistant retrieval evaluation from a five-case smoke check into a deterministic 40-case benchmark over 10 documents.

Changes

  • adds stable case IDs, tags, multi-source labels, and dataset validation;
  • covers direct lookup, paraphrase, terminology variation, cross-chunk wording, and hard-negative categories;
  • reports recall@1, recall@3, MRR, per-case results, and per-tag slices;
  • adds a versioned deterministic HashEmbedder baseline and regression rule;
  • updates the CLI, smoke and re-ranker evaluation runners, README, and repository results documentation;
  • adds benchmark and metric regression tests without changing retrieval algorithms.

Deterministic baseline

  • cases: 40
  • documents: 10
  • chunks: 31
  • embedder: HashEmbedder
  • recall@1: 0.500
  • recall@3: 0.775
  • MRR: 0.629

These values are a project regression baseline, not a leaderboard or general retrieval-quality claim.

Validation

  • RAG Assistant: 34 passed, no skips
  • LLM evaluation bridge: 2 passed, no skips
  • Python compilation: 7 changed Python files
  • README and RESULTS relative links verified
  • whitespace and exact 21-file change boundary verified

Closes #18.

@lmdixon23
lmdixon23 marked this pull request as ready for review July 27, 2026 21:14
@lmdixon23
lmdixon23 merged commit 6aa2e30 into main Jul 27, 2026
21 checks passed
@lmdixon23
lmdixon23 deleted the issue-18-rag-benchmark-20260727 branch July 27, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RAG assistant: expand retrieval evaluation into a discriminative benchmark

1 participant