[Enhancement] Retrieval-quality evaluation and metrics for the RAG pipeline

[zeemscript]

Summary

Retrieval-augmented generation (#24) is the campaign's biggest bet on answer quality — grounding answers in authenticated Quran and Hadith instead of the model's memory. But RAG only helps if it retrieves the right passages, and RAG fails silently: it hands the model irrelevant or missing context and the model confidently answers anyway. The eval harness (#16) measures whether the final answer is good; it does not measure whether retrieval did its job. This issue builds the retrieval-quality evaluation layer: a labeled retrieval test set, standard IR metrics (recall@k, precision@k, MRR, nDCG), and answer-groundedness/faithfulness checks — so the team can tell a retrieval problem from a generation problem and tune the RAG pipeline with numbers instead of vibes.

Current state

• There is no retrieval today at all — /chat sends the raw prompt to Gemini (main.py, lines 143–151) with no grounding, so there is nothing to evaluate yet. This issue is scoped to land with or just after the RAG pipeline ([Enhancement] Retrieval-augmented generation over authenticated Quran and Hadith sources #24); it defines the metrics and harness that [Enhancement] Retrieval-augmented generation over authenticated Quran and Hadith sources #24's retriever must be measured against.
• The offline answer-eval harness is issue [Enhancement] Evaluation harness for Islamic answer quality #16 — it scores final answer quality. It has no notion of retrieved passages, retrieval recall, or groundedness. That is the gap here.
• CI (.github/workflows/ci.yml) lints and compiles only main.py, installs pytest but runs no tests, and has no evaluation step — so no quality metric of any kind currently runs in CI.

What to build

1. A labeled retrieval dataset — a set of questions each annotated with the relevant source passages (surah:ayah / collection:number) that a correct answer should draw on. Cover the question types RAG must serve (direct ayah lookup, thematic ("verses about patience"), hadith-on-a-topic, multi-source). Store it in a documented format that [Enhancement] Retrieval-augmented generation over authenticated Quran and Hadith sources #24's retriever can be run against and that [Enhancement] Evaluation harness for Islamic answer quality #16's answer cases can cross-reference — coordinate, don't fork the dataset conventions.
2. Retrieval metrics — compute recall@k, precision@k, MRR, and nDCG for the retriever over the labeled set, plus a coverage report (which query types retrieve well vs. poorly). Make k and the embedding/retriever config parameters so the harness can compare configurations.
3. Groundedness / faithfulness checks — beyond IR metrics, verify the generated answer is actually supported by the retrieved passages: flag answers that assert claims absent from the retrieved context (an LLM-judge or NLI-style check is acceptable; reuse #ai-18's claim-extraction if available rather than a second extractor), and flag when the model ignored good retrieved context.
4. A CLI + CI wiring — a runnable command that reports the metrics table, and an optional CI job (gated, since it may need API access) or an offline-mockable subset that runs in CI to catch retrieval regressions. Respect that CI currently has no secrets — keep the default CI path offline with fixtures.
5. Regression thresholds — configurable minimum recall@k / groundedness that a RAG change must not drop below, so the harness can fail a PR that degrades retrieval.
6. Tests — offline unit tests for each metric against hand-computed expected values (recall@k, MRR, nDCG on toy rankings), so the evaluator itself is trustworthy. No live API calls in the default CI path.

Acceptance criteria

• A labeled retrieval dataset exists in a documented format, covering direct, thematic, hadith-topic, and multi-source query types, cross-referenceable with [Enhancement] Evaluation harness for Islamic answer quality #16's answer cases.
• The harness computes recall@k, precision@k, MRR, and nDCG for [Enhancement] Retrieval-augmented generation over authenticated Quran and Hadith sources #24's retriever, with k and retriever config as parameters and a per-query-type coverage report.
• Groundedness checks flag answers asserting claims absent from retrieved context and answers that ignore good context, reusing #ai-18's claim extraction where available.
• A CLI reports the metrics table; the default CI path runs an offline, fixture-based subset (no secrets required); configurable thresholds can fail a regressing PR.
• The metric implementations have offline unit tests against hand-computed expected values, so the evaluator itself is verified.
• The dataset and formats coordinate with [Enhancement] Evaluation harness for Islamic answer quality #16 and [Enhancement] Retrieval-augmented generation over authenticated Quran and Hadith sources #24 rather than forking conventions; CI stays green.

Pointers

• main.py — /chat handler (lines 118–184) is where [Enhancement] Retrieval-augmented generation over authenticated Quran and Hadith sources #24 will inject retrieved context; the harness measures the retriever feeding it.
• Adjacent issues this depends on / composes with: [Enhancement] Retrieval-augmented generation over authenticated Quran and Hadith sources #24 (RAG pipeline — the system under test; align on retriever interface and passage IDs), [Enhancement] Evaluation harness for Islamic answer quality #16 (answer-quality eval — share dataset conventions; this adds the retrieval layer [Enhancement] Evaluation harness for Islamic answer quality #16 lacks), [Enhancement] Citation verification: check quoted Quran text against a bundled corpus and flag unverifiable references #40 (Quran corpus + citation verification — relevance labels reference the same passage IDs), #ai-18 (claim extraction — reuse for groundedness), [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27 (embeddings — the retriever config under evaluation).
• CI (.github/workflows/ci.yml) installs pytest but runs nothing and has no secrets — add a pytest step and keep the default eval path offline with fixtures.
• PRs target the dev branch (see CONTRIBUTING.md).

Difficulty

High — trustworthy retrieval evaluation means building a well-labeled relevance dataset (labor-intensive and judgment-heavy), correctly implementing and unit-testing IR metrics, and designing groundedness checks that fairly separate retrieval failures from generation failures — all while depending on and coordinating tightly with the RAG (#24) and eval (#16) work rather than colliding with them.

---

🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the dev branch. Quality bar: CI must stay green.

💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0