feat: grade hadith citations by sanad strength - #63
Conversation
Adds a hadith authenticity grading layer so the assistant never presents a weak or fabricated narration as authentic without saying so. hadith.py parses hadith references out of a model answer, normalizes collection names/numbering across common phrasings, and looks each one up in a bundled grading dataset covering Sahih al-Bukhari, Sahih Muslim, the four Sunan, and Muwatta Malik. Each grader's raw grade string is decomposed into authenticity strength (sahih/hasan/da'if/mawdu') and chain type (marfu/mauquf/maqtu/mursal), since a mauquf or maqtu report is not actually a saying of the Prophet regardless of how strong its chain is. When a hadith has multiple graders who disagree, the weakest strength and most cautious chain type win, and every individual grader's raw grade is kept in the record so nothing is hidden. The dataset is built by scripts/build_hadith_data.py from the public-domain fawazahmed0/hadith-api (pinned tag 1), bundling only grading metadata (never hadith text, sidestepping translation-copyright questions). Full provenance and the normalization policy are documented in data/hadith/PROVENANCE.md. main.py now appends a caution note to any answer that leans on a da'if or mawdu' hadith without already caveating it, or where an unverified reference is the sole hadith support in the answer. Weak hadith mentioned for targhib/tarhib with an explicit caveat are left alone. ChatResponse gains an optional hadith_references field so a frontend can render authenticity badges; existing clients are unaffected. The system prompt gets explicit hadith-citation rules (name the collection, state the grade, prefer the two Sahihs for evidentiary claims). CI now lints and compiles hadith.py and the build script, and runs the new offline test suite (tests/test_hadith.py).
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdds hadith grading and citation annotation, bundled metadata and generation tooling, caution policies, chat-response integration, tests, and CI coverage. ChangesHadith grading flow
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant ChatHandler
participant HadithAnnotator
participant BundledDataset
Client->>ChatHandler: Submit chat request
ChatHandler->>HadithAnnotator: Annotate generated or cached response
HadithAnnotator->>BundledDataset: Look up collection and hadith number
BundledDataset-->>HadithAnnotator: Return grading metadata
HadithAnnotator-->>ChatHandler: Return references and caution note
ChatHandler-->>Client: Return response with hadith references
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
Introduces an offline hadith authenticity grading layer that parses hadith citations from model answers, normalizes references, looks them up in a bundled grading dataset, and enforces a safety-first policy by appending a caution note when weak/unverified narrations are used without appropriate caveats.
Changes:
- Added
hadith.pyto parse/normalize hadith references, look up bundled grades, and generate a policy-driven caution note + structuredHadithReferencemetadata. - Added an offline dataset build script and provenance documentation for the bundled grading metadata.
- Integrated grading into
/chatresponses (including cache behavior) and extended CI to lint/compile/test the new module.
Reviewed changes
Copilot reviewed 6 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
hadith.py |
New grading/normalization/policy module and structured response metadata. |
main.py |
Appends hadith citation rules to the system context and annotates responses with grading + caution notes. |
scripts/build_hadith_data.py |
Builds the bundled grading dataset JSON from the pinned upstream source. |
data/hadith/PROVENANCE.md |
Documents provenance, licensing rationale, and normalization policy. |
.github/workflows/ci.yml |
Extends lint/compile and adds hadith test execution in CI. |
tests/test_hadith.py |
Offline unit + integration tests for tokenization, parsing, aggregation, and policy behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hadith.py`:
- Around line 388-402: Update _has_nearby_caveat to use the existing
_contains_word matcher for each keyword instead of plain substring containment
in the nearby window. Preserve the current window bounds and caveat keyword
list, ensuring unrelated words such as “weakness” do not suppress warnings while
valid single- and multi-word caveats still match.
In `@main.py`:
- Around line 290-297: Prevent synchronous hadith grading from blocking the
event loop in the async chat handler: update both the normal response path
around annotate_hadith/build_caution_note and the cache-hit path to offload the
grading work with asyncio.to_thread, or warm get_default_source at application
startup so its cached JSON index is loaded before requests arrive. Preserve the
existing caution text and cache behavior.
- Around line 290-297: Update the semantic-cache HIT handling around
cached.response and annotate_hadith so re-annotation uses only the response text
before the appended caution note, not the full cached text. Preserve returning
the cached caution note to the user while ensuring hadith_references match the
original pre-caution annotation and do not duplicate citations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 065cde6d-1103-457e-9669-b52eaddaa1b2
📒 Files selected for processing (13)
.github/workflows/ci.ymldata/hadith/PROVENANCE.mddata/hadith/abudawud.jsondata/hadith/bukhari.jsondata/hadith/ibnmajah.jsondata/hadith/malik.jsondata/hadith/muslim.jsondata/hadith/nasai.jsondata/hadith/tirmidhi.jsonhadith.pymain.pyscripts/build_hadith_data.pytests/test_hadith.py
| _CAVEAT_KEYWORDS = ( | ||
| "weak", "da'if", "daif", "fabricat", "not authentic", "not to be relied", | ||
| "not reliable", "should not be relied", "for encouragement", "targhib", | ||
| "tarhib", "not be used as evidence", "grading unverified", "unverified", | ||
| "not a strong basis", "narrated for", | ||
| ) | ||
|
|
||
| _CAVEAT_WINDOW = 150 | ||
|
|
||
|
|
||
| def _has_nearby_caveat(text: str, span: Tuple[int, int]) -> bool: | ||
| start = max(0, span[0] - _CAVEAT_WINDOW) | ||
| end = min(len(text), span[1] + _CAVEAT_WINDOW) | ||
| window = text[start:end].lower() | ||
| return any(keyword in window for keyword in _CAVEAT_KEYWORDS) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Caveat detection uses unbounded substring matching — risks false negatives on the exact safety check this PR exists for.
_has_nearby_caveat does keyword in window (plain substring containment) rather than reusing the file's own word-boundary helper _contains_word. Several keywords are common substrings of unrelated words: "weak" matches inside "tweak", "weaker", "weakness"; this means any incidental nearby occurrence of these substrings — even in unrelated context — will suppress the flagged warning for a genuinely unqualified da'if/mawdu' citation. That directly undermines the PR's stated goal of flagging weak/fabricated narrations used without qualification.
🛡️ Proposed fix: reuse the existing word-boundary matcher
def _has_nearby_caveat(text: str, span: Tuple[int, int]) -> bool:
start = max(0, span[0] - _CAVEAT_WINDOW)
end = min(len(text), span[1] + _CAVEAT_WINDOW)
window = text[start:end].lower()
- return any(keyword in window for keyword in _CAVEAT_KEYWORDS)
+ return any(_contains_word(window, keyword) for keyword in _CAVEAT_KEYWORDS)_contains_word already handles multi-word phrases correctly (\b anchors around the whole escaped phrase), so this is a drop-in fix. Worth adding a regression test alongside it, e.g. asserting a nearby "weakness of iman" (unrelated to the citation) does not suppress flagging of a genuinely unqualified da'if citation.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| _CAVEAT_KEYWORDS = ( | |
| "weak", "da'if", "daif", "fabricat", "not authentic", "not to be relied", | |
| "not reliable", "should not be relied", "for encouragement", "targhib", | |
| "tarhib", "not be used as evidence", "grading unverified", "unverified", | |
| "not a strong basis", "narrated for", | |
| ) | |
| _CAVEAT_WINDOW = 150 | |
| def _has_nearby_caveat(text: str, span: Tuple[int, int]) -> bool: | |
| start = max(0, span[0] - _CAVEAT_WINDOW) | |
| end = min(len(text), span[1] + _CAVEAT_WINDOW) | |
| window = text[start:end].lower() | |
| return any(keyword in window for keyword in _CAVEAT_KEYWORDS) | |
| _CAVEAT_KEYWORDS = ( | |
| "weak", "da'if", "daif", "fabricat", "not authentic", "not to be relied", | |
| "not reliable", "should not be relied", "for encouragement", "targhib", | |
| "tarhib", "not be used as evidence", "grading unverified", "unverified", | |
| "not a strong basis", "narrated for", | |
| ) | |
| _CAVEAT_WINDOW = 150 | |
| def _has_nearby_caveat(text: str, span: Tuple[int, int]) -> bool: | |
| start = max(0, span[0] - _CAVEAT_WINDOW) | |
| end = min(len(text), span[1] + _CAVEAT_WINDOW) | |
| window = text[start:end].lower() | |
| return any(_contains_word(window, keyword) for keyword in _CAVEAT_KEYWORDS) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hadith.py` around lines 388 - 402, Update _has_nearby_caveat to use the
existing _contains_word matcher for each keyword instead of plain substring
containment in the nearby window. Preserve the current window bounds and caveat
keyword list, ensuring unrelated words such as “weakness” do not suppress
warnings while valid single- and multi-word caveats still match.
| # --- Hadith authenticity grading --- | ||
| # Baked into response_text *before* the cache write so a cached hit | ||
| # replays the same caution the user originally saw. | ||
| hadith_refs = annotate_hadith(response_text) | ||
| caution = build_caution_note(response_text, hadith_refs) | ||
| if caution: | ||
| response_text = f"{response_text.rstrip()}\n\n{caution}" | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Synchronous hadith grading work runs inline inside the async request handler.
annotate_hadith/build_caution_note (and, on the cache-hit path, the same call at line 217) do regex parsing plus — on the first request touching each collection — a blocking open()/json.load() via BundledGradingSource._index. Since chat is async def, this synchronous work runs directly on the event loop thread, so a cold-start request can briefly stall other concurrent requests while the JSON dataset loads.
Consider warming get_default_source() at app startup (so the lru_cached index is already populated before traffic arrives), or wrapping the calls in await asyncio.to_thread(...) if warming isn't practical.
As per path instructions for **/*.py: "Flag blocking calls inside async endpoints (network calls should be awaited or offloaded)."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@main.py` around lines 290 - 297, Prevent synchronous hadith grading from
blocking the event loop in the async chat handler: update both the normal
response path around annotate_hadith/build_caution_note and the cache-hit path
to offload the grading work with asyncio.to_thread, or warm get_default_source
at application startup so its cached JSON index is loaded before requests
arrive. Preserve the existing caution text and cache behavior.
Source: Path instructions
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Cache write bakes the caution note into the cached text, then the cache-hit path re-annotates that same text — producing duplicated/inconsistent hadith_references on cache hits.
response_text (with the caution note already appended) is what gets stored via semantic_cache.put(...) at line 303. On a later semantic-cache HIT (line 217), hadith_references=annotate_hadith(cached.response) re-scans the entire cached text — which now includes the caution note's quoted citation strings (e.g. - "Sunan Abu Dawud 3" — graded daif — ...). Since parse_references just regex-scans for collection+number patterns, it matches the citation a second time inside the note itself, so a cache-hit response can report duplicate entries for the same hadith — and the duplicate can even end up with a different flagged value than the original (the note text itself contains caveat words like "daif", so _has_nearby_caveat returns True for the copy sitting inside the note). This makes the new hadith_references field unreliable specifically for cached responses, which is a meaningful chunk of production traffic.
Note the non-cached path (line 318) is fine — it uses hadith_refs computed before the caution note was appended, so it doesn't duplicate.
🩹 Minimal fix: re-annotate only the pre-caution-note portion of the cached text
+# hadith.py — export the marker so main.py doesn't hardcode it
+HADITH_CAUTION_MARKER = "Note on hadith authenticity in this answer:" return ChatResponse(
response=cached.response,
chat_id=chat_id,
history=cached.history,
fiqh=fiqh_info,
- hadith_references=annotate_hadith(cached.response),
+ hadith_references=annotate_hadith(
+ cached.response.split(HADITH_CAUTION_MARKER, 1)[0]
+ ),
)A more robust long-term fix is to persist the computed hadith_refs alongside the cached entry (in semantic_cache) instead of re-deriving them from text — worth a follow-up if semantic_cache's schema can be extended.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@main.py` around lines 290 - 297, Update the semantic-cache HIT handling
around cached.response and annotate_hadith so re-annotation uses only the
response text before the appended caution note, not the full cached text.
Preserve returning the cached caution note to the user while ensuring
hadith_references match the original pre-caution annotation and do not duplicate
citations.
Summary
Closes #53. Adds a hadith authenticity grading layer so the assistant never presents a weak or fabricated narration as authentic without saying so.
hadith.pyparses hadith references out of a model answer, normalizes collection names/numbering across common phrasings ("Sahih al-Bukhari 1" / "Bukhari [Enhancement] AI service responds slowly with no streaming or caching #1" / etc.), and looks each one up in a bundled grading dataset covering Sahih al-Bukhari, Sahih Muslim, the four Sunan, and Muwatta Malik.scripts/build_hadith_data.pyfrom the public-domainfawazahmed0/hadith-api(pinned tag1, Unlicense). Only grading metadata is bundled, never hadith text, which sidesteps translation-copyright questions entirely. Full provenance and the normalization policy are documented indata/hadith/PROVENANCE.md.main.pyappends a caution note to any answer that leans on a da'if/mawdu' hadith without already caveating it, or where an unverified reference is the sole hadith support in the answer. A weak hadith mentioned for targhib/tarhib with an explicit caveat is left alone.ChatResponsegains an optionalhadith_referencesfield (additive — existing clients unaffected) so a frontend can render authenticity badges. The system prompt gets explicit hadith-citation rules.SAHIHwith grader"Scholarly consensus (Bukhari and Muslim)", not left as a gap.hadith.pyand the build script, and runs the new offline test suite.Test plan
flake8 main.py stellar.py safety tests/redteam study.py fiqh.py hadith.py scripts/build_hadith_data.py --max-line-length=120 --ignore=E501,W503python -m compileall -q main.py stellar.py safety tests/redteam study.py fiqh.py hadith.py scripts/build_hadith_data.pypytest -q tests/test_hadith.py(72 tests: grade-string tokenizer against real dataset strings, collection alias normalization, reference parsing, policy enforcement for weak/mawdu/mauquf/unverified-as-sole-evidence cases, and integration smoke tests against the real bundled dataset)pytest -q tests/redteam tests/test_study.py tests/test_semantic_cache.py tests/test_fiqh.py— full existing suite still green/chatwith a mocked Gemini client: a da'if citation gets flagged with a caution note appended and structuredhadith_references; a Sahih al-Bukhari citation is verified and unflagged with no caution appendedSummary by CodeRabbit