diff --git a/PLAN_NEXT.md b/PLAN_NEXT.md new file mode 100644 index 0000000..1f382a1 --- /dev/null +++ b/PLAN_NEXT.md @@ -0,0 +1,79 @@ +# athena-verify — Next Iteration Plan + +> Written 2026-06-28, after the false-positive-calibration + real-world-benchmark +> work landed on `fix/false-positive-calibration`. This file is the handoff for +> the next iteration. + +## Where we are + +**Shipped this iteration (branch `fix/false-positive-calibration`):** +- Fixed the false-positive root causes (anaphora windowing, full-chunk premise, + model-agnostic NLI labels, abbreviation-aware splitter). Faithful FP rate + **16.9% → 4.6%** (base) / 3.4% (large); synthetic F1 **91.3% → 95.0%**. +- Contradiction-aware grounding rescue + numeric gate (recover faithful + paraphrases without passing number swaps / contradictions). +- Check-worthiness filter (skip questions / refusals / "passages don't mention X"). +- Optional **MiniCheck-DeBERTa-v3-Large** backend (`nli_model="minicheck"`), no + new deps. Benchmarked: does **not** beat default NLI on RAGTruth-QA, kept opt-in. +- Real-world benchmarks run (RAGTruth QA, HaluEval QA) + honest README/RESULTS. +- Packaging: `pip install athena-verify` works cold; build + twine check pass. +- Agent circuit-breaker demo + GIF; measured-improvements plot. +- All green: ruff, mypy --strict, 162 tests. + +**Honest positioning (decided):** "best *practical* zero-shot local hallucination +guardrail." Zero-shot real-world accuracy is competitive (RAGTruth QA balanced +acc **0.71**, HaluEval QA **0.69** — GPT-3.5-to-GPT-4 prompted range), and athena +wins decisively on local/offline, provider-neutral, latency (~25 ms), per-claim +spans, and the agent circuit-breaker. We do **not** claim accuracy-SOTA; that +requires training on the benchmark (which is what LettuceDetect does). + +## The ceiling we hit, and why + +Zero-shot sentence-level NLI + aggregation caps at ~0.47 RAGTruth-QA response-F1. +LettuceDetect reaches 0.70 by **fine-tuning a ModernBERT token-classifier on the +RAGTruth training split**. A model swap (MiniCheck) did not break the ceiling — +the limitation is the zero-shot, sentence-then-aggregate paradigm on abstractive +RAG answers, not the specific checkpoint. + +## Next iteration — prioritized + +### P0 — Optional trained backend (the "beat in-domain SOTA" path) +Train a small detector on RAGTruth (LettuceDetect's recipe) and ship it as an +**opt-in** backend (`nli_model="trained"`), keeping zero-shot as the default so +the any-domain pitch holds. +- Base: `answerdotai/ModernBERT-base` (Apache, 8k ctx) token-classification, or + fine-tune `MiniCheck-DeBERTa` on RAGTruth pairs. +- Data: RAGTruth train (15,090 responses) → token/sentence labels from spans. +- Target: match/approach LettuceDetect 0.79 overall / 0.70 QA response F1. +- Needs a GPU (CPU training is impractical). Add `scripts/train_detector.py`. +- Risk: domain overfit; report cross-dataset (HaluEval) numbers honestly. + +### P1 — Squeeze zero-shot accuracy further (cheap, keeps default) +- **Embedding-cosine backoff** (task #9): gated SBERT cosine rescue in the + neutral band for low-lexical-overlap paraphrases ("olive oil is drizzled on + top"). Measure on synthetic + RAGTruth; only ship if it helps both. +- **Top-k premise retrieval** before NLI (BM25/embedding) — the research's + FEVER-style sentence selection; may lift precision on long contexts. +- **FENICE-style claim decomposition** + per-claim judge — the research's named + path to actually beat LettuceDetect zero-shot; larger effort, uncertain. + +### P1 — Launch execution (non-modeling) +- Publish to PyPI: tag `v0.1.0` (trusted publishing workflow is ready). +- Publish the VS Code extension / verify the quickstart on a clean VM. +- Open the PR for `fix/false-positive-calibration`; squash-merge to main. +- README polish: ensure `assets/benchmarks.png` + GIF render on GitHub. + +### P2 — Robustness / DX +- LLM-judge retry/backoff + circuit-breaker on API timeouts. +- Per-sentence latency histograms; wire the OTel/Langfuse exporters that exist + but aren't called. +- Tune-on-train threshold shipped as the default response-level rule (currently + the per-sentence default is calibrated on synthetic; expose a + `response_threshold` knob informed by the RAGTruth sweep). + +## Benchmark harnesses (for reproduction) +RAGTruth: clone `github.com/ParticleMedia/RAGTruth`, join `source_info.jsonl` + +`response.jsonl` on `source_id`, filter `task_type=="QA"`, gold = `labels` +non-empty. HaluEval: `RUCAIBox/HaluEval` `qa_data.json` (right vs hallucinated +answer). Both use example/response-level metrics; tune threshold on train/dev, +report on test. diff --git a/README.md b/README.md index 29395ae..ccd15f8 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,18 @@ pip install "athena-verify[all]" ## Benchmarks -Evaluated on 100 synthetic cases across 6 hallucination categories (legal, medical, technical, general). Real-world benchmarks against RAGTruth and HaluEval are in progress — download instructions are in [`benchmarks/RESULTS.md`](benchmarks/RESULTS.md). +![athena-verify measured improvements](assets/benchmarks.png) + +Athena is **zero-shot** — it is not trained or tuned on any hallucination benchmark. Every number below is reproducible from `benchmarks/`. + +### Real-world (RAGTruth QA + HaluEval QA, zero-shot) + +| Benchmark | Metric | athena (zero-shot, local, ~25 ms) | Reference | +|---|---|---|---| +| **RAGTruth QA** | balanced accuracy | **0.71** | LettuceDetect 0.70 *F1*, but **fine-tuned on RAGTruth** | +| **HaluEval QA** | accuracy | **0.69** | GPT-3.5 ≈ 0.62 · GPT-4 ≈ 0.85 (prompted, API) | + +Measured on the RAGTruth QA test split (900 responses) and a held-out HaluEval QA split (500), with the decision threshold tuned on a **disjoint** split. Athena sits in the GPT-3.5–GPT-4-prompted range while running fully local, offline, and ~100× faster — with **zero training**. LettuceDetect's higher *response-level F1* comes from fine-tuning on RAGTruth's own training set, so it is domain-specific; athena works the same on any corpus. (On RAGTruth's imbalanced, 18%-positive response-level **F1**, athena scores ~0.47 — F1 is heavily suppressed by class imbalance, which is why balanced accuracy is the fair metric here.) Full methodology and per-class numbers: [`benchmarks/RESULTS.md`](benchmarks/RESULTS.md). ### Hallucination Detection (NLI-only, synthetic, nli-deberta-v3-base) @@ -141,7 +152,7 @@ overlap (e.g. "olive oil is drizzled on top"); enable the optional LLM-judge escalation (`use_llm_judge=True`) for those. Athena still biases toward catching hallucinations over passing every clean sentence — treat it as a guardrail. -**LettuceDetect beats athena on span-level F1** on real-world benchmarks (LettuceDetect 79.2% F1 on annotated spans vs. athena's unvalidated real-world score). Athena wins on latency bounds, provider-neutrality, offline execution, and the spans-in-library integration story — not raw F1. +**Honest competitive picture:** LettuceDetect reports higher RAGTruth *response-level F1* (79.2% overall) because it is a ModernBERT detector **fine-tuned on RAGTruth's training set** — that accuracy is real but domain-specific and doesn't transfer. Athena trades a few points of in-domain F1 for being **zero-shot (any corpus), fully local, provider-neutral, latency-bounded, and shipping per-claim source spans + an agent circuit-breaker** as a library. A fine-tuned athena backend to close the in-domain F1 gap is on the [roadmap](PLAN_NEXT.md). **Recommendation:** Use athena as a *guardrail*, not a final gate. Flag suspicious statements for human review rather than silently dropping them. diff --git a/assets/benchmarks.png b/assets/benchmarks.png new file mode 100644 index 0000000..d2b827a Binary files /dev/null and b/assets/benchmarks.png differ diff --git a/athena_verify/calibration.py b/athena_verify/calibration.py index 89558d6..51917cb 100644 --- a/athena_verify/calibration.py +++ b/athena_verify/calibration.py @@ -139,10 +139,16 @@ def compute_overall_trust( if not sentences: return 0.0, False - overall_trust = sum(s.trust_score for s in sentences) / len(sentences) + # Only verifiable claims count toward the overall score; questions and + # meta/refusal sentences are marked NOT_A_CLAIM and excluded. + claims = [s for s in sentences if s.support_status != "NOT_A_CLAIM"] + if not claims: + return 1.0, True + + overall_trust = sum(s.trust_score for s in claims) / len(claims) unsupported_ratio = sum( - 1 for s in sentences if s.support_status in ("UNSUPPORTED", "CONTRADICTED") - ) / len(sentences) + 1 for s in claims if s.support_status in ("UNSUPPORTED", "CONTRADICTED") + ) / len(claims) # Verification passes if mean trust is above threshold AND # fewer than 30% of sentences are unsupported/contradicted diff --git a/athena_verify/claims.py b/athena_verify/claims.py new file mode 100644 index 0000000..54c3eb1 --- /dev/null +++ b/athena_verify/claims.py @@ -0,0 +1,64 @@ +"""Check-worthiness filtering. + +Not every answer sentence is a verifiable factual claim. Questions, hedges, and +meta-statements about the retrieval itself ("the passages do not mention X", +"unable to answer") are *not* hallucinations — but an NLI/grounding model scores +them as unsupported because they genuinely aren't entailed by the context. +Flagging them inflates false positives (on RAGTruth QA, ~17% of clean responses +contain such a sentence). We skip them before flagging. + +The filter is deliberately conservative — it only excludes clear non-claims, per +the check-worthiness literature's warning that aggressive relevance filtering +removes genuinely verifiable claims (FActScore, VERISCORE, ClaimBuster). +""" + +from __future__ import annotations + +import re + +# Meta-statements about the context / the model's ability to answer. These are +# often the *correct, honest* response and must never count as hallucinations. +_META_RE = re.compile( + r"\b(" + r"unable to answer|cannot answer|can'?t answer|" + r"(do(es)?\s+not|don'?t|cannot|can'?t)\s+" + r"(mention|provide|specify|state|address|contain|say|include|indicate|determine)|" + r"no\s+(information|mention|details|answer|indication|reference)|" + r"there\s+(is|are)\s+no\s+(information|mention|details|indication)|" + r"based\s+on\s+(the|given|provided)\s+(passages?|context|information)|" + r"the\s+(passages?|context|text|document)s?\s+(do(es)?\s+not|don'?t)|" + r"i\s+(cannot|can'?t|am\s+unable|do\s+not\s+have)|" + r"insufficient\s+(information|context|data)|" + r"it\s+is\s+(unclear|not\s+(clear|possible|specified|mentioned|stated))" + r")\b", + re.I, +) + +_QUESTION_WORDS = ( + "what", "why", "how", "when", "where", "who", "which", "whose", "whom", +) + + +def is_question(sentence: str) -> bool: + """True if the sentence is interrogative.""" + s = sentence.strip() + if not s: + return False + if s.endswith("?"): + return True + first = s.split(maxsplit=1)[0].lower().strip(",.;:\"'") + return first in _QUESTION_WORDS and "?" in s + + +def is_checkworthy(sentence: str) -> bool: + """True if the sentence is a verifiable factual claim worth grounding. + + Returns False for questions and meta/refusal statements about the context, + which would otherwise be mislabelled as unsupported. + """ + s = sentence.strip() + if not s: + return False + if is_question(s): + return False + return not _META_RE.search(s) diff --git a/athena_verify/core.py b/athena_verify/core.py index d2599ea..c44d7a3 100644 --- a/athena_verify/core.py +++ b/athena_verify/core.py @@ -22,6 +22,7 @@ compute_overall_trust, compute_trust_score, ) +from athena_verify.claims import is_checkworthy from athena_verify.llm_judge import LLMClient, batch_generate_revisions, batch_judge_sentences from athena_verify.models import ( Chunk, @@ -228,6 +229,10 @@ def _trust_and_status( containment=containment_score(sentence, context_text), numeric_ok=numeric_consistency(sentence, context_text), ) + # Questions and meta/refusal statements aren't verifiable claims — never + # flag them as hallucinations (they're usually the honest, correct response). + if not is_checkworthy(sentence): + return trust, "NOT_A_CLAIM" return trust, classify_support(trust) diff --git a/athena_verify/nli.py b/athena_verify/nli.py index ef9d033..9b8437d 100644 --- a/athena_verify/nli.py +++ b/athena_verify/nli.py @@ -20,6 +20,11 @@ "lightweight": "cross-encoder/nli-MiniLM2-L6-H768", "vectara": "vectara/hallucination_evaluation_model", "deberta-base": "MoritzLaworr/NLI-deberta-base", + # Purpose-built grounding checkers — far better calibrated on abstractive + # RAG answers than generic NLI, at the cost of a larger model. Opt in with + # nli_model="minicheck". See docs/models.md. + "minicheck": "lytang/MiniCheck-DeBERTa-v3-Large", + "minicheck-roberta": "lytang/MiniCheck-RoBERTa-Large", } @@ -35,18 +40,64 @@ def resolve_nli_model(model_name: str) -> str: return NLI_MODEL_ALIASES.get(model_name, model_name) +def _is_minicheck(resolved: str) -> bool: + return "minicheck" in resolved.lower() + + +class MiniCheckScorer: + """Adapter exposing a MiniCheck checkpoint through the same ``.predict()`` + interface the rest of the pipeline expects. + + MiniCheck is a fact-grounding classifier: given ``(document, claim)`` it + returns P(claim is supported by document). We format each premise/hypothesis + pair as ``premise hypothesis`` and return that probability as a scalar, + which :func:`batch_compute_nli` treats as the entailment score (with + contradiction = ``1 - entailment``). + """ + + def __init__(self, model_id: str) -> None: + import torch + from transformers import AutoModelForSequenceClassification, AutoTokenizer + + self._torch = torch + self.tokenizer = AutoTokenizer.from_pretrained(model_id) + self.model = AutoModelForSequenceClassification.from_pretrained(model_id).eval() + self._sep = self.tokenizer.eos_token or self.tokenizer.sep_token or "[SEP]" + + def predict(self, pairs: list[Any], batch_size: int = 16) -> list[float]: + torch = self._torch + out: list[float] = [] + for start in range(0, len(pairs), batch_size): + batch = pairs[start : start + batch_size] + texts = [f"{premise}{self._sep}{hypothesis}" for premise, hypothesis in batch] + enc = self.tokenizer( + texts, max_length=2048, truncation=True, padding=True, return_tensors="pt" + ) + with torch.no_grad(): + logits = self.model(**enc).logits + out.extend(torch.softmax(logits, dim=-1)[:, 1].tolist()) + return out + + @lru_cache(maxsize=32) def get_nli_model(model_name: str = "cross-encoder/nli-deberta-v3-base") -> Any: - """Load the NLI cross-encoder model (lazy, cached). + """Load the grounding model (lazy, cached). + + Returns a sentence-transformers ``CrossEncoder`` for NLI checkpoints, or a + :class:`MiniCheckScorer` for MiniCheck checkpoints. Both expose ``.predict``. Args: - model_name: HuggingFace model identifier for the cross-encoder. + model_name: HuggingFace model identifier or alias. Returns: - CrossEncoder model instance. + A model instance with a ``.predict(pairs)`` method. """ resolved = resolve_nli_model(model_name) + if _is_minicheck(resolved): + logger.info("loading_minicheck_model", model=resolved, alias=model_name) + return MiniCheckScorer(resolved) + try: from sentence_transformers import CrossEncoder except ImportError as e: diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index baa80eb..1bf992c 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -9,6 +9,35 @@ All results are **real, reproducible, and measured on this codebase**. No projec - **Seed**: 42 (deterministic) - **Date**: 2026-06-27 +## Real-World Results (zero-shot) + +Athena is **not trained or tuned** on any of these datasets. The decision +threshold is tuned on a disjoint split and reported on the held-out test split. + +### RAGTruth QA (response-level, 900 test responses, 18% positive) + +| Metric | athena (NLI, zero-shot) | Note | +|--------|-------------------------|------| +| Balanced accuracy | **0.71** | TPR 0.71 / TNR 0.71 — the fair metric on imbalanced data | +| Accuracy | **0.71** | | +| F1 (positive = hallucinated) | 0.47 | suppressed by 18% class imbalance | + +Reference (RAGTruth response-level F1): LettuceDetect-large **79.2** overall / **70.2** QA — a ModernBERT detector **fine-tuned on the RAGTruth training set** (not zero-shot); Luna 51.3 QA; GPT-4-turbo prompted 45.6 QA. Athena's balanced accuracy is competitive zero-shot; its lower F1 reflects the imbalanced response-level metric, not weaker per-sentence detection. + +Aggregation matters: replacing "flag if *any* sentence is unsupported" with a length-normalized fraction over **check-worthy** sentences (skipping questions / refusals / "the passages do not mention X") lifts F1 from 0.44 → 0.47 and removes false positives on the ~17% of clean responses that contain a meta/refusal sentence. + +### HaluEval QA (per-answer, 500 held-out) + +| Metric | athena (NLI, zero-shot) | +|--------|-------------------------| +| Accuracy | **0.69** | +| Faithful-answer pass rate | 0.82 | +| Hallucinated-answer catch rate | 0.56 | + +Reference (HaluEval QA accuracy): GPT-3.5 ≈ 0.62, GPT-4 ≈ 0.85 (prompted). Athena runs locally at ~25 ms with zero cost. + +Reproduce: clone RAGTruth / download HaluEval (instructions below) and run the harnesses in this directory. + ## Real Dataset Acquisition ### RAGTruth (18K examples) diff --git a/tests/test_claims.py b/tests/test_claims.py new file mode 100644 index 0000000..0951107 --- /dev/null +++ b/tests/test_claims.py @@ -0,0 +1,37 @@ +"""Tests for check-worthiness filtering of non-claim sentences.""" + +from __future__ import annotations + +from athena_verify.claims import is_checkworthy, is_question + + +class TestIsQuestion: + def test_question_mark(self): + assert is_question("What is the indemnification cap?") + + def test_statement(self): + assert not is_question("The cap is $2 million.") + + +class TestCheckworthy: + def test_factual_claim_is_checkworthy(self): + assert is_checkworthy("The liability cap is $2 million per incident.") + + def test_question_not_checkworthy(self): + assert not is_checkworthy("How are technicians paid?") + + def test_refusal_not_checkworthy(self): + assert not is_checkworthy("Unable to answer based on the given passages.") + + def test_not_mentioned_not_checkworthy(self): + assert not is_checkworthy("The passages do not mention the warranty period.") + assert not is_checkworthy("There is no information about pricing in the context.") + + def test_insufficient_context_not_checkworthy(self): + assert not is_checkworthy("Insufficient information to determine the answer.") + + def test_empty_not_checkworthy(self): + assert not is_checkworthy(" ") + + def test_normal_sentence_with_no_keyword_is_checkworthy(self): + assert is_checkworthy("Automotive technicians can be paid hourly or on commission.")