Skip to content

RESULTS.md

Latest commit

 

History

History
199 lines (146 loc) · 7.99 KB

RESULTS.md

File metadata and controls

199 lines (146 loc) · 7.99 KB
Raw

Benchmark Results

All results are real, reproducible, and measured on this codebase. No projections, no estimates.

Hardware

  • Machine: Apple M1 Max, 64 GB RAM, macOS
  • Python: 3.13
  • Seed: 42 (deterministic)
  • Date: 2026-06-27

Real-World Results (zero-shot)

Athena is not trained or tuned on any of these datasets. The decision threshold is tuned on a disjoint split and reported on the held-out test split.

RAGTruth QA (response-level, 900 test responses, 18% positive)

Metric athena (NLI, zero-shot) Note
Balanced accuracy 0.71 TPR 0.71 / TNR 0.71 — the fair metric on imbalanced data
Accuracy 0.71
F1 (positive = hallucinated) 0.47 suppressed by 18% class imbalance

Reference (RAGTruth response-level F1): LettuceDetect-large 79.2 overall / 70.2 QA — a ModernBERT detector fine-tuned on the RAGTruth training set (not zero-shot); Luna 51.3 QA; GPT-4-turbo prompted 45.6 QA. Athena's balanced accuracy is competitive zero-shot; its lower F1 reflects the imbalanced response-level metric, not weaker per-sentence detection.

Aggregation matters: replacing "flag if any sentence is unsupported" with a length-normalized fraction over check-worthy sentences (skipping questions / refusals / "the passages do not mention X") lifts F1 from 0.44 → 0.47 and removes false positives on the ~17% of clean responses that contain a meta/refusal sentence.

HaluEval QA (per-answer, 500 held-out)

Metric athena (NLI, zero-shot)
Accuracy 0.69
Faithful-answer pass rate 0.82
Hallucinated-answer catch rate 0.56

Reference (HaluEval QA accuracy): GPT-3.5 ≈ 0.62, GPT-4 ≈ 0.85 (prompted). Athena runs locally at ~25 ms with zero cost.

Reproduce: clone RAGTruth / download HaluEval (instructions below) and run the harnesses in this directory.

Real Dataset Acquisition

RAGTruth (18K examples)

Download from: https://github.com/fluencelab/RAGTruth

# Clone the repository
git clone https://github.com/fluencelab/RAGTruth.git
cp -r RAGTruth/data/* ./data/ragtruth/

# Run benchmark
python benchmarks/run_ragtruth.py \
    --data-dir ./data/ragtruth \
    --subset qa \
    --output benchmarks/results/ragtruth.json

HaluEval (35K examples)

Download from: https://github.com/RUCAIBox/HaluEval

# Clone and prepare data
git clone https://github.com/RUCAIBox/HaluEval.git
cp -r HaluEval/data/* ./data/halueval/

# Run benchmark
python benchmarks/run_halueval.py \
    --data-dir ./data/halueval \
    --subset qa \
    --output benchmarks/results/halueval.json

FaithBench (multi-domain faithfulness)

Download from: https://huggingface.co/datasets/CogComp/FaithBench

# Using HF datasets library
python -c "from datasets import load_dataset; \
ds = load_dataset('CogComp/FaithBench'); \
import json; \
[print(json.dumps(x)) for x in ds['test'][:1000]]" > ./data/faithbench/faithbench.jsonl

# Run benchmark
python benchmarks/run_faithbench.py \
    --data-dir ./data/faithbench \
    --output benchmarks/results/faithbench.json

If datasets are unavailable, benchmarks can run on synthetic data:

python benchmarks/run_faithbench.py --synthetic \
    --num-synthetic 100 \
    --output benchmarks/results/faithbench_synthetic.json

Comprehensive Synthetic Benchmark (100 test cases, 298 sentences)

Six hallucination categories across legal, medical, technical, and general domains.

NLI-Only Mode (nli-deberta-v3-base, ~23ms p50)

Category Precision Recall F1
Fabricated claims 100.0% 96.0% 97.9%
Out-of-context 100.0% 96.7% 98.3%
Number substitutions 82.1% 95.8% 88.5%
Subtle contradictions 100.0% 96.7% 98.3%
Partial support 95.2% 90.9% 93.0%
Overall 94.5% 95.6% 95.0%
  • False positive rate on faithful sentences: 4.6% (4/87 sentences incorrectly flagged)
  • Latency p50: ~22.5ms per verification call
  • Latency p95: ~34.5ms per verification call
  • Cost: $0 (local model, no API calls)

NLI-Only Mode (nli-deberta-v3-large, ~53ms p50)

Category Precision Recall F1
Fabricated claims 100.0% 98.7% 99.3%
Out-of-context 100.0% 93.3% 96.5%
Number substitutions 82.1% 95.8% 88.5%
Subtle contradictions 100.0% 93.3% 96.5%
Partial support 90.9% 90.9% 90.9%
Overall 94.5% 95.6% 95.0%
  • False positive rate on faithful sentences: 3.4% (3/87 sentences incorrectly flagged)
  • Latency p50: ~53.4ms per verification call
  • Latency p95: ~89.2ms per verification call

How It Works

Context chunks are split into individual sentences before NLI scoring. Each answer sentence is scored against every context sentence and the full chunk, and the maximum entailment score is used. This avoids the "neutral trap" where NLI models classify a hypothesis as neutral when the premise contains information beyond the hypothesis, while still catching facts spread across several context sentences. Answer sentences that open with an anaphor ("This cap…", "It also…") are joined with the previous sentence before scoring so the referent is preserved.

NLI still scores many faithful paraphrases as neutral (entailment ≈ 0). A guarded rescue recovers them without admitting hallucinations: a not-entailed sentence is lifted to partially supported only when (a) the most on-topic context unit does not contradict it — read from the 3-class NLI distribution, picked by lexical relevance so an unrelated unit can't veto a faithful claim — and (b) every number in the sentence appears in the context, and (c) most of its content words appear in the context. This is what cut the faithful false-positive rate from 17% to 4.6% (base) / 3.4% (large) while holding hallucination recall at ~96%.

The Right Tool for the Right Job

Use case Recommended mode Why
General RAG QA NLI-only (base) 95.0% F1 in ~23ms
High-stakes docs NLI-only (large) Lower false-positive rate at ~53ms
Real-time chat NLI-only (base) ~24ms latency is production-ready
Maximum accuracy NLI + LLM-judge LLM catches paraphrases NLI misses

Latency Comparison

Mode p50 p95 Notes
NLI only (base) ~23ms ~35ms Fastest, 95.0% F1, 4.6% FP on faithful
NLI only (large) ~53ms ~89ms Lower FP (3.4%), 95.0% F1
LLM judge (local) ~7.4s ~10s Per sentence, local gemma-4-31b-it
GPT-4 judge (API) ~2s ~5s Per sentence, network round-trip

FaithBench Synthetic Benchmark (100 synthetic examples)

Simple faithful vs. hallucinated classification. Synthetic data is by design clear-cut: all hallucinated sentences are obviously false, all faithful sentences are obviously true.

Metric Value
Precision 100.0%
Recall 100.0%
F1 100.0%
ECE 0.144
Latency p50 22ms
Latency p95 86ms
Cost Free (local model)

Note: Perfect scores on synthetic data don't guarantee real-world performance. Real hallucinations are subtler (paraphrases, number approximations, missing context).

Reproduction

# NLI-only benchmark (comprehensive)
pip install -e ".[nli]"
python benchmarks/run_full_eval.py

# FaithBench synthetic benchmark
python benchmarks/run_faithbench.py --synthetic

# Hybrid NLI + LLM-judge benchmark
# Requires LM Studio running gemma-4-31b-it on localhost:1234
pip install -e ".[nli]" openai
python benchmarks/run_hybrid_eval.py

Baselines

Baseline Type Open source? Cost
Ragas faithfulness Offline eval Yes API calls
GPT-4-as-judge LLM prompt No ~$3/1K sentences
Vectara HHEM Metric model Yes (weights) Free (local)
Athena NLI-only Runtime guardrail Yes Free (local)