Evidence Drift

A clinical RAG benchmark for preserving guideline disagreement, temporal drift, and corpus boundaries.

When using RAG in the context of clinical AI, an answer can cite a legitimate guideline but still misrepresent the entire picture of evidence. One example is retrieving one authoritative source while missing another equally relevant source with a different recommendation, and presenting the result as consensus.

This repo shares a way to make that failure mode testable. The repo uses breast cancer screening guidance from the USPSTF, American Cancer Society, ACOG, and American College of Radiology to evaluate whether a retrieval pipeline can:

• retrieve the set of sources needed to represent disagreement
• distinguish current guidance from a superseded recommendation
• preserve “insufficient evidence” rather than translating it into “recommended against”
• abstain when the question falls outside the declared corpus scope

Research demonstration only. Not medical advice, not a validated clinical tool, and not intended for patient care.

This Project vs a simple RAG chatbot

The project separates the system into inspectable layers instead of producing one clean answer. These layers are:

source provenance
      ↓
ingestion and chunking
      ↓
semantic + BM25 retrieval
      ↓
pinned-date status scoring
      ↓
organization-aware diversification
      ↓
scope guard + absolute support threshold
      ↓
answer or abstain
      ↓
evidence map / optional synthesis

Retrieval is evaluated separately from generation. A strong model can conceal weak retrieval with a plausible explanation so I left these separate.

Current regression snapshot

This benchmark is small and single-author. The holdout slice is frozen for regression testing, but it is not independently clinician reviewed and should not be interpreted as clinical validation.

Default configuration:

Embedding provider: local HashingVectorizer
Top-k: 6
Reference date: 2026-06-24
Abstention threshold: 0.12
Organization diversification: enabled

Hybrid retrieval

Slice	Cases	Answerable	Abstention	Recall@6	MRR	Abstention accuracy	Overall decision accuracy
Canonical	12	10	2	1.000	0.883	1.000	1.000
Frozen holdout	8	5	3	1.000	1.000	1.000	1.000

Retrieval ablation

Slice	Mode	Recall@6	MRR	Abstention accuracy	Overall decision accuracy
Canonical	BM25	1.000	0.900	1.000	1.000
Canonical	Semantic	0.917	0.883	1.000	0.750
Canonical	Hybrid	1.000	0.883	1.000	1.000
Holdout	BM25	0.950	1.000	1.000	0.875
Holdout	Semantic	1.000	1.000	1.000	1.000
Holdout	Hybrid	1.000	1.000	1.000	1.000

These are deterministic local regression results for the bundled corpus and questions.

Tracked snapshots are in results/snapshots.

CI recomputes the metrics on every push and fails if they no longer match the committed files. When a change is intended to move them, regenerate and review the snapshots:

python scripts/update_snapshots.py          # rewrite snapshots after an intended change
python scripts/update_snapshots.py --check   # verify (this is what CI runs)

Example behavior

Question:

How often should an average-risk 48-year-old receive screening mammography,
and do major organizations agree?

The hybrid retriever returns current evidence from all four core organizations rather than letting multiple ACR or ACOG pages crowd out USPSTF. The evidence map exposes each source’s organization, status, dates, rank score, and absolute support score.

Out-of-scope question:

What screening should a 32-year-old BRCA1 carrier receive?

Expected behavior:

{
  "abstain": true,
  "scope_flags": ["known_high_risk_genetics"],
  "abstain_reason": "Question is outside the declared average-risk screening corpus scope: known_high_risk_genetics"
}

The evaluator scores this as abstention. It does not assign perfect recall just because the gold-document list is empty.

Corpus

The repository contains short, project-authored evidence cards that paraphrase public recommendations (instead of redistributing complete guideline pages).

Included sources:

• USPSTF 2024 current recommendation
• USPSTF 2016 superseded recommendation
• American Cancer Society average-risk guidance
• ACOG 2024 update
• ACOG patient FAQ
• ACR current position
• ACR response to the 2024 USPSTF recommendation

The declared scope is asymptomatic, average-risk screening. Known high-risk genetics, previous chest radiation, previous breast cancer, pregnancy, male screening, and diagnostic symptoms are excluded.

See data/corpus/manifest.yaml for canonical URLs and provenance.

Date semantics

The manifest separates:

• published_date: date stated by the source
• effective_date: date a recommendation took effect, when known
• verified_on: date the repository author checked the page

verified_on is never presented as a publication date and is not used for freshness scoring.

Freshness uses a pinned REFERENCE_DATE so evaluation output does not drift if the code runs on a different day.

Abstention design

Ranking scores are normalized within a query which is useful for ordering results but unsafe for abstention, because even an irrelevant query tends to have a top-ranked passage.

Evidence Drift therefore uses two separate mechanisms:

1. Explicit corpus-boundary guard for declared exclusions such as BRCA-associated risk, prior chest radiation, and diagnostic breast symptoms.
2. Absolute support score combining raw semantic similarity with saturated BM25 for questions that do not match a known exclusion phrase.

The benchmark labels should_abstain directly. Abstention cases receive abstention accuracy; they do not receive Recall@k.

Organization-aware diversification

Clinical comparison questions often need one source from each organization. The default retriever selects the highest-ranked current passage from each organization before filling remaining positions.

For explicitly historical questions, superseded sources remain eligible in the first pass. This prevents duplicate institutional pages from hiding a major conflicting organization while preserving temporal-drift retrieval.

Quickstart

1. Create an environment

Requires Python 3.11 or newer. Check your installed version:

python3 --version

git clone https://github.com/sidoody/evidence-drift-rag.git
cd evidence-drift-rag
python3 -m venv .venv
source .venv/bin/activate

Windows PowerShell:

py -m venv .venv
.\.venv\Scripts\Activate.ps1

2. Install

python -m pip install --upgrade pip
pip install -e ".[dev]"
cp .env.example .env

Windows:

Copy-Item .env.example .env

The default setup runs locally without an API key. The OpenAI SDK is not installed unless you explicitly enable the optional integration.

3. Validate

ruff check .
pytest

4. Run the evaluations

python -m evidence_drift.cli evaluate --slice canonical --mode hybrid
python -m evidence_drift.cli evaluate --slice holdout --mode hybrid

5. Run the ablation

python -m evidence_drift.cli ablation --slice canonical
python -m evidence_drift.cli ablation --slice holdout

6. Launch the app

streamlit run app.py

Open the local URL printed by Streamlit, usually http://localhost:8501.

Streamlit interface

The app uses a compact product-style layout with:

• a clear answer-versus-abstain decision banner
• compact support and source-diversity metrics
• structured synthesis cards instead of raw JSON
• a dedicated evidence tab with small, readable excerpts
• source-status pills and individual retrieval signals
• responsive spacing for laptop and mobile widths

All advanced retrieval settings remain in the sidebar so the main workflow stays focused.

CLI examples

python -m evidence_drift.cli ask \
  "How did USPSTF guidance for women in their forties change from 2016 to 2024?"

python -m evidence_drift.cli ask \
  "What screening should a patient with a PALB2 pathogenic variant receive?"

Compare modes:

python -m evidence_drift.cli ask "What does ACS recommend at age 48?" --mode bm25
python -m evidence_drift.cli ask "What does ACS recommend at age 48?" --mode semantic
python -m evidence_drift.cli ask "What does ACS recommend at age 48?" --mode hybrid

Optional OpenAI synthesis

Retrieval remains local unless EMBEDDING_PROVIDER=openai is selected.

Install the optional OpenAI integration:

pip install -e ".[openai]"

Then set an explicit model available in your account:

ENABLE_GENERATION=true
OPENAI_API_KEY=your_key_here
OPENAI_GENERATION_MODEL=your_available_model_name

There is intentionally no default generation model. This prevents the repository from failing because a hard-coded model is unavailable to the user.

The generation prompt instructs the model to:

• use only supplied excerpts
• preserve disagreement
• identify superseded guidance
• distinguish insufficient evidence from recommendation against
• describe source status as corpus metadata
• abstain when the retrieval decision says the corpus cannot support an answer
• cite excerpt IDs

Evaluation outputs

Each evaluation writes:

results/<slice>_<mode>.csv
results/<slice>_<mode>.json

Per-case output includes:

should_abstain
predicted_abstain
abstention_correct
recall_at_k
reciprocal_rank
decision_correct
max_support_score
retrieved_ids
gold_document_ids

See docs/EVALUATION.md for metric definitions.

Repository structure

app.py
pyproject.toml
.env.example

src/evidence_drift/
  cli.py
  config.py
  embeddings.py
  evaluate.py
  generation.py
  index.py
  ingest.py
  models.py
  pipeline.py
  retrieval.py
  scope.py

data/corpus/
  manifest.yaml
  *.md

data/eval/
  canonical.jsonl
  holdout.jsonl

results/snapshots/
tests/
.github/workflows/ci.yml

Limitations

• The corpus contains concise paraphrases, not complete guidelines.
• The benchmark is small and authored by the same person who built the system.
• The holdout is frozen for regression testing but is not independently clinician reviewed.
• The explicit scope guard covers known corpus exclusions and will not capture every possible out-of-domain question.
• The support threshold was calibrated on this small project corpus and should not be transferred to another corpus without validation.
• Retrieval metrics do not measure claim-level citation support or generated-answer correctness.
• The local hashing embedder is a reproducible development baseline, not a production embedding model.
• Current/superseded status requires ongoing source maintenance.

Next steps

The highest-value next step is independent clinician review of the evidence cards and gold labels. After that:

• expand the frozen holdout
• add claim-level citation grading
• test unsupported-consensus and insufficient-evidence errors
• add confidence calibration for abstention
• compare reranking and query decomposition
• track source changes over time

Disclaimer

Reference implementation for clinical RAG evaluation. Not medical advice, not a medical device, not a validated clinical decision-support system, and not intended for diagnosis, screening decisions, or patient care.

License

MIT for the code. Source organizations retain all rights to their original guidance.