ClinicalBr-Bench

ClinicalBr is the first bilingual benchmark for clinical decision-making built from real Brazilian case reports. The corpus contains 2,892 cases from 36 SciELO medical journals across 18 specialties, structured as parallel Portuguese–English pairs. This repository is the reviewer-facing benchmark package: it lets you (1) reproduce the paper's headline table offline with one command and (2) inspect and re-run the full inference + evaluation pipeline.

Dataset (HuggingFace) · Paper (DOI pending)

Introduction

We study three questions with one benchmark:

1. Cross-lingual gap — do LLMs perform worse in Brazilian Portuguese than in English on the same clinical cases?
2. Open-ended readiness — how far are current models from clinically useful diagnosis, work-up, and treatment planning?
3. Endemic conditions — are Brazil-endemic (tropical) diseases disproportionately affected?

ClinicalBr supports four tasks, each scored as the paper specifies:

#	Task	Input	Metric
1	Final diagnosis	masked case (presentation + exams)	3-judge LLM consensus accuracy
2	Differential diagnosis	masked case	UMLS-CUI F1 + Recall@5
3	Exam recommendation	masked case (presentation only)	UMLS-CUI precision / recall / F1
4	Treatment planning	masked case (diagnosis revealed)	3-judge 1–5 rubric → [0,1] (accuracy, completeness, clarity)

We evaluate four models in both languages: MedGemma-27B, Sabiá-4, DeepSeek-R1, and o3-mini.

Headline findings. Final diagnosis favored English for all four models (+6.5 to +9.9 accuracy points; only the two specialized models survive Holm correction). Differential diagnosis and treatment planning were statistically language-equivalent. Performance stayed far from clinical usefulness (final-diagnosis accuracy peaked at 0.68; differential F1 0.20–0.27; exam F1 < 0.10). Brazilian-endemic conditions scored at least as well as the full corpus.

Files organization

├── assets/                         # paper figures (teaser, headline, results/)
├── data/
│   └── README.md                   # the released HF parquet dataset + schema (data is not vendored)
├── src/
│   ├── Inference/                  # run the four tasks against the masked cases
│   │   ├── llm_client.py           # pluggable backends (OpenAI / Fireworks / Maritaca / Novita / ...)
│   │   ├── diagnosis_inference.py  # task 1+2: final diagnosis + differential
│   │   ├── exam_inference.py       # task 3: exam recommendation
│   │   ├── treatment_inference.py  # task 4: treatment planning
│   │   └── instructions/           # the exact task prompts (pt + en), as .txt
│   └── Evaluation/
│       ├── reproduce_paper.py      # ► offline Table 2 reproduction (no keys, no UMLS)
│       ├── diagnosis_judge.py      # task 1: 3-judge consensus accuracy
│       ├── ddx_eval.py             # task 2: CUI F1 / Recall@5
│       ├── exam_eval.py            # task 3: CUI precision / recall / F1
│       ├── treatment_judge.py      # task 4: 1–5 rubric → [0,1]
│       ├── metrics/                # reusable scoring core (cui_f1, consensus, utils)
│       └── instructions/           # the exact judge + canonicalization prompts
├── pyproject.toml + Makefile       # pinned deps; `infer` extra adds the LLM SDKs
└── README.md

How to use

Requires Python ≥3.11 and uv.

1. Environment setup

uv sync                 # base deps: reproduce + offline scoring (no keys needed)
uv sync --extra infer  

To re-run inference or the live LLM-as-judge, set the relevant API keys in your shell or a git-ignored .env (see src/Inference/llm_client.py for the full list): OPENAI_API_KEY, FIREWORKS_API_KEY, MARITACA_API_KEY, NOVITA_API_KEY, … The UMLS Metathesaurus is not redistributed; the released parquets already carry the linked CUIs, so offline scoring needs no UMLS license.

2. Quick start — reproduce the paper (recommended)

uv run python -m src.Evaluation.reproduce_paper      # downloads the released parquets, prints Table 2

Every cell prints as paper / reproduced for all four models × both languages. Pass --data-dir DIR to score a local clone of the dataset instead of downloading.

3. Inspect any task offline (no API keys)

Each evaluator can re-derive a single Table-2 cell straight from the released results — useful to audit one number end to end:

uv run python -m src.Evaluation.diagnosis_judge --model o3-mini --lang en              # → accuracy 0.681
uv run python -m src.Evaluation.ddx_eval        --model deepseek-r1-fireworks --lang pt # → F1 0.262, Recall@5
uv run python -m src.Evaluation.exam_eval       --model o3-mini --lang en              # → P / R / F1
uv run python -m src.Evaluation.treatment_judge --model sabia-4 --lang pt              # → accuracy / completeness / clarity

Model slugs: medgemma-27b-hf, sabia-4, deepseek-r1-fireworks, o3-mini.

4. Re-run inference (needs --extra infer + API keys + UMLS Account)

uv run python -m src.Inference.diagnosis_inference \
    --model accounts/fireworks/models/deepseek-r1 --provider fireworks \
    --lang pt --run-dir runs/diag_deepseek_pt --max-cases 5

exam_inference and treatment_inference follow the same CLI. Then score your own predictions with the live judges:

uv run python -m src.Evaluation.diagnosis_judge --lang pt \
    --predictions runs/diag_deepseek_pt/predictions.jsonl \
    --out runs/diag_deepseek_pt/judgments.jsonl

Metric definitions

CUI scoring (tasks 2–3) is pure set overlap of predicted vs gold UMLS Concept Unique Identifiers; entity linking (text → CUIs via SapBERT over the UMLS Metathesaurus) is run upstream and embedded in the released results_* parquets. The LLM-judge panel (tasks 1, 4) is {qwen-2.5-72b, llama-3.3-70b, gpt-oss-120b}; the 4th stored judge (gpt-4o) is excluded, matching the paper. See data/README.md for the parquet schema.

CUI linking & canonicalization (upstream, placeholder)

The CUIs scored in tasks 2–3 are produced upstream in two steps, both shipped under src/Evaluation/linking/ so the pipeline is inspectable end to end:

1. canonicalize.py — an LLM rewrites verbose / bundled exam mentions into atomic, standardized names (the same canonicalization the gold side gets), so one mention links to one CUI. Runnable with --extra infer + an API key.
2. link_cuis.py — SapBERT-XLMR encodes each mention and a FAISS cosine index over the UMLS Metathesaurus returns the best CUI (threshold 0.75, TUI-scoped per task). This is a documented placeholder: it ships the real interface and recipe but downloads no embedding model and bundles no UMLS index (UMLS is licensed separately by the NLM; the runnable linker lives in the full research repo). Its embed() / search() raise NotImplementedError.

Neither step is needed to reproduce Table 2 — the released results_* parquets already carry the linked predicted_cuis, and ddx_eval / exam_eval score pure set overlap over them with no keys and no UMLS.

Results

Differential diagnosis (EN–PT forest)	assets/results/ddx_forest.png
Exam recommendation (P / R / F1)	assets/results/exam_pr_f1.png
Inter-judge agreement	assets/results/judge_agreement.png
Brazil-endemic (tropical) analysis	assets/results/tropical.png
Corpus composition (anatomy / disease)	assets/results/corpus_anatomy.png, assets/results/corpus_disease.png

Citation

A BibTeX entry will be added once the ClinicalBr paper has a DOI. For now, please cite the ClinicalBr paper and credit the ClinicalBr authors.

License

• Code — MIT, see LICENSE.
• Benchmark data — CC BY 4.0, see LICENSE-DATA.

Acknowledgements

Built on open-access case reports from SciELO. Differential-diagnosis and exam scoring use UMLS Concept Unique Identifiers; entity linking uses SapBERT over the UMLS Metathesaurus, which must be licensed independently from the U.S. National Library of Medicine.