Chest-VLM

Fine-tune MedGemma 1.5 with QLoRA on the Indiana University Chest X-ray Collection (OpenI) to extract structured findings from dual-view images, then generate radiology report prose with a frozen LLM — with a full demo workstation and 22 reproducible ablation runs.

Methodology (V1 + V2): documentation/methodology.md · Pipeline: documentation/architecture.md · Ablations: hyperparameter_tuning/RESULTS.md

Disclaimer: Research code only. Not validated for clinical use.

---

At a glance

• Core stack: PyTorch, Transformers, QLoRA/PEFT, MedGemma, RadGraph, Qwen2.5, FastAPI, Next.js, GitHub Actions CI
• Project depth: 22 systematic ablation runs, tracked as parameter deltas from two base configs, with metrics for every run
• Deliverables: dual-view VLM inference, ontology extraction, LLM report generation, radiologist-style demo UI
• Public artifacts: maximehpe/chest-vlm-run12 and maximehpe/chest-vlm-run17 on HuggingFace
• Headline result: study accuracy 0.757 (run 17_weighted_loss_no_study)
• Full technical narrative: documentation/methodology.md

---

Recommended path (V2)

The current system is a two-stage pipeline: MedGemma extracts a structured ontology from dual-view X-rays; a frozen Qwen2.5 LLM turns that into HTML report prose.

flowchart LR
  img[Dual-view 896px X-rays] --> vlm[MedGemma 1.5 QLoRA]
  vlm --> ont["Ontology v=2"]
  ont --> llm[Qwen2.5 LLM]
  llm --> report[HTML report + demo UI]

Loading

Stages: data prep → RadGraph ontology → MedGemma fine-tuning → LLM report generation → demo workstation. Details: documentation/architecture.md.

Quick start

Requirements: Python 3.10+, NVIDIA GPU (≥16 GB VRAM), HuggingFace account with MedGemma access.

git clone https://github.com/MaximeHuppe/VLM-Chest-Xray.git
cd VLM-Chest-Xray
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env          # add HF_TOKEN=hf_...
python tools/hub/download_model.py --run_name 17_weighted_loss_no_study

Download chest X-ray images to data/images/images_normalized/ (see Data setup), then launch the demo:

./demo/run_demo.sh

---

Best results (V2)

Config	Best for	Study Acc	Study F1	Precision	find_macro F1
17_weighted_loss_no_study (recommended)	Balanced overall	0.757	0.735	0.671	0.223
12_medgemma_ontology_study_loss	Safety / precision	0.749	0.677	0.724	0.228

Run 17 maximises accuracy and study F1 with a simpler loss (no study auxiliary head). Run 12 trades recall for precision — fewer missed pathologies when calling a study Normal (34 false normals vs 56 in run 17). Full tables: hyperparameter_tuning/RESULTS.md.

---

Project evolution (V1 → V2)

This repository documents a deliberate architectural progression:

• Phase 1 (runs 01–06): PaliGemma 2 3B single-stage baseline — direct <Tags> + <Report> generation from a concatenated dual-view canvas. Deep methodology: documentation/methodology.md.
• Phase 2 (runs 08–22): MedGemma 1.5 4B + RadGraph ontology supervision + frozen Qwen2.5 LLM — separates image understanding from language fluency for better structure and safety.
• 22 ablations across backbone, input format, loss, sampling, and regularization. Full V1/V2 comparison: documentation/methodology.md.

V1 remains fully reproducible — every run resolves via hyperparameter_tuning/ablations.yaml, e.g. python script/train.py --run_name 01_baseline.

---

Testing

pytest -m "not gpu" --tb=short

Fast unit tests cover adapter logic, ontology pipeline, config parsing, report generation, and demo API endpoints — no GPU or HF_TOKEN required. GPU-marked tests (pytest -m gpu) run a live MedGemma processor smoke test when HF_TOKEN is set.

CI runs on every push/PR: CPU torch + requirements-ci.txt, unit tests (including demo API HTTP tests), frontend build, and slim Docker backend build. See .github/workflows/ci.yml. Maintainer notes: documentation/contributing.md.

Data setup

Tracked in git: data/indiana_reports.csv, data/indiana_projections.csv, and derived CSVs under data/ and data/temp/. These are derived from public Indiana OpenI metadata (reports and projections), not raw DICOM or patient identifiers. See Data attribution below.

Not tracked (see .gitignore): raw/normalized PNGs under data/images/. Download the Indiana OpenI images and place them at data/images/images_normalized/, then run the preprocessing pipeline:

python script/generate_dataset.py
python script/generate_ontology.py --export-only   # structured ontology targets, parallel pipeline

Full instructions (folder layout, dataset CSV variants, optional V2 stages): documentation/data-preparation.md and documentation/ontology-pipeline.md.

Downloading model weights

Model adapters for the best runs are hosted on HuggingFace Hub. Download before running inference or the demo:

python tools/hub/download_model.py --run_name 17_weighted_loss_no_study
python tools/hub/download_model.py --run_name 12_medgemma_ontology_study_loss

# Explicit Hub ID (without touching ablations.yaml):
python tools/hub/download_model.py --hub_id maximehpe/chest-vlm-run17 --run_name 17_weighted_loss_no_study

The adapter lands in hyperparameter_tuning/runs/{run_name}/checkpoints/ and is picked up automatically by infer.py and the demo backend.

Note: HF_TOKEN is required to download the base MedGemma model (gated), but the LoRA adapter repos can be public.

Running an experiment

Every tracked run is launched by name — its full config resolves from a base config + the run's overrides in hyperparameter_tuning/ablations.yaml:

python script/train.py    --run_name 17_weighted_loss_no_study
python script/infer.py    --run_name 17_weighted_loss_no_study
python script/evaluate.py --run_name 17_weighted_loss_no_study

Or use the shell helper (edit RUN_NAME in tools/run_train.sh):

bash tools/run_train.sh

Full config-resolution mechanics, config anatomy, and how to add a new ablation: documentation/training.md. Inference/evaluation flags and metric definitions: documentation/inference-and-evaluation.md.

Outputs per run

Path	Contents
hyperparameter_tuning/runs/{run_name}/training.yaml	Frozen, fully-resolved training config
hyperparameter_tuning/runs/{run_name}/evaluation.yaml	Frozen, fully-resolved eval config
hyperparameter_tuning/runs/{run_name}/checkpoints/	LoRA adapter weights (weights gitignored; TensorBoard logs kept)
hyperparameter_tuning/runs/{run_name}/results/prediction.csv	Test-set generations (gitignored; local/regenerated)
hyperparameter_tuning/runs/{run_name}/results/metrics.json	Evaluation metrics
hyperparameter_tuning/runs/{run_name}/results/tb_scalars.csv	TensorBoard scalars exported to CSV

Export TensorBoard scalars and view live curves:

python tools/export_tb_logs.py --run_name 17_weighted_loss_no_study
tensorboard --logdir hyperparameter_tuning/runs/<run_name>/checkpoints/logs

Full ablation tables (runs 01–22), per-run interpretation, and best-config trade-offs: hyperparameter_tuning/RESULTS.md. Precomputed metrics.json / tb_scalars.csv for completed runs are under hyperparameter_tuning/runs/. Demo replay predictions for runs 12 and 17 live under demo/predictions/.

Citation

If you use this code or models, please cite:

@software{huppe2026chestvlm,
  author  = {Huppe, Maxime},
  title   = {Clinically Accurate Chest X-Ray Report Generation Using Lightweight Vision-Language Models},
  year    = {2026},
  url     = {https://github.com/MaximeHuppe/VLM-Chest-Xray},
  note    = {See documentation/methodology.md for the full project arc and hyperparameter_tuning/RESULTS.md for the ablation study.}
}

Machine-readable metadata: CITATION.cff.

Data attribution

Chest X-ray images and the original reports come from the Indiana University Chest X-ray Collection (OpenI), distributed on Kaggle. This repository includes processed CSVs and derived training targets built from that public corpus; it does not redistribute the full image archive.

When using the dataset or reproducing results, cite OpenI appropriately and comply with the source terms. Model weights on Hugging Face (maximehpe/chest-vlm-run12, maximehpe/chest-vlm-run17) are research artifacts built on top of gated base models (MedGemma, PaliGemma 2); accept those licenses before download.

License

Released under the Apache License 2.0. Not for clinical use.

Repository structure

Chest-VLM/
├── README.md                  ← you are here
├── LICENSE                    Apache 2.0
├── CITATION.cff                Citation metadata
├── documentation/              → documentation/README.md (task-oriented guides index)
│   ├── data-preparation.md     Downloading images, generate_dataset.py, CSV variants
│   ├── ontology-pipeline.md    RadGraph extraction, <Ontology v=2> targets
│   ├── training.md             Launching runs, ablations.yaml config resolution
│   ├── inference-and-evaluation.md   infer.py / evaluate.py, metric regimes
│   ├── architecture.md         End-to-end pipeline flowcharts and narrative
│   ├── methodology.md          V1 + V2 design decisions (canonical)
│   ├── medgemma-integration.md PaliGemma → MedGemma adapter migration reference
│   ├── demo-app.md             Demo workstation guide
│   ├── contributing.md         PR workflow + branch protection
│   └── assets/                 Figures referenced from the guides above
├── .github/workflows/          CI (ci.yml)
├── tests/                      pytest suite
├── config/                     → config/README.md (V1/V2 base configs, data/ontology templates)
│   └── ontology/                Ontology pipeline config + vocabulary JSON
├── data/                       → data/README.md (CSVs; images downloaded separately)
├── hyperparameter_tuning/      → hyperparameter_tuning/README.md
│   ├── ablations.yaml           Single source of truth: per-run parameter deltas
│   ├── RESULTS.md                Full ablation analysis
│   └── runs/{run_name}/          Frozen configs, checkpoints, results per run
├── demo/                        Full-stack demo (FastAPI + Next.js)
│   └── predictions/             Curated replay CSVs for runs 12 and 17
├── notebook/                    demo_one_case.ipynb
├── script/                      → script/README.md (5 pipeline CLIs)
├── tools/                       → tools/README.md
└── src/                         → src/README.md