Skip to content

RESULTS.md

Latest commit

 

History

History
472 lines (329 loc) · 30.6 KB

RESULTS.md

File metadata and controls

472 lines (329 loc) · 30.6 KB
Raw

Chest-VLM — Experiment Results (V1 + V2)

Research code only. Not validated for clinical use.

Executive summary

  • Best overall (V2): run 17_weighted_loss_no_study — study accuracy 0.757, study F1 0.735
  • Best precision (V2): run 12_medgemma_ontology_study_loss — precision 0.724, fewer false normals
  • Key lesson: weighted ontology token loss + MedGemma dual-view inputs outperform the PaliGemma single-stage baseline; study auxiliary loss trades recall for precision
  • Scope: 22 ablation runs (V1 runs 01–06, V2 runs 08–22); full design narrative in documentation/methodology.md

This document describes the V2 phase of the project: changes relative to the PaliGemma baseline (runs 01–06), the two-stage VLM + LLM pipeline, all ablation results, and an interpretation of what worked and what did not.

See also: documentation/methodology.md (V1 + V2 narrative) · documentation/architecture.md (pipeline flowcharts) · ablations.yaml (run parameter deltas) · README.md (folder index)

Run index

Each run stores artifacts under hyperparameter_tuning/runs/{run_name}/results/ (metrics.json, tb_scalars.csv). Fresh inference still writes prediction.csv locally (gitignored). Demo replay uses the curated copies under demo/predictions/.

Run run_name Base Key change
01 01_baseline paligemma PaliGemma baseline
02 02_augmentation_weighted_sampling paligemma + augmentation + weighted sampling
03 03_regularized_weighted_alignment paligemma + cosine LR, warmup, weight decay
04 04_full_vision_alignment paligemma + LoRA on SigLIP layers
05 05_frontal_only paligemma Frontal view only
06 06_zero_entropy_normals paligemma Canonical normal reports
08 08_medgemma_dual_view medgemma MedGemma, old tags+report format
09 09_medgemma_dual_view_ontology medgemma Structured ontology targets
10 10_medgemma_dual_view_ontology_present medgemma Absent findings removed
11 11_medgemma_ontology_weighted_loss medgemma Weighted token loss
12 12_medgemma_ontology_study_loss medgemma + study auxiliary loss — best precision
13 13_no_early_stop medgemma No early stopping
14 14_normal_boost_unweighted medgemma normal_weight=3.0
15 15_study_loss_normal_boost medgemma normal_weight=3.0 + weighted sampling
16 16_study_loss_unweighted_sampling medgemma Study loss, no weighted sampling
17 17_weighted_loss_no_study medgemma Weighted loss only — best overall
18 18_study_loss_coef2 medgemma Study loss coef=2.0
19 19_low_lr_8ep medgemma Low LR, 8 epochs
20 20_lora_r16 medgemma LoRA rank 16
21 21_label_smoothing medgemma Label smoothing ε=0.05
22 22_structure_weight_03 medgemma Structure weight 0.3 (eval pending)

Every run's exact parameter deltas from its base config are in ablations.yaml; see documentation/training.md for how to reproduce any run with python script/train.py --run_name <name>.

Table of contents

  1. What changed from V1 to V2
  2. New pipeline architecture
  3. New target format: structured ontology
  4. New evaluation metrics
  5. Results — full ablation table
  6. Detailed interpretation of each change
  7. Best configurations
  8. Absolute performance and limits
  9. Appendix — which logs to publish

1. What changed from V1 to V2

V1 (runs 01–06) fine-tuned PaliGemma 2 3B to directly produce a <Tags> list and a <Report> free-text paragraph from dual-view chest X-rays. V2 makes four independent changes:

Axis V1 (runs 01–06) V2 (runs 08–22)
Base model PaliGemma 2 3B (google/paligemma2-3b-pt-448) MedGemma 1.5 4B (google/medgemma-1.5-4b-it)
Image resolution 448×448 px per view 896×896 px per view
Image input Single concatenated canvas 896×448 (frontal || lateral side-by-side) sent as one image token Two separate images (frontal + lateral) via MedGemma native multi-image support
Target format <Tags> … <Report> … (flat text) <Ontology v=2> structured blocks
Output pipeline VLM direct output only VLM structured output → LLM prose report
Loss function Standard cross-entropy Weighted token loss + optional study auxiliary loss
Evaluation BERTScore + RadGraph on report text Structural ontology metrics (study + finding F1)
max_length 1536 tokens 1024 tokens

Training hyperparameters that are identical in both phases: QLoRA (r=8, α=16), LoRA dropout 0.1, target modules (q/k/v/o/gate/up/down proj + multimodal projector), batch size 1, gradient accumulation 8, learning rate 5e-5, cosine LR scheduler, warmup ratio 0.2, weight decay 0.1, AdamW fused, 5 epochs, augmentation (rotation ±10°, translation 0.1, zoom 0.9–1.0, brightness/contrast 0.2), weighted sampling, bf16, seed 42.

2. New pipeline architecture

V1 — direct generation

Image (frontal + lateral) ──► PaliGemma 2 3B (QLoRA) ──► "<Tags> … <Report> …"

The model generates a comma-separated tag list and a free-text report in a single forward pass. Quality is measured by BERTScore (semantic text similarity) and RadGraph (entity+relation F1 on clinical NLP extractions).

V2 — two-stage VLM + LLM

Image (frontal + lateral) ──► MedGemma 1.5 4B (QLoRA) ──► <Ontology v=2> structured output
                                                                     │
                                                              parse findings
                                                                     │
                                                     Qwen2.5-7B-Instruct (frozen)
                                                                     │
                                                       HTML report + impression paragraph

Stage 1 — VLM (MedGemma, fine-tuned): produces a machine-parseable structured summary of the study: study-level assessment (STUDY=NORMAL|ABNORMAL|INDETERMINATE) and individual <Finding> blocks with NAME, ASSERTION, optional SEVERITY, ORGAN, SIDE.

Stage 2 — LLM (Qwen2.5-7B-Instruct, frozen): receives the parsed finding list as a structured prompt and generates a clinical impression paragraph. The LLM is never trained — only prompted.

Why this design:

  • The VLM specialises in clinical accuracy: identifying what is wrong, where, and how severe. This is the image-understanding task.
  • The LLM specialises in linguistic quality: turning a structured list of findings into fluent, clinical prose. This is a pure language task that large instruction-tuned LLMs already excel at without fine-tuning.
  • The two components can be improved independently. Better VLM finding detection directly improves the report. A better prose LLM can be swapped in without retraining the VLM.
  • Report style and formatting are controlled via the LLM prompt, not the VLM — no retraining needed to change output conventions.

3. New target format: structured ontology

V1 format — <Tags><Report>

<Tags> Cardiomegaly - borderline, Pulmonary Artery - enlarged <Report> Borderline cardiomegaly with an enlarged pulmonary artery. The lungs are clear.
<Tags> Normal <Report> The cardiac silhouette and mediastinum size are within normal limits. No acute cardiopulmonary process.

Flat, comma-separated MeSH-derived labels followed by concatenated findings + impression text.

V2 format — <Ontology v=2>

Normal study:

<Ontology v=2>

STUDY=NORMAL

Abnormal study:

<Ontology v=2>

STUDY=ABNORMAL

<Finding>
NAME=cardiomegaly
ASSERTION=present
SEVERITY=borderline
</Finding>

<Finding>
NAME=pulmonary artery
ASSERTION=present
</Finding>

Each field is explicit and machine-parseable. ASSERTION distinguishes present, possible, and absent findings. Optional attributes (SEVERITY, ORGAN, SIDE) are included when present in the source annotations.

Practical dataset variant used in most V2 runs: dataset_medgemma_ontology_present.csv — the <Finding> blocks with ASSERTION=absent are removed, so the model only needs to enumerate findings that are actually there.

Why the format switch matters

The structured format enables capabilities that are impossible with free text:

  • Per-token loss weighting: XML structure tokens (<Finding>, NAME=, </Finding>, etc.) carry no clinical information — they are grammar, not content. Assigning them a weight of 0.1 vs 1.0 for content value tokens lets the model focus its learning budget on clinical values rather than repeated markup.
  • Auxiliary study loss: the STUDY= value token can be targeted with a dedicated cross-entropy loss and custom class weights, providing a direct gradient signal for the most safety-critical prediction.
  • Structural evaluation: instead of measuring whether the generated text sounds like a radiology report (BERTScore), we can measure whether the model identified the right findings (finding F1). This is a more faithful proxy of clinical correctness.
  • LLM report generation: a structured output is a reliable input for the LLM stage. Free-text generation cannot guarantee parseable structure.

4. New evaluation metrics

Because the output format changed, the evaluation regime changed entirely from V1 to V2. The two metric sets are not numerically comparable.

V1 metrics — tags_report regime

Metric What it measures
accuracy Study-level normal/abnormal classification accuracy
precision / recall / f1_score For the Normal class (normal = positive)
tag_micro_f1 / tag_macro_f1 Overlap of predicted vs reference tag sets
hamming_loss / jaccard_index Multilabel tag agreement
bertscore_f1 Semantic similarity of generated report (distilroberta-base)
radgraph_average_er_f1 Entity + relation F1 on clinical NLP graph (radgraph-xl)

V2 metrics — structured (ontology) regime

Metric What it measures
accuracy Study-level normal/abnormal accuracy (same definition)
precision / recall / f1_score For the Normal class (same definition)
finding_micro_f1 Micro-averaged F1 on (finding_name, assertion) pairs
finding_macro_f1 Mean per-finding-name F1 across top-30 most frequent findings
finding_hamming_loss Fraction of wrong binary finding predictions
finding_jaccard_index Per-sample finding set overlap (IoU)
study_parse_failures Predictions where STUDY= could not be parsed (0 in all runs)

Confusion matrix convention (both regimes): Normal = positive class. TP = correctly predicted Normal, FP = predicted Normal but actually Abnormal (a missed pathology — clinically dangerous), FN = predicted Abnormal but actually Normal (unnecessary flagging), TN = correctly predicted Abnormal.

The test set has 123 normal and 215 abnormal studies (338 total, ~36% normal). This imbalance means a model predicting "Abnormal" for everything would achieve 63.6% accuracy — contextualising the ~75% ceiling seen across runs.

5. Results — full ablation table

V1 — PaliGemma 2 3B, tags_report format

Run Change Acc F1 Prec Recall BERTScore F1 RadGraph avgER
01 Baseline (no aug, linear LR) 0.562 0.598 0.449 0.894 0.380 0.181
02 + augmentation + weighted sampling 0.592 0.606 0.467 0.862 0.433 0.227
03 + cosine LR, warmup, weight decay, LoRA dropout 0.654 0.558 0.521 0.602 0.406 0.193
04 + LoRA on SigLIP vision layers 0.642 0.547 0.507 0.593 0.387 0.184
05 Frontal view only 0.669 0.556 0.543 0.569 0.362 0.158
06 Zero-entropy canonical normals 0.645 0.375 0.522 0.293 0.406 0.305

Run 08 — MedGemma 1.5 4B, old tags_report format (bridge run)

Run Change Acc F1 Prec Recall BERTScore F1 RadGraph avgER
08 MedGemma, same format as V1 0.757 0.680 0.654 0.707 0.423 0.194

V2 — MedGemma 1.5 4B, structured ontology format

Run Change Acc F1 Prec Recall find_micro F1 find_macro F1 find_jaccard
09 Structured ontology targets 0.746 0.739 0.642 0.871 0.256 0.168 0.454
10 Absent findings removed 0.716 0.723 0.607 0.893 0.217 0.167 0.444
11 + weighted token loss 0.663 0.703 0.553 0.964 0.231 0.169 0.479
12 + study auxiliary loss (coef=1.0) 0.749 0.677 0.724 0.636 0.257 0.228 0.396
13 No early stopping 0.731 0.676 0.674 0.679 0.263 0.206 0.404
14 normal_weight=3.0, no weighted sampling 0.740 0.740 0.631 0.893 0.267 0.204 0.482
15 normal_weight=3.0 + weighted sampling 0.737 0.692 0.671 0.714 0.237 0.201 0.402
16 Study loss, no weighted sampling 0.751 0.732 0.661 0.821 0.246 0.192 0.456
17 Weighted token loss only (no study loss) 0.757 0.735 0.671 0.814 0.245 0.223 0.457
18 Study loss coef=2.0 0.722 0.695 0.637 0.764 0.194 0.125 0.395
19 Low LR (2e-5), 8 epochs, no early stop 0.737 0.696 0.667 0.729 0.264 0.224 0.427
20 LoRA r=16 (vs r=8) 0.719 0.667 0.655 0.679 0.240 0.165 0.395
21 Label smoothing (ε=0.05) 0.707 0.680 0.621 0.750 0.203 0.115 0.387
22 Structure weight=0.3 (vs 0.1) — — — — — — —

Bold = best in column across V2 structured runs. Run 22 evaluation pending.

6. Detailed interpretation of each change

6.1 V1 ablations — PaliGemma 2 3B

Run 01 (baseline): The untrained model collapses toward predicting Normal for almost everything (135 of 215 abnormal cases called Normal). Recall is very high (0.894) but precision is catastrophic (0.449). Accuracy of 0.562 is barely above random. The model has not learned to discriminate clinical pathology.

Run 02 (+augmentation, +weighted sampling): Both changes help. BERTScore improves by +5 pts and RadGraph by +4.6 pts — the model generates more clinically relevant text when training data is diversified and class balance is corrected. Study metrics improve marginally.

Run 03 (+cosine LR, +warmup, +weight decay, +LoRA dropout): Regularisation is the most impactful change in the PaliGemma phase. Accuracy rises +6 pts (0.592 → 0.654), and dangerous false normals drop from 121 to 68. The model becomes substantially more conservative: it no longer defaults to Normal for everything. This is the best PaliGemma configuration when accounting for clinical safety, despite a lower F1 than run 02.

Run 04 (+LoRA on SigLIP vision encoder): Expanding LoRA to the vision attention layers regresses slightly. The 3,376-study dataset cannot support the additional capacity. The vision encoder's representations were already adequate without per-layer adaptation.

Run 05 (frontal only): Achieves the highest PaliGemma accuracy (0.669) despite using only one view. This is counterintuitive and suggests that the dual-view 896×448 layout creates spatial complexity that PaliGemma's 448×448-native backbone has difficulty processing. However, RadGraph falls to 0.158 — the lateral view carries clinically useful information for report text even if it slightly confuses the study-level signal.

Run 06 (zero-entropy canonical normals): Standardising all normal reports to a single canonical string teaches the model to write clean, structured normal reports. RadGraph reaches 0.305 — the best text quality in the entire V1 phase. But study recall collapses to 0.293: the model now over-flags normals as abnormal, possibly because canonical target text diverges from varied test descriptions. This run is a useful proof that target text quality matters for generation, not for classification.

PaliGemma phase ceiling: Best study accuracy ~67%, best RadGraph 0.305. Precision for Normal never exceeds 0.55 — roughly 1 in 2 predictions of "Normal" is a missed pathology.

6.2 Model change — PaliGemma → MedGemma (run 08)

This is the single largest improvement across the entire ablation series.

Switching to MedGemma 1.5 4B (keeping the same <Tags><Report> format, same augmentation, same hyperparameters) raises:

  • Accuracy: 0.654 → 0.757 (+10 pts)
  • Study precision: 0.521 → 0.654 (+13 pts)
  • False normals (FP): 68 → 46 (−33%)
  • Study F1: 0.558 → 0.680 (+12 pts)

MedGemma is explicitly pre-trained on medical imaging datasets and clinical text. It arrives at fine-tuning with structural knowledge of chest X-ray vocabulary, pathology terminology, and visual-clinical associations. No training recipe change can substitute for this domain prior on a small dataset like Indiana OpenI.

The gain is entirely due to the pre-trained backbone. All other hyperparameters are identical to run 03.

6.3 Structured ontology format (runs 09–11)

Run 09 (structured targets, all findings): Study F1 reaches 0.739 — the highest in the structured phase — because the model learns to predict an explicit STUDY= value rather than implicitly encoding it in tag vocabulary. The structured format naturally separates the classification signal from the finding descriptions. Finding macro F1 is only 0.168, showing that structured targets alone do not immediately improve per-finding recognition.

Run 10 (absent findings removed): Removing ASSERTION=absent blocks simplifies targets. Counter-intuitively, this slightly hurts: accuracy falls −3 pts and finding F1 falls −4 pts. The absent finding supervision provided useful negative-class examples that helped the model learn which findings should not be mentioned. Without it, the model over-predicts normal (FP rises from 68 to 81).

Run 11 (weighted token loss, structure=0.1, content=1.0): Downweighting XML structure tokens and boosting content value tokens sends the gradient almost entirely to clinical values. The effect is severe: recall jumps to 0.964 (nearly all normals captured), but precision collapses to 0.553 and accuracy falls to 0.663. The model learns to generate STUDY=NORMAL almost unconditionally. This is clinically the worst V2 configuration — 109 of 215 abnormal cases are called Normal.

Conclusion from runs 09–11: Structured targets improve study F1 through explicit STUDY= supervision, but naive weighted loss without additional study-level guidance tips the model toward extreme recall at the expense of precision.

6.4 Study auxiliary loss (run 12) — new V2 baseline

Run 12 adds a dedicated cross-entropy on the STUDY= value tokens, with class weights (abnormal × 2.0, indeterminate × 2.0, normal × 1.0) to penalise pathology misses more strongly. Early stopping is switched from lowest eval loss to best generation-evaluated study accuracy (patience=6).

This is the most clinically important training change in V2:

  • Precision rises to 0.724 — the highest across all structured runs and a major safety improvement.
  • False normals fall to only 34 (vs 109 in run 11, 68 in run 09). The model now rarely calls a pathological study Normal.
  • Finding macro F1 rises to 0.228 — the best across all structured runs. The study loss appears to regularise the rest of the output by grounding it in an explicit clinical judgment.
  • The trade-off is lower recall (0.636): more normal studies are flagged as abnormal than in runs 09–11. This is the clinically preferable error direction (false alarm > missed pathology).

Run 12 defines the new baseline for ablations 13–21.

6.5 Ablations from run 12

Run 13 — no early stopping: Removing early stopping regresses accuracy by −2 pts. The model slightly over-trains on the training distribution. Early stopping with patience=6 was correctly tuned.

Run 14 — normal weight=3.0, no weighted sampling: Boosting the normal class weight in the study loss (from 1.0 to 3.0) while removing dataset-level weighted sampling shifts the model back toward high recall (0.893). Study F1 reaches 0.740, tied for best across structured runs, and finding jaccard reaches 0.482 (best across all structured runs). However, false normals rise to 73 — nearly double run 12. This config is optimal for high-recall scenarios, at the cost of precision.

Run 15 — normal weight=3.0 + weighted sampling: Combining normal weight boosting with weighted dataset sampling overshoots. The two normal-boosting mechanisms partially cancel each other, and finding metrics regress. Study F1 (0.692) is lower than both run 12 and run 14. The mechanisms are redundant and conflict.

Run 16 — study loss, no weighted sampling: Removing dataset-level weighted sampling from run 12 improves both accuracy (0.749 → 0.751) and study F1 (0.677 → 0.732). This confirms that dataset weighted sampling was creating a redundant normal-boosting pressure already handled by the study loss class weights. Without sampling, the model finds a more natural balance: fewer false normals (59 vs 34 in run 12 — more than run 12, but F1 is much better). Finding macro F1 drops to 0.192.

Run 17 — weighted token loss only, no study auxiliary loss: Removing the study auxiliary loss entirely and relying solely on the structure/content token weighting achieves the best accuracy (0.757) and second-best study F1 (0.735) in the structured phase, with finding macro F1 of 0.223 (second only to run 12). This is a remarkably strong result for a simpler objective — no auxiliary loss head, no class weight tuning. The weighted token loss alone provides sufficient differentiation between structure and clinical content. This is one of the two recommended configurations.

Run 18 — study loss coef=2.0: Doubling the study loss coefficient makes the study classification signal dominate the total loss. Finding macro F1 collapses to 0.125 — the worst across all structured runs. The model attends almost exclusively to predicting STUDY= correctly and ignores the detail of individual findings. Accuracy also regresses (0.722). The coef=1.0 in run 12 was already optimally balanced.

Summary of loss ablations:

Configuration Study F1 Precision find_macro F1
No weighted loss, no study loss (run 09) 0.739 0.642 0.168
Weighted loss only (run 11) 0.703 0.553 0.169
Study loss only, coef=1.0 (run 12) 0.677 0.724 0.228
Study loss, coef=2.0 (run 18) 0.695 0.637 0.125
Weighted loss only, no study loss (run 17) 0.735 0.671 0.223
Weighted loss + study loss, coef=1.0 (run 16) 0.732 0.661 0.192

6.6 Hyperparameter ablations

Run 19 — lower LR (2e-5), 8 epochs, no early stop: A slower, longer schedule gives slightly better finding metrics (find_macro F1 = 0.224, close to run 12) but hurts study accuracy (0.737) and F1 (0.696). The default learning rate of 5e-5 with early stopping is better overall. The lower LR may be partially beneficial for finding detection at the cost of coarser study-level classification.

Run 20 — LoRA rank 16 (vs 8): Doubling the LoRA rank hurts across the board. Accuracy falls to 0.719, finding macro F1 to 0.165. The 3,376-study training set cannot support the larger parameter count — the model overfits the training distribution. LoRA rank 8 is correctly sized for this dataset scale.

Run 21 — label smoothing (ε=0.05): Label smoothing consistently harms performance. Accuracy falls to 0.707 and finding macro F1 collapses to 0.115 — the worst across any run, tied with run 18. The structured format requires high token-level precision: the model must generate exact strings like NORMAL, cardiomegaly, borderline. Smoothing these target distributions away from one-hot diffuses the signal for correct finding name generation. Label smoothing is actively harmful for structured generation targets.

7. Best configurations

There is a genuine trade-off between two clinical objectives.

Option A — Maximise safety (minimise missed pathologies): Run 12

model: google/medgemma-1.5-4b-it
dataset: dataset_medgemma_ontology_present.csv
loss_weighting:
  structure_weight: 0.1
  content_weight: 1.0
  study_loss:
    enabled: true
    coef: 1.0
    abnormal_weight: 2.0
    normal_weight: 1.0
    indeterminate_weight: 2.0
metric_for_best_model: eval_gen_study_accuracy
early_stopping_patience: 6
Metric Value
Study accuracy 0.749
Study F1 0.677
Precision (Normal) 0.724
Recall (Normal) 0.636
False normals (FP) 34
finding_macro F1 0.228
finding_micro F1 0.257

The study auxiliary loss explicitly penalises pathology misses. When the model predicts "Normal," it is correct 72.4% of the time — the highest precision across all runs. Only 34 abnormal cases are misclassified as Normal (vs 68+ in earlier runs). Best finding_macro_F1 across all runs.

Option B — Maximise balanced discrimination: Run 17

model: google/medgemma-1.5-4b-it
dataset: dataset_medgemma_ontology_present.csv
loss_weighting:
  structure_weight: 0.1
  content_weight: 1.0
  study_loss:
    enabled: false
metric_for_best_model: eval_gen_study_accuracy
early_stopping_patience: 6
Metric Value
Study accuracy 0.757
Study F1 0.735
Precision (Normal) 0.671
Recall (Normal) 0.814
False normals (FP) 56
finding_macro F1 0.223
finding_micro F1 0.245

Simpler training objective (no auxiliary loss). Highest overall accuracy, strong study F1, and nearly matching finding macro F1. Recommended as the default configuration.

8. Absolute performance and limits

What the numbers mean

Study accuracy ~75%: Promising for a research prototype on a 3,376-study dataset. A clinical AI screening tool typically requires ≥95% sensitivity for pathology detection before deployment. The gap reflects the fundamental constraint of dataset scale, not architectural failure.

Precision 0.671–0.724 for Normal: When the best models call a study "Normal," they are wrong 27–33% of the time — meaning a meaningful fraction of patients with pathology would be incorrectly cleared. This level of precision requires mandatory radiologist review of all predictions before any clinical action.

Finding macro F1 ~0.228 (best): Averaged across the top-30 most frequent findings, the model achieves roughly 23% F1 per finding type. Many specific findings are missed or mislabelled. This reflects the limited training set size: rare findings may appear only a handful of times across 3,376 studies.

Zero study parse failures across all 21 completed runs: The model reliably produces parseable structured output in every case. The structured format has been robustly learned — the output grammar is stable even when clinical content quality varies.

Structural limits of the dataset

  • Scale: 3,376 studies is small for medical AI. Clinical-grade models typically require tens to hundreds of thousands of annotated cases.
  • Single institution: Indiana University OpenI is a historical US archive. Results may not transfer to different scanners, patient populations, or radiology reporting conventions.
  • Annotation quality: Ground-truth tags are MeSH-derived, not clinical expert annotations. Some findings may be incompletely captured or inconsistently labelled across the dataset.

Advantage of the VLM + LLM approach

Despite absolute metric limitations, the two-stage architecture provides a structurally sound framework for scaling:

  • Replacing the VLM (e.g., larger MedGemma or domain-fine-tuned checkpoint) automatically propagates improvements through the full pipeline.
  • The LLM report generation stage is entirely decoupled. As VLM finding detection improves, prose report quality improves without any LLM modification.
  • Structured evaluation metrics (finding F1) now directly measure clinical correctness — future improvements can be tracked meaningfully, unlike BERTScore which rewards fluent but potentially incorrect text.

9. Appendix — which logs to publish

Recommended: track in git

Artifact All runs Notes
hyperparameter_tuning/runs/*/results/metrics.json Yes — all Small JSON, definitive evaluation results
hyperparameter_tuning/ablations.yaml Yes — single file Complete reproducibility of training + evaluation for every run
hyperparameter_tuning/runs/*/training.yaml Yes — all Frozen, fully-resolved copy at training time
hyperparameter_tuning/runs/*/evaluation.yaml Yes — all Frozen, fully-resolved copy at eval time

Optional: track in git (useful for reproducibility)

Artifact Recommendation Notes
demo/predictions/{12,17}_*.csv Yes — demo assets only Curated replay files for the workstation (DEMO_RUN=12|17); not a full experiment archive
TensorBoard events.out.tfevents.* Do not track Binary, numerous, large — not human-readable in git

Not recommended: do not track

Artifact Reason
hyperparameter_tuning/runs/*/results/prediction.csv Regeneratable via infer.py; gitignored. Keep only the two demo copies under demo/predictions/.
TensorBoard log files Binary, large, not git-friendly. Export key plots as PNG instead.
LoRA checkpoint weights Already gitignored; too large for git

Recommended TensorBoard exports (PNG figures) for documentation

If you want visual training curves without binary tfevents files, export the following scalars from TensorBoard for the key runs (08, 12, 17):

  • loss_train and loss_validation — convergence curves
  • eval_gen_study_accuracy — validation study accuracy during training (most informative for V2 runs)
  • study_loss_train / study_loss_validation — auxiliary loss behaviour (runs 12, 15, 16, 18)
  • gen_study_fp_rate / gen_study_fn_rate — precision/recall trade-off during training

For all other runs, metrics.json alone is sufficient documentation of final test performance.