Benchmarking Foundation Models for Echocardiogram Analysis

Assessing Video Foundation Models for Mitral Regurgitation Severity Classification

MICCAI ASMUS 2026

This repository provides a comprehensive benchmarking framework for evaluating video foundation models on mitral regurgitation (MR) severity classification from echocardiogram videos. We systematically compare five model families — spanning supervised 3D CNNs, CLIP-style multimodal models, masked autoencoders, and JEPA-based self-supervised approaches — across multiple evaluation protocols including grid-search hold-out, nested cross-validation, and multi-task adaptation.

Models Benchmarked

Model	Architecture	Pretraining	Modality	Original Repo
3D CNN (R2+1D / Swin3D)	Supervised 3D convolutions	Kinetics-400	Video	—
EchoCLIP	CLIP-finetuned ViT	>1M echo image-text pairs	Image + Text	echonet/echo_CLIP
EchoFM	Video MAE (ViT)	290K echocardiography clips	Video	SekeunKim/EchoFM
V-JEPA 2	Video JEPA (ViT-g/L)	Millions of natural videos	Video	facebookresearch/vjepa2
EchoJEPA	V-JEPA 2 adapted to echo	Natural videos + echo fine-tuning	Video	bowang-lab/EchoJEPA

All models are evaluated using frozen encoders + probe heads (linear, MLP, or attentive) to ensure fair comparison across architectures.

Repository Structure

mr-foundation-assessment/
├── assets/                         # Paper figures and assets
├── models/                         # Per-model training & inference
│   ├── 3dcnn/                      # 3D CNN baselines (R2+1D, Swin3D)
│   ├── echoclip/                   # EchoCLIP (CLIP-based)
│   ├── echofm/                     # EchoFM (Video MAE)
│   ├── vjepa2/                     # V-JEPA 2 & V-JEPA 2.1
│   └── echojepa/                   # EchoJEPA (V-JEPA adapted for echo)
├── eval/                           # Unified evaluation framework
│   ├── evaluate_unified.py         # Main evaluation entry point
│   ├── evaluate_cv.py              # Cross-validation evaluation
│   ├── shared_evaluation.py        # Shared metrics, plots, aggregation
│   ├── batch_evaluate.py           # Batch evaluation across model outputs
│   ├── batch_bootstrap.py          # Bootstrap confidence intervals
│   ├── combine_metrics.py          # Combine per-model metrics
│   ├── combine_metrics_cv.py       # Combine CV metrics
│   ├── plot_multi_roc.py           # Multi-model ROC comparison plots
│   └── replot_umap.py             # UMAP visualization
├── scripts/                        # Utility scripts
│   ├── cv_split.py                 # Generate cross-validation folds
│   └── visualize_config.py         # Visualize YAML configs
└── README.md

Evaluation Protocol

Tasks

• 4-class MR severity classification: Normal / Mild / Moderate / Severe
• 2-class clinical significance: Non-significant (0/1) vs Significant (2/3)

Approaches

1. Grid-search hold-out: Train on 70% of subjects, validate on 15%, test on 15%. Multiple probe heads (linear, MLP, attentive) with hyperparameter search.
2. 5-fold nested cross-validation: Inner loop selects best probe configuration; outer loop evaluates on held-out fold. Patient-level splits ensure no data leakage.

Metrics

All evaluations report per-class and aggregated metrics at both video-level and subject-level:

• Accuracy, Precision (PPV), Recall/Sensitivity, F1-score, Specificity, NPV, ROC AUC
• Clinical-significance binary metrics for 4-class tasks
• Bootstrap confidence intervals for robust comparison

Quick Start

1. Choose a Model

Each model directory contains a README.md with detailed setup and training instructions:

• 3D CNN — PyTorch Lightning-based training
• EchoCLIP — HuggingFace Hub weights, probe training
• EchoFM — MAE-based, DDP-supported probe training
• V-JEPA 2 — JEPA probes via evals framework
• EchoJEPA — JEPA probes with multi-task support

2. Train Probes

Each model follows the same paradigm: freeze the foundation encoder, train a lightweight probe head. See model READMEs for specific commands.

3. Run Inference

After training, run inference to produce unified prediction JSONs:

# Single checkpoint inference (example for vjepa2/echojepa)
python models/<model>/scripts/inference.py \
    --eval_config configs/eval/vitg-384/miracle_grid_4class.yaml \
    --probe_checkpoint /path/to/checkpoint.pt \
    --output_dir ./predictions/

# Cross-validation inference
python models/<model>/scripts/inference_cv.py \
    --eval_config configs/eval/vitg-384/miracle_crossval_4class.yaml \
    --cv_checkpoint_dir /path/to/cv_outputs/ \
    --output_dir ./cv_predictions/

4. Evaluate

Use the unified evaluation framework:

# Single predictions
python eval/evaluate_unified.py \
    --predictions ./predictions/unified_predictions.json \
    --output_dir ./eval_results/ \
    --class_names "Normal,Mild,Moderate,Severe"

# Cross-validation
python eval/evaluate_cv.py \
    --predictions_dir ./cv_predictions/ \
    --output_dir ./eval_cv_results/ \
    --class_names "Normal,Mild,Moderate,Severe"

Data Format

Input CSV

All model training scripts expect CSV files with:

• video_filename or video_path — path to the video file
• label — integer class label (0-indexed)
• patient_id — subject identifier for aggregation

For cross-validation, an additional fold_id column (0 to N-1) is required.

Unified Prediction JSON

The evaluation framework expects a common format:

[
  {
    "subject_id": "patient_001",
    "video_id": "videos/study_001.mp4",
    "true_label": 2,
    "probs": [0.05, 0.10, 0.60, 0.25]
  }
]

Key Findings

• EchoJEPA (V-JEPA adapted to echocardiography) consistently outperforms other models on both 4-class and clinical-significance tasks
• Attentive probes significantly improve over linear/MLP heads, especially for larger models
• Training probes on multiple related tasks (MR severity + LVEF + RVSP) improves generalization
• Cross-validation reveals substantial variation across patient splits; reporting mean±std is essential
• Video foundation models pretrained on natural videos (V-JEPA 2) transfer surprisingly well to medical domains

Citation

If you use this benchmark or find it useful, please cite our paper:

Will be released soon

License

This repository is released for non-commercial, academic research purposes. Individual models may have their own licenses — see each model's README for details.