MedCLIPSeg: Probabilistic Vision–Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari, Berardino Barile, Yiming Xiao†, Hassan Rivaz†
† Co-senior authors
Abstract: Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision–language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages nuanced semantic learning across diverse textual prompts, MedCLIPSeg improves data efficiency and domain generalizability. Extensive experiments across 16 datasets, spanning five imaging modalities and six organs, demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight the local reliability of segmentation results. This work demonstrates the potential of probabilistic vision–language modeling for text-driven medical image segmentation.
Overall architecture of MedCLIPSeg. The framework integrates probabilistic vision–language fusion into a CLIP-based segmentation pipeline.
Schematic illustration of the proposed Probabilistic Vision–Language (PVL) adapters used for bidirectional cross-modal interaction.
-
Bidirectional Vision–Language Fusion: Introduce representation-level fusion modules that enable efficient bidirectional interaction between image and text features while keeping CLIP encoders frozen, improving data efficiency and robustness.
-
Probabilistic Cross-Modal Attention: Model vision–language attention using variational Key–Value formulations to capture uncertainty, leading to improved segmentation accuracy and cross-domain generalization.
-
Pixel-Level Uncertainty Estimation: Generate dense uncertainty maps by sampling attention Values from learned probability distributions, providing intuitive reliability estimates for clinical interpretation.
-
Extensive Multi-Modal Segmentation Evaluation: Conduct comprehensive evaluation against state-of-the-art methods across 5 imaging modalities and 6 organs and 16 datasets, assessing data efficiency, domain generalization, and the contribution of individual model components.
Results reported below show DSC scores (%) for data efficiency and domain generalization evaluation benchmarks across 16 biomedical image segmentation datasets averaged.
| Method | 10% Data | 25% Data | 50% Data | 100% Data |
|---|---|---|---|---|
| UNet | 60.95 | 62.74 | 71.61 | 78.49 |
| UNet++ | 63.72 | 65.86 | 73.15 | 78.44 |
| DeepLabv3 | 61.32 | 65.39 | 68.58 | 73.28 |
| Attention U-Net | 62.78 | 64.97 | 71.34 | 76.30 |
| nnU-Net | 73.45 | 76.73 | 78.86 | 81.40 |
| Swin-UNet | 53.04 | 54.69 | 55.89 | 65.03 |
| TransUNet | 52.69 | 55.25 | 55.22 | 67.22 |
| LViT | 66.51 | 75.66 | 78.88 | 83.35 |
| Ariadne’s Thread | 61.34 | 63.09 | 65.65 | 70.07 |
| EoMT-CLIP | 74.07 | 76.29 | 79.19 | 82.93 |
| CLIPSeg | 74.66 | 78.31 | 79.63 | 84.87 |
| DenseCLIP | 67.84 | 70.23 | 72.09 | 74.19 |
| ZegCLIP | 61.25 | 72.46 | 76.21 | 78.98 |
| SAN | 74.13 | 76.13 | 78.80 | 81.62 |
| MaPLe | 66.27 | 71.53 | 74.60 | 74.60 |
| MaPLe + Decoder | 74.81 | 79.64 | 82.81 | 84.94 |
| VLSM-Adapter | 74.47 | 77.63 | 80.83 | 83.85 |
| CausalCLIPSeg | 71.19 | 75.42 | 78.60 | 81.34 |
| CAT-Seg | 78.76 | 81.12 | 83.32 | 85.90 |
| MedCLIPSeg (Ours) | 81.10 | 85.08 | 87.18 | 88.66 |
| Method | ID | OOD | HM |
|---|---|---|---|
| LViT | 83.31 | 64.99 | 73.02 |
| Ariadne’s Thread | 68.25 | 27.23 | 38.93 |
| CLIPSeg | 84.95 | 69.22 | 76.28 |
| DenseCLIP | 77.69 | 58.11 | 66.49 |
| ZegCLIP | 77.16 | 61.33 | 68.34 |
| SAN | 84.45 | 69.87 | 76.47 |
| MaPLe | 76.55 | 59.30 | 66.83 |
| MaPLe + Decoder | 84.78 | 66.85 | 74.76 |
| VLSM-Adapter | 85.78 | 73.28 | 79.04 |
| CausalCLIPSeg | 81.52 | 53.86 | 64.86 |
| CAT-Seg | 86.10 | 74.57 | 79.92 |
| MedCLIPSeg (Ours) | 89.11 | 79.02 | 83.76 |
Uncertainty peaks along lesion boundaries and remains consistent across diverse datasets, indicating reliable calibration and generalization. In-distribution (ID) data are shown in blue, while out-of-distribution (OOD) data are shown in red.
All the checkpoints can be found on the official Hugging Face repo for the Data Efficiency and Domain Generalization evaluation benchmarks. Take a look here to see how to run and reproduce all the results.
For installation and other package requirements, please follow the instructions detailed in INSTALL.md.
Please follow the instructions at DATASETS.md to prepare all datasets.
Please refer to the RUN.md for detailed instructions on training, evaluating and reproducing the results using our pre-trained models.
If you use our work, please consider citing:
@article{koleilat2026medclipseg,
title={MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation},
author={Koleilat, Taha and Asgariandehkordi, Hojat and Manzari, Omid Nejati and Barile, Berardino and Xiao, Yiming and Rivaz, Hassan},
journal={arXiv preprint arXiv:2602.20423},
year={2026}
}We are grateful to the authors of CLIP, MaPLe, and LViT for making their code publicly available. If you use our model or code, we kindly request that you also consider citing these foundational works.

