Derma MedGemma — Dermatology Vision-Language Fine-Tuning

Fine-tuning MedGemma 1.5 4B for dermatological image diagnosis using a custom dataset on Hugging Face Hub.

Source Paper

The dataset is derived from DermaCon-IN (derma-dataset.pdf): DermaCon-IN: A Multi-concept Annotated Dermatological Image Dataset of Indian Skin Disorders for Clinical AI Research (NeurIPS 2025 Datasets and Benchmarks).

Overview

This project trains a vision-language model to analyze skin images and produce structured diagnoses. The dataset combines dermatological images with rich metadata (age, sex, body part, skin tone, clinical descriptors) and ground-truth labels. Prompts are built on-the-fly from metadata at training time—the dataset stores images and metadata only, not pre-built prompts.

Dataset

Property	Value
Hugging Face	sharl-008/derma-dataset
Train	~4.36k samples
Test	~1.09k samples
Modalities	Image, Text (metadata)

Schema

• Image — Dermatological image
• Metadata — Age, Body_part, Descriptors, Confidence, Disease_label, Fitzpatrick, Gradability, Image_name, Main_class, Monk_skin_tone, Quality, Sex, Sub_class, Subject_ID

Workflow

Run the notebooks in order:

1. dataset_curation.ipynb — Consolidates images with derma_data.csv, stratified 80/20 train/test split, uploads to sharl-008/derma-dataset.
2. model_training.ipynb — Unsloth-assisted LLM fine-tuning with images: loads dataset from Hub, builds prompts from metadata, fine-tunes MedGemma 1.5 4B (4-bit).

Project Structure

derma_med_gemma/
├── model_training.ipynb      # Unsloth-assisted LLM fine-tuning with images
├── dataset_curation.ipynb    # Dataset consolidation & HF upload
├── derma_data.csv            # Metadata (age, body_part, descriptors, disease labels, etc.)
├── derma-dataset.pdf         # DermaCon-IN research paper (source)
└── README.md

Model Output Format

Each sample is a conversation:

• User: Prompt (patient info, descriptors) + image
• Assistant: Diagnosis, Main class, Sub-class, Confidence

Example response:

Diagnosis: Steroid Modified Tinea
Main Classification: Infectious Disorders
Sub-Classification: Infectious skin conditions -Fungal
Confidence: 5/5

Usage

1. Download the datasets as mentioned in the paper (derma-dataset.pdf).
2. Ensure python 3.10+ is installed and a virtual environment is activated.
3. Run dataset_curation.ipynb.
4. Run model_training.ipynb.

Contribution

Contributions are welcome. Open an issue or submit a pull request for bug fixes, dataset improvements, or model enhancements.