Computational Imaging Meets Large Language Models (CIM-LLM)

Radiogenomics Pipeline for Brain Tumor (Glioma) IDH Genotype Classification

CIM-LLM is a comprehensive research pipeline that performs IDH (Isocitrate Dehydrogenase) genotype classification using multiparametric MRI data and GPT-based large language models. The system extracts semantic visual features and quantitative metrics from brain MRI scans to predict tumor molecular characteristics.

Table of Contents

• Features
• Architecture
• Prerequisites
• Installation
• Workflow
  • Step 1: Dataset Preparation
  • Step 2: JSON Feature Extraction
  • Step 3: IDH Classification
• Configuration
• Output Files
• Advanced Usage
• Performance Considerations
• Citation

---

Features

• Automated MRI Feature Extraction: Comprehensive radiological feature extraction from multiparametric MRI (T1, T2, T1CE, FLAIR)
• Atlas-Based Analysis: Location analysis using Harvard-Oxford, Hammers, and Juelich atlases
• Semantic Visual Features: T2-FLAIR mismatch, ring enhancement, tumor morphology, deep gray nuclei involvement
• Quantitative Metrics: Volumetric measurements, sphericity, boundary sharpness, transition zone characteristics
• MNI-152 Registration: Standardized spatial normalization using ANTs
• LLM-Based Classification: IDH mutation status prediction using GPT-4o, GPT-5, Groq, or HuggingFace models
• Multi-Dataset Support: BraTS2021, UCSF-PDGM, EGD datasets
• Tumor Type Classification: Supports astrocytoma, oligodendroglioma, and GBM

---

Architecture

The pipeline consists of two main components:

1. JSON Creation Pipeline (json_creation/)

Note: This folder is currently under review. Updates and changes may occur frequently.

Extracts comprehensive radiological features from MRI data:

• Main script: create_json.py - Orchestrates the entire feature extraction pipeline
• Core class: JSONCreator - Handles MRI preprocessing, registration to MNI-152 space, and feature computation
• Utilities (utils/):
  • visual_attributes_utils.py: T2-FLAIR mismatch metrics, ring enhancement, tumor morphology
  • tumor_proximity_utils.py: Atlas-based location analysis, proximity to eloquent areas
  • make_wm_mask.py: White matter segmentation
  • get_multi_seg.py: Tumor segmentation generation
  • compute_assymetry_index.py: Ventricular asymmetry analysis
  • yaml_utils.py: Configuration file parsing

Key Processing Steps:

1. Image registration to MNI-152 standard space using ANTs
2. Optional N4 bias field correction preprocessing
3. Location feature extraction using Harvard-Oxford and Hammers atlases
4. Semantic visual feature computation (T2-FLAIR mismatch, ring enhancement, etc.)
5. Quantitative metric calculation (volumes, sphericity, boundary sharpness)
6. JSON output generation with lowercase normalized keys

2. API Classification Pipeline (api_code/)

Uses LLM APIs to classify IDH mutation status:

• Main script: idh_classification.py - Batch processes subjects for IDH classification
• Configuration: prompts.py - System and user prompts for radiologist role-playing
• Supported APIs: OpenAI (GPT-4o, GPT-5), Groq, HuggingFace
• Output: Excel files with predicted vs. ground truth IDH labels

Classification Flow:

1. Loads JSON feature files from dataset/{tumor_type}/
2. Retrieves clinical data (age, gender) from master Excel file
3. Sends structured prompt + JSON data to LLM API
4. Parses response for IDH mutation prediction (mutant/wildtype)
5. Saves results to Excel and individual text files

---

Prerequisites

Required Files

• Master Excel File: excel_sheets/master_file_v4.xlsx

  • Contains subject metadata, ground truth labels, and clinical data
  • Required columns: BraTS2021, Local ID, WHO 2021 (original), WHO 2021 (generated), IDH (original), Dataset, Age (years), Gender

• Juelich Atlas Mapping: excel_sheets/juelich_selected_label_mapping.csv

  • Maps Juelich atlas regions to functional categories
  • Required columns: index, name, hemisphere, categories

• Configuration File: config.yaml

  • Contains parameters for feature extraction
  • Located in project root

• API Keys: .env file

  OPENAI_API_KEY=your_openai_key_here
  GROQ_API_KEY=your_groq_key_here

Required Dependencies

Core libraries:

• Medical imaging: nibabel, antspyx, antspynet, nilearn
• Scientific computing: numpy, scipy, pandas
• Image processing: scikit-image, trimesh
• Deep learning: transformers (for HuggingFace models)
• LLM APIs: openai, groq
• Utilities: python-dotenv, pyyaml, requests, openpyxl

---

Installation

# 1. Clone repository
git clone https://github.com/your-repo/Computational-GPT
cd Computational-GPT

# 2. Install dependencies
pip install -r requirements.txt

# 3. Create .env file
cat > .env << EOF
OPENAI_API_KEY=your_key_here
GROQ_API_KEY=your_key_here
EOF

# 4. Verify required files exist
ls excel_sheets/master_file_v4.xlsx
ls excel_sheets/juelich_selected_label_mapping.csv
ls config.yaml

---

Workflow

The pipeline consists of three main steps that must be executed in order:

1. Dataset Preparation  →  2. JSON Feature Extraction  →  3. IDH Classification
   (dataset_preparation.py)    (create_json.py)              (idh_classification.py)

Step 1: Dataset Preparation

Script: dataset/dataset_preparation.py

Purpose: Organizes raw MRI data into the required folder structure by tumor type.

Input Requirements

• Raw MRI data from multiple sources:
  • BraTS2021 Training Data
  • BraTS2021 Validation Data
  • UCSF-PDGM Dataset
  • EGD Dataset
• Master Excel file with subject metadata

Output Structure

dataset/
├── astrocytoma/
│   ├── {subject_id}/
│   │   ├── {subject_id}_t1.nii.gz
│   │   ├── {subject_id}_t2.nii.gz
│   │   ├── {subject_id}_t1ce.nii.gz
│   │   ├── {subject_id}_flair.nii.gz
│   │   └── {subject_id}_seg.nii.gz
├── oligodendroglioma/
│   └── {subject_id}/...
└── gbm/
    └── {subject_id}/...

Usage

cd dataset
python dataset_preparation.py \
  --brats-train /path/to/BraTS2021/TrainingData \
  --brats-val /path/to/BraTS2021/ValidationData \
  --ucsf-path /path/to/UCSF-PDGM/Images \
  --egd-path /path/to/EGD \
  --excel-path ../excel_sheets/master_file_v4.xlsx \
  --out-dir .

What It Does

1. Reads subject IDs from master Excel file
2. Categorizes subjects by WHO 2021 tumor type
3. Locates source data from appropriate dataset
4. Copies and renames files to standardized format
5. Creates three tumor type subfolders in dataset/

---

Step 2: JSON Feature Extraction

Script: json_creation/create_json.py

Purpose: Extracts comprehensive radiological features from MRI data and saves as JSON.

Input Requirements

• Organized dataset from Step 1
• config.yaml with analysis parameters
• Atlas files (downloaded automatically if needed)

Output Structure

Each subject folder gets a JSON file:

dataset/
├── astrocytoma/
│   └── {subject_id}/
│       ├── {subject_id}_*.nii.gz
│       └── {subject_id}.json  ← NEW

JSON Structure

{
  "clinical_data": {
    "age (years)": X,
    "gender": "..."
  },
  "semantic_visual": {
    "flair: tumor core suppressed": true/false,
    "flair: rim hyperintense": true/false,
    "hollowness": X,
    "rim-core adjacency": X,
    "deep gray nuclei involvement": {...},
    "bilateral frontal involvement": true/false,
    "location features": [...],
    "mass effect metrics": {...}
  },
  "quantitative": {
    "volumes (ml)": {...},
    "fractions": {...},
    "boundary sharpness metrics (intensity/mm)": {...},
    "sphericity metrics (0-1)": {...}
  }
}

Usage

# From project root
python json_creation/create_json.py

# Process specific tumor type
# Edit create_json.py main() call:
main(data_dir='dataset', tumor_type='astrocytoma', ...)

# Process specific subject
main(data_dir='dataset', target_subject='BraTS2021_00001', ...)

What It Does

1. Image Registration: Registers all MRI sequences to MNI-152 standard space
2. Optional Preprocessing: Applies N4 bias field correction (configurable)
3. Location Analysis:
   • Harvard-Oxford atlas overlap (subcortical regions)
   • Juelich atlas distances (eloquent areas)
   • Hammers atlas overlap (lobes)
   • Midline crossing and shift metrics
4. Semantic Visual Features:
   • T2-FLAIR mismatch detection
   • Ring enhancement scoring
   • Deep gray nuclei involvement
   • Bilateral frontal involvement
5. Quantitative Metrics:
   • Volumetric measurements
   • Morphological features (sphericity, boundary sharpness)
   • Transition zone characteristics
6. JSON Output: Saves all features with lowercase normalized keys

Important Implementation Details

MRI Processing:

• All images are registered to MNI-152 standard space using rigid transformation
• Segmentation labels follow BraTS convention: 1=NCR/NET, 2=ED, 3/4=ET
• White matter masks are created by erosion for reference normalization
• Contralateral hemisphere used as reference for intensity normalization

Feature Extraction:

• T2-FLAIR mismatch uses robust statistics (median, MAD) for z-score normalization
• Location features computed using Harvard-Oxford atlases (subcortical, cortical, insular)
• Lobe overlap uses Hammers atlas
• Deep gray nuclei involvement checks: putamen, caudate, thalamus, pallidum
• Bilateral frontal involvement uses conservative frontal parcel definitions

---

Step 3: IDH Classification

Script: api_code/idh_classification.py

Purpose: Uses LLM APIs to predict IDH mutation status based on extracted features.

Input Requirements

• JSON files from Step 2
• Master Excel file with ground truth labels
• .env file with API keys
• Prompt templates in api_code/prompts.py

Output Structure

output/
└── idh_classification/
    └── {tumor_type}/
        ├── idh_classification_{tumor_type}_{model}.xlsx  ← Master results
        └── gpt_outputs/  (or groq_outputs/, huggingface_outputs/)
            └── {subject_id}_idh_classification_{model}.txt  ← Individual responses

Usage

# Edit idh_classification.py __main__ section to configure:
api = 'openai'  # or 'groq', 'huggingface'
who_subtype = 'gbm'  # or 'astrocytoma', 'oligodendroglioma'
model_id = 'gpt-4o'  # or 'gpt-5-chat-latest'

# Run from project root
python api_code/idh_classification.py

What It Does

1. Load Configuration:
   • Reads master Excel file for ground truth labels and clinical data
   • Loads API keys from .env
   • Gets prompt templates
2. For Each Subject:
   • Loads JSON feature file
   • Retrieves clinical data (age, gender) from master Excel
   • Combines clinical + radiological features
   • (Optional) Removes specific feature categories for ablation studies
3. API Call:
   • Sends system prompt + user prompt + JSON data to LLM
   • Parses response for IDH prediction (mutant/wildtype)
   • Saves full response as text file
4. Result Aggregation:
   • Compiles predictions into Excel file
   • Saves incrementally after each subject (prevents data loss)
   • Compares predicted vs. ground truth labels

Excel Output Format

Columns:

• Subject ID: Subject identifier
• Dataset: Data source (BraTS2021, UCSF, EGD)
• IDH (original): Ground truth (1=mutant, 0=wildtype)
• IDH (predicted): Model prediction (1=mutant, 0=wildtype, -1=unknown)

LLM Classification Details

• System prompt establishes radiologist expert persona
• JSON features presented to model for IDH classification
• Response parsed using regex to extract bold prediction (**IDH mutant** or **IDH wildtype**)
• Results saved incrementally after each subject to prevent data loss
• Output directory structure: output/idh_classification/{who_subtype}/{api}_outputs/

---

Configuration

JSON Creation (config.yaml)

Required parameters:

• general_parameters: cache_dir, atlas_dir, resolution_mm, spacing
• location_parameters: registration_speed, midline_parameters, overlap_parameters
• t2_flair_mismatch_parameters: thresholds for FLAIR suppression detection
• ring_enhancement_parameters: rim adjacency threshold
• ventricle_compression_parameters: include_inferior_horns flag
• patchiness_parameters: min_component_size, connectivity
• transition_zone_parameters: distance sampling parameters

IDH Classification (.env)

Required API keys:

OPENAI_API_KEY=your_key_here
GROQ_API_KEY=your_key_here

Also needs master Excel file at excel_sheets/master_file_v4.xlsx with columns:

• BraTS2021 or Local ID
• Dataset
• IDH (original): 'mutated' or 'wildtype'
• Age (years)
• Gender

---

Output Files

JSON Feature Files

Located in subject directories: dataset/{tumor_type}/{subject_id}/{subject_id}.json

Classification Results

• Excel file: output/idh_classification/{who_subtype}/idh_classification_{who_subtype}_{model}.xlsx
• Individual text files: output/idh_classification/{who_subtype}/{api}_outputs/{subject_id}_idh_classification_{model}.txt

---

Advanced Usage

Feature Ablation Studies

Edit api_code/idh_classification.py to remove feature categories:

# Uncomment to remove specific features
json_data['semantic_visual'].pop("flair: tumor core suppressed", None)
json_data['semantic_visual'].pop("location features", None)
json_data['quantitative'].pop("volumes (ml)", None)

Modifying Classification Prompts

Edit api_code/prompts.py to change system/user prompts:

from prompts import get_prompt, set_prompt
prompts = get_prompt()
set_prompt('user_prompt_2', 'your new prompt here')

Processing Specific Subjects

# In create_json.py
main(data_dir='dataset', target_subject='BraTS2021_00001')

# In idh_classification.py
# Edit folder_list filtering:
folder_list = [f.name for f in input_folder.iterdir() 
               if f.is_dir() and f.name.startswith('BraTS2021_000')]

Adding New Features

1. Implement extraction function in appropriate utils/ module
2. Call from JSONCreator._extract_semantic_visual_features() or compute_all_features()
3. Add configuration parameters to YAML if needed

Changing Models

# In idh_classification.py
api = 'groq'  # Switch API provider
model_id = 'llama-3.1-70b-versatile'  # Change model

---

Performance Considerations

Processing Time

• Dataset Preparation: ~5-10 minutes for 1000 subjects (I/O bound)
• JSON Creation: ~5-10 minutes per subject (CPU/memory intensive)
  • Depends on registration speed setting
  • Uses N4 correction if enabled
• IDH Classification: ~10-30 seconds per subject (network bound)
  • Depends on API response time
  • Cached responses are instant

Disk Space

• Input Data: ~500 MB per subject (5 MRI sequences)
• Cache Directory: ~200 MB per subject (MNI-registered images)
• JSON Files: ~50 KB per subject
• Output: ~2 KB per text response

Parallelization

• JSON creation can be parallelized by tumor type
• API calls are sequential to avoid rate limits
• Consider using screen or tmux for long-running jobs

Clean Cache

# Remove MNI-registered images
rm -rf cache/*

# JSON files will need to be regenerated

---

Complete Workflow Example

# Step 1: Prepare dataset
cd dataset
python dataset_preparation.py \
  --brats-train /data/BraTS2021/TrainingData \
  --brats-val /data/BraTS2021/ValidationData \
  --ucsf-path /data/UCSF-PDGM/Images \
  --egd-path /data/EGD
cd ..

# Step 2: Extract features (may take several hours)
python json_creation/create_json.py

# Step 3: Run classification
python api_code/idh_classification.py

Expected Output

dataset/
├── astrocytoma/{subjects with JSON files}
├── oligodendroglioma/{subjects with JSON files}
└── gbm/{subjects with JSON files}

output/
└── idh_classification/
    ├── astrocytoma/
    ├── oligodendroglioma/
    └── gbm/

---

Maintenance

Adding New Subjects

1. Add entry to excel_sheets/master_file_v4.xlsx
2. Run dataset_preparation.py
3. Run create_json.py with target_subject parameter
4. Run idh_classification.py

Updating Prompts

Edit api_code/prompts.py:

from prompts import set_prompt
set_prompt('user_prompt_1', 'your new prompt here')

Changing Configuration

Edit config.yaml and re-run create_json.py for affected subjects

---

Citation

If you use this pipeline, please cite:

@article{mahmood2025computational,
  title={Computational Imaging Meets LLMs: Zero-Shot IDH Mutation Prediction in Brain Gliomas},
  author={Mahmood, Syed Muqeem and Mohy-ud-Din, Hassan},
  journal={arXiv preprint arXiv:2511.03376},
  year={2025}
}