OUBS - Open Unlearning Benchmark Suite

Minh Nguyen, Anjani Kumar Avadhanam, Mohana Pravallika Pakala

OUBS is a unified framework for reproducible comparison of unlearning methods across multiple domains and deletion scenarios. The benchmark is designed with modularity and extensibility in mind, enabling both practical evaluation and research-oriented insights.

Overview

The Open Unlearning Benchmark Suite (OUBS) provides a standardized platform for evaluating machine unlearning methods across:

2 Baseline Methods: Retrain from Scratch + SISA
2 Datasets: CIFAR-10 (images) + Purchase100/Texas100 (tabular)
2 Deletion Scenarios: Random + Cluster-based
3 Evaluation Dimensions: Effectiveness, Utility, Efficiency

Features

Comprehensive Evaluation

Effectiveness: Membership inference attacks, model distance metrics
Utility: Accuracy, F1-score, calibration, per-class fairness
Efficiency: Training time, memory usage, computational cost

Modular Design

Easy to add new datasets
Simple to implement new unlearning methods
Flexible deletion scenario generators

Reproducible Experiments

Seed-based reproducibility
Automated experiment tracking
JSON-formatted results

Production-Ready

Command-line interface
Comprehensive logging

Project Structure

oubs/
├── data/                          # Dataset loaders
│   ├── cifar_loader.py           # CIFAR-10 loader with index tracking
│   └── purchase_loader.py        # Purchase100/Texas100 loader
│
├── models/                        # Model architectures
│   ├── cifar_cnn.py              # CNN for CIFAR-10
│   ├── purchase_mlp.py           # MLP for tabular data
│   └── trainer.py                # Unified training utilities
│
├── deletion_scenarios/            # Deletion strategy generators
│   ├── random.py                 # Random deletion
│   └── cluster.py                # Cluster-based deletion
│
├── methods/                       # Unlearning methods
│   ├── retrain/
│   │   └── retrain_baseline.py  # Retrain from scratch
│   └── sisa/
│       └── sisa.py              # SISA implementation
│
├── evaluation/                    # Evaluation metrics
│   ├── effectiveness.py          # MIA, model distance
│   ├── utility.py                # Accuracy, fairness
│   └── efficiency.py             # Time, resources
│
├── experiments/                   # Experiment runners
│   ├── run_cifar.py              # CIFAR-10 experiments
│   └── run_purchase.py           # Purchase100 experiments
│
├── results/                       # Experiment outputs
└── requirements.txt              # Dependencies

Installation

Prerequisites

Python 3.10+ (tested with 3.13)
GPU recommended (CUDA or Apple Silicon with MPS support)

Setup

# Clone the repository
git clone https://github.com/ndminhvn/oubs.git
cd oubs

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt

# For CUDA support (NVIDIA GPUs), install PyTorch with CUDA (refer to https://pytorch.org/get-started/locally/)
# pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

Note: The benchmark automatically detects and uses the best available device (CUDA, MPS, or CPU).

Quick Start

CIFAR-10 Experiment (Retrain Baseline)

python experiments/run_cifar.py \
    --method retrain \
    --deletion random \
  --num_delete 500 \
    --epochs 50 \
    --verbose

CIFAR-10 with SISA

python experiments/run_cifar.py \
    --method sisa \
    --deletion cluster \
  --delete_percent 5 \
    --shards 10 \
    --epochs 50

Purchase100 Experiment

python experiments/run_purchase.py \
    --dataset purchase100 \
    --method sisa \
    --deletion random \
  --delete_percent 1 \
    --shards 10

Percentage-Based Deletions

Use --delete_percent 1 for 1% deletions or --delete_ratio 0.01 if you prefer fractional inputs.
Percentage / ratio arguments override --num_delete; the framework computes the exact sample count per dataset.
Config files now record the resolved deletion summary (percent + absolute count) to streamline experiment tracking.

Usage Examples

Example 1: Random Deletion with Retrain

python experiments/run_cifar.py \
    --method retrain \
    --deletion random \
  --delete_percent 2 \
    --epochs 50 \
    --lr 0.001 \
    --batch_size 128 \
    --save_models \
    --output_dir ./results/retrain_random

Example 2: Cluster-based Deletion with SISA

python experiments/run_cifar.py \
    --method sisa \
    --deletion cluster \
  --delete_ratio 0.02 \
    --cluster_method class-based \
    --shards 10 \
    --epochs 50 \
    --save_models

Example 3: Purchase100 with Custom Settings

python experiments/run_purchase.py \
    --dataset purchase100 \
    --method sisa \
    --deletion cluster \
  --delete_percent 1 \
    --shards 15 \
    --epochs 40 \
    --lr 0.0005 \
    --hidden_dims 256 128 64 \
    --exp_name purchase_sisa_cluster

Evaluation Metrics

1. Unlearning Effectiveness

Measures how well deleted data's influence is removed:

Membership Inference Attack (MIA): Lower AUC = better unlearning
Attack Accuracy / Threshold: Reported alongside AUC to show how confident an adversary could be
Model Distance: KL/JS divergence to retrain baseline
Confidence Gap & Variance: Difference in means plus per-group std devs to diagnose residual leakage

2. Utility Preservation

Evaluates performance on remaining data:

Test Accuracy: Classification accuracy
F1-Score: Macro and weighted F1
Expected Calibration Error (ECE): Model calibration
Fairness: Per-class accuracy variance

3. Efficiency

Quantifies computational cost:

Training Time: Total time for unlearning
Speedup: Relative to retrain baseline
Memory Usage: Peak RAM/GPU memory
Model Parameters: Number of retrained parameters
Shard Coverage: Fraction of SISA shards touched during unlearning

Extending the Benchmark

Adding a New Dataset

Create a loader in data/:

class NewDatasetLoader:
    def __init__(self, data_dir, seed=42):
        # Load data
        pass

    def get_loaders(self, batch_size):
        # Return train, val, test loaders
        pass

    def get_train_indices(self):
        # Return all training indices
        pass

Create corresponding model in models/
Create experiment runner in experiments/

Adding a New Unlearning Method

Create method file in methods/your_method/:

class YourMethod:
    def __init__(self, model_fn, device):
        self.model_fn = model_fn
        self.device = device

    def unlearn(self, delete_indices, retain_loader, ...):
        # Implement unlearning logic
        return unlearned_model, metrics

Import and integrate into experiment runners

Adding a New Deletion Scenario

Create scenario file in deletion_scenarios/:

def select_your_deletion(all_indices, num_delete, **kwargs):
    # Generate deletion indices
    delete_indices = ...
    retain_indices = ...
    return delete_indices, retain_indices, info

Results Format

Results are saved as JSON files with the following structure:

{
  "config": {
    "method": "sisa",
    "deletion": "cluster",
    "num_delete": 500,
    "delete_percent": 1.0,
    "deletion_summary": "1% (500/50000 samples)",
    "total_train_samples": 50000
  },
  "utility_metrics": {
    "test_accuracy": 0.8542,
    "f1_macro": 0.8498,
    "ece": 0.0234
  },
  "effectiveness_metrics": {
    "mia_auc": 0.5123,
    "mia_attack_accuracy": 0.541,
    "mia_attack_threshold": 0.7621,
    "kl_divergence_to_retrain": 0.0045
  },
  "efficiency_metrics": {
    "training_time": 234.56,
    "unlearn_time": 45.67,
    "speedup_vs_full": 5.14,
    "shard_fraction": 0.2
  }
}

Reporting & Visualization Pipeline

Once experiments finish, aggregate every results.json into publication-ready artifacts:

python scripts/py/generate_benchmark_report.py \
    --results_dir ./results/benchmark_suite \
    --output_dir ./results/benchmark_suite/overall/analysis

This produces:

summary_table.csv / benchmark_report.md: human-readable overview with deltas vs. retrain
results_table.tex: LaTeX-ready table for papers
*_comparison.(png|pdf) plus radar_*.pdf: figures covering effectiveness, utility, efficiency, and trade-offs

Benchmark Scope

Supported Configurations

Dataset	Method	Deletion	Status
CIFAR-10	Retrain	Random	✅
CIFAR-10	Retrain	Cluster	✅
CIFAR-10	SISA	Random	✅
CIFAR-10	SISA	Cluster	✅
Purchase100	Retrain	Random	✅
Purchase100	Retrain	Cluster	✅
Purchase100	SISA	Random	✅
Purchase100	SISA	Cluster	✅

Contributing

We welcome contributions! Areas for improvement:

Additional unlearning methods (e.g., gradient ascent, influence functions)
More datasets (ImageNet, NLP datasets)
Advanced deletion scenarios
Additional evaluation metrics
Performance optimizations

Acknowledgments

SISA implementation based on Bourtoule et al. "Machine Unlearning" (2021)
Purchase100/Texas100 datasets from Purchase100-Texas100-datasets
Inspired by various machine unlearning research

Contact

Minh Nguyen: GitHub
Project Issues: GitHub Issues

Status: Active Development | Version: 1.0.0 | Last Updated: December 2025

Name		Name	Last commit message	Last commit date
Latest commit History 20 Commits
data		data
deletion_scenarios		deletion_scenarios
evaluation		evaluation
experiments		experiments
methods		methods
models		models
results		results
scripts		scripts
utils		utils
.gitignore		.gitignore
BENCHMARK_GUIDE.md		BENCHMARK_GUIDE.md
LICENSE		LICENSE
Machine_Unlearning.ipynb		Machine_Unlearning.ipynb
README.md		README.md
__init__.py		__init__.py
analyze_results.py		analyze_results.py
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation