A unified benchmark and library for evaluating model-extraction attacks and ownership defenses on graph neural networks under a single reproducible black-box protocol.
Why GraphIP-Bench · Highlights · Quick Start · Research Questions · Layout · Citation
Graph neural networks deployed as cloud services can be stolen through model-extraction attacks: an adversary submits carefully chosen queries, records the labels or confidence scores that the endpoint returns, and trains a local surrogate which reproduces the target's behaviour. A successful theft leaks the owner's intellectual property, undermines pay-per-query revenue, and lets competitors recreate proprietary functionality at low cost.
A growing line of ownership defenses — watermarking, fingerprinting, output perturbation, query-pattern detection — tries to prevent or trace such theft. But experimental practice in this area is fragmented: studies use private splits, incompatible budgets, and inconsistent metrics; the few existing testbeds focus on robustness or privacy and exclude model extraction together with ownership verification.
GraphIP-Bench addresses this gap. It standardises the evaluation of two complementary tracks — the extraction track (an adversary trains a surrogate that imitates a deployed GNN) and the ownership track (the model owner verifies a watermark or fingerprint after extraction) — under a single black-box protocol with shared splits, queries, and budgets. We then add a joint attack-and-defense track that runs every extraction attack on every defended target and measures watermark survival on the extracted surrogate — the setting that actually determines whether a defense is useful.
| Component | What it gives you |
|---|---|
| 12 attacks | Six MEA-style baselines (MEA0–MEA5), adversarial (AdvMEA), centrality-driven (CEGA), structure-aware (Realistic), and three data-free variants (DFEA_I/II/III) |
| 12 defenses | 5 ownership-tracing (BackdoorWM, RandomWM, SurviveWM, ImperceptibleWM, Integrity) + 7 information-limiting (OP_low, OP_high, PR_2bit, PR_top1, PRADA, AdaptMisinfo, GradRedir) |
| 10 datasets | Homophilic citation (Cora, CiteSeer, PubMed), coauthor/product (CoauthorCS, CoauthorPhysics, Computers, Photo), large-scale (OGBN-Arxiv), heterophilic (RomanEmpire, AmazonRatings) |
| 3 backbones × 3 tasks | GCN, GAT, GraphSAGE on node classification, link prediction, graph classification |
| Joint track | Surrogate fidelity and watermark survival on the extracted surrogate, all under one protocol |
| Reproducibility | Public splits, shared query sets, fixed seeds, JSON-Lines outputs, ready-made aggregation scripts |
|
Every attack is evaluated against every defense under the same splits, budgets, and endpoint assumptions. Fidelity, ownership verification, utility, and compute cost are reported on a shared metric suite. |
The joint track measures whether a watermark still verifies after the model is actually extracted — exposing the gap that single-model evaluations miss. |
|
Beyond the seven homophilic graphs of the core protocol, we add OGBN-Arxiv (169K nodes, 40 classes) and the heterophilic RomanEmpire / AmazonRatings to stress test attack and defense behaviour. |
Fixed seeds, shared query sets, and JSON-Lines output records per (dataset, attack, defense, regime, budget, seed) cell. A single-seed run is directly comparable to the corresponding cell in the paper. |
| Signal | Scale in the benchmark |
|---|---|
| Extraction attacks | 12 |
| Ownership / information-limiting defenses | 12 |
| Graph datasets | 10 |
| GNN backbones | 3 (GCN, GAT, GraphSAGE) |
| Graph-learning tasks | 3 (node / link / graph) |
| Data-availability regimes | 4 (both, features-only, structure-only, data-free) |
| Standardized query budgets | 5 (0.05× — 1.00× test size) |
| Seeds per cell | 3 |
| Evaluation tracks | 3 (extraction, ownership, joint) |
git clone https://github.com/LabRAI/GraphIP-Bench.git
cd GraphIP-Bench
conda create -n graphip python=3.11 -y
conda activate graphip
pip install -r requirements.txtRun one cell of the extraction track:
python scripts/run_rq1_single.py \
--dataset Cora \
--attack MEA0 \
--regime both \
--budget 0.25 \
--seed 0 \
--gpu \
--output-dir outputs/RQ1Run the joint attack-and-defense track on a single dataset:
python examples/run_joint_evaluation.py \
--dataset Cora \
--seed 0 \
--gpuEach run writes one JSON-Lines record per (attack, defense, regime, budget, seed) tuple to outputs/. Records can be merged and analysed with the helpers in scripts/ (e.g. merge_rq1_rq5_faithful_results.py).
GraphIP-Bench supports Linux (recommended) with Python 3.10+. We pin PyTorch / DGL / PyG versions because DGL 2.1.0 graphbolt kernels only ship pre-built shared libraries for PyTorch 2.0–2.2 on CUDA 12.1.
conda create -n graphip python=3.11 -y
conda activate graphip
pip install -r requirements.txtpython -m venv .venv && source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt📦 Pinned stack
| Library | Version | Why pinned |
|---|---|---|
| PyTorch | 2.2.1 + cu121 | DGL 2.1.0 graphbolt wheels only support PT 2.0–2.2 / CUDA 12.1 |
| DGL | 2.1.0 + cu121 | Graphbolt kernels used by the loaders |
| PyTorch Geometric | 2.7.0 | Link-prediction and graph-classification adapters |
| OGB | 1.3.6 | OGBN-Arxiv loader |
| NumPy | < 2.0 | DGL kernel ABI |
GraphIP-Bench is organised around five research questions; each maps to a top-level reproduction script.
| RQ | Question | Entry point |
|---|---|---|
| RQ1 | How does extraction effectiveness change with query budget, and does the trend hold on heterophilic and large-scale graphs? | scripts/run_rq1_single.py |
| RQ2 | How effective are existing defenses on the protected model? | scripts/run_defense_single.py |
| RQ3 | How well do defenses balance protection and utility? | scripts/run_defense_single.py + scripts/run_defense_hp_ablation.py |
| RQ4 | What are the computational complexity and practical efficiency of attacks and defenses? | wall-clock and peak-mem fields recorded in every JSONL record |
| RQ5 | How effective are defenses in the joint adversarial setting, and does the watermark signal survive on the extracted surrogate? | examples/run_joint_evaluation.py |
Supplementary tracks:
| Track | Entry point |
|---|---|
| Cross-architecture extraction (undefended) | scripts/run_cross_arch.py |
| Cross-architecture on defended targets | scripts/run_cross_arch_defended.py |
| Link prediction (Cora) | scripts/run_link_prediction.py, examples/run_link_pred_experiments.py |
| Graph classification (ENZYMES, PROTEINS) | scripts/run_graph_class.py |
| Endpoint ablation (hard-label vs. confidence scores) | examples/run_endpoint_ablation.py |
| Budget grid ablation | scripts/run_budget_ablation.py |
| Query-split / structure analysis | scripts/run_query_split_ablation.py, scripts/run_structure_analysis.py |
| Baseline utility across backbones | scripts/run_baseline_utility.py |
| Category | Name | Key idea |
|---|---|---|
| Data-driven | MEA0–MEA1 | Random / shuffled-order subgraph queries (Wu 2022 baselines) |
MEA2 | Structure-only extraction with one-hot node-ID features | |
MEA3–MEA5 | Shadow-graph and feature-shuffled variants of MEA | |
AdvMEA | Adversarial-query attack with policy search | |
CEGA | Centrality- and entropy-driven node selection | |
Realistic | Structure-aware pipeline with auxiliary edge model | |
| Data-free | DFEA_I | KL-divergence soft-label distillation |
DFEA_II | Hard-label supervision (label-only) | |
DFEA_III | Label-only + consistency loss between two surrogates |
Each attack supports four data-availability regimes: both, x_only, a_only, and data_free, controlled by --regime.
| Family | Name | Mechanism |
|---|---|---|
| Ownership-tracing | BackdoorWM | Trigger-based backdoor watermark |
RandomWM | Random-graph trigger watermark | |
SurviveWM | SNNL-based watermark designed to survive extraction | |
ImperceptibleWM | Representation-level imperceptible watermark | |
Integrity | Query-time fingerprint verifier (no model-side trigger) | |
| Output perturbation | OP_low / OP_high | Gaussian noise on returned logits at two scales |
PR_2bit | Two-bit quantization of returned probabilities | |
PR_top1 | Top-1 label-only output | |
| — | ||
| Query detection | PRADA | Distance-based query-stream detector |
AdaptMisinfo | Adaptive misinformation on flagged queries | |
GradRedir | Gradient-redirection on flagged queries |
All datasets are downloaded automatically on first use and cached under data/.
| Dataset | Nodes | Edges | Classes | Edge homophily | Source |
|---|---|---|---|---|---|
| Cora | 2,708 | 5,278 | 7 | 0.81 | DGL Planetoid |
| CiteSeer | 3,327 | 4,614 | 6 | 0.74 | DGL Planetoid |
| PubMed | 19,717 | 44,325 | 3 | 0.80 | DGL Planetoid |
| Computers | 13,752 | 252,737 | 10 | 0.78 | DGL Amazon |
| Photo | 7,650 | 122,906 | 8 | 0.83 | DGL Amazon |
| CoauthorCS | 18,333 | 81,894 | 15 | 0.81 | DGL Coauthor |
| CoauthorPhysics | 34,493 | 247,962 | 5 | 0.93 | DGL Coauthor |
| OGBN-Arxiv | 169,343 | 667,793 | 40 | 0.70 | OGB |
| RomanEmpire | 22,662 | 44,258 | 18 | 0.29 | Heterophilic |
| AmazonRatings | 24,492 | 105,296 | 5 | 0.45 | Heterophilic |
ENZYMES and PROTEINS (graph classification) are loaded through TUDataset in PyG.
Every paper figure or table is generated from JSON-Lines records produced by the entry-point scripts above. A typical workflow:
# 1. Sweep RQ1 over (dataset × attack × regime × budget × seed)
for ds in Cora CiteSeer PubMed Computers Photo CoauthorCS CoauthorPhysics OGBNArxiv RomanEmpire AmazonRatings; do
for atk in MEA0 MEA1 MEA2_Wu2022 MEA3 MEA4 MEA5 AdvMEA CEGA Realistic \
DFEA_I_RealGraph DFEA_II_E DFEA_III_E; do
for regime in both x_only a_only data_free; do
for budget in 0.05 0.10 0.25 0.50 1.00; do
for seed in 0 1 2; do
python scripts/run_rq1_single.py \
--dataset $ds --attack $atk \
--regime $regime --budget $budget \
--seed $seed --gpu \
--output-dir outputs/RQ1
done
done
done
done
done# 2. Sweep RQ5 (joint track) at the medium budget 0.25×
for ds in Cora CiteSeer PubMed Computers Photo CoauthorCS CoauthorPhysics OGBNArxiv RomanEmpire AmazonRatings; do
for seed in 0 1 2; do
python examples/run_joint_evaluation.py --dataset $ds --seed $seed --gpu
done
done# 3. Merge & summarise into the unified table format
python scripts/merge_rq1_rq5_faithful_results.pyAll runs use fixed seeds (0, 1, 2) and the shared query sets defined in pygip/datasets/, so a single-seed run is directly comparable to the corresponding cell in the paper.
GraphIP-Bench/
├── pygip/ # Core library
│ ├── datasets/ # 10 graph datasets + link-pred & graph-class loaders
│ ├── models/
│ │ ├── attack/ # 12 extraction attacks
│ │ │ ├── mea/MEA.py # MEA0–MEA5
│ │ │ ├── AdvMEA.py # adversarial-query attack
│ │ │ ├── CEGA.py # centrality + entropy
│ │ │ ├── Realistic.py # structure-aware pipeline
│ │ │ ├── DataFreeMEA.py # DFEA_I / DFEA_II / DFEA_III
│ │ │ └── linkpred_attacks.py # link-prediction adapters
│ │ ├── defense/ # 12 defenses
│ │ │ ├── BackdoorWM.py · SurviveWM.py · RandomWM.py
│ │ │ ├── ImperceptibleWM.py · Integrity.py
│ │ │ ├── OutputPerturbation.py · PredictionRounding.py
│ │ │ ├── NonWatermarkDefenses.py (PRADA / AdaptMisinfo / GradRedir)
│ │ │ └── linkpred_defenses.py
│ │ └── nn/ # GCN, GAT, GraphSAGE backbones (DGL + PyG)
│ ├── evaluation/ # Watermark-survival evaluator
│ └── utils/ # Metrics, hardware probing
├── examples/ # Reproduction entry points
│ ├── run_joint_evaluation.py # RQ5 joint attack × defense × watermark survival
│ ├── run_cross_arch_attacks.py
│ ├── run_cross_arch_node_class.py
│ ├── run_endpoint_ablation.py
│ └── run_link_pred_experiments.py
├── scripts/ # Per-cell single-run scripts & post-processing
│ ├── run_rq1_single.py # one (dataset, attack, regime, budget, seed) cell
│ ├── run_defense_single.py
│ ├── run_baseline_utility.py
│ ├── run_cross_arch.py · run_cross_arch_defended.py
│ ├── run_budget_ablation.py · run_query_split_ablation.py
│ ├── run_defense_hp_ablation.py · run_structure_analysis.py
│ ├── run_link_prediction.py · run_graph_class.py
│ ├── run_joint_new_defense.py · run_new_defense.py
│ └── merge_rq1_rq5_faithful_results.py
├── data/ # Auto-populated dataset cache
├── outputs/ # JSON-Lines run records
├── requirements.txt
├── LICENSE
└── README.md
Every script writes one JSON-Lines record per run to outputs/. The schema is consistent across all tracks so records can be filtered and joined trivially:
{
"track": "RQ1",
"dataset": "Cora",
"attack": "MEA0",
"regime": "both",
"budget": 0.25,
"seed": 0,
"fidelity": 87.55,
"accuracy": 79.7,
"f1": 77.92,
"train_target_time": 1.21,
"query_target_time": 0.0019,
"train_surrogate_time": 0.78,
"total_time": 2.00,
"peak_gpu_mem(GB)": 0.094,
"status": "ok"
}For the joint track, the record additionally carries defense, defense_arch, surrogate_fidelity_to_defended, and wm_acc_on_surrogate (watermark survival).
All reported results use a single NVIDIA A100 80 GB GPU with CUDA 12.1. The lightweight attacks and defenses run comfortably on a single GPU with at most 16 GB; only the Realistic pipeline and ImperceptibleWM representation-level optimization require the full 80 GB allocation. OGBN-Arxiv runs request 192 GB system memory.
This project is released under the MIT License.
If you use GraphIP-Bench, please cite:
@inproceedings{graphipbench2026,
title={GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?},
author={Zhao, Kaixiang and Shen, Bolin and Dai, Yuyang and Chakraborty, Shayok and Dong, Yushun},
booktitle={NeurIPS},
year={2026}
}GraphIP-Bench turns model extraction and ownership defense into a single reproducible benchmark — and exposes the gap that single-model evaluations miss.