This work presents EEG-EditBench. Beyond 200-way retrieval, it asks a more direct question: can EEG-to-image models distinguish the viewed image from an edited version in which only one visual element has been changed?
EEG-EditBench is built from the 200 THINGS-EEG2 test images. It contains 2,137 quality-controlled edits covering object identity, attributes, background, and object presence.
The complete dataset, including the 200 source images, edited images, and metadata, is available on Hugging Face. After downloading it, place the images under data/images/:
data/images/
└── 00001_aircraft_carrier/
├── aircraft_carrier_06s.jpg
├── attribute/
├── background/
├── replacement/
└── removal/
Download the original and preprocessed EEG recordings and stimulus images from the official resources:
When evaluating a model directly, preprocess the EEG data according to that model's requirements and pass its location through --eeg-data.
git clone https://github.com/XiaoZhangYES/EEG-EditBench.git
cd EEG-EditBench
python -m pip install -e .Prepare an embeddings.npz file containing:
query_idsandquery_embeddingsfor EEG queries;image_idsandimage_embeddingsfor the source and edited images.
Then run:
python scripts/evaluate.py embeddings \
--manifest manifests/eeg_editbench.jsonl \
--input embeddings.npz \
--metric cosine \
--output results.jsonThe similarity metric can be cosine or dot.
Copy examples/model_adapter_template.py and connect its three methods to your model. The adapter loads the checkpoint and produces EEG-to-image scores; the benchmark handles image paths and metric computation.
python scripts/evaluate.py model \
--manifest manifests/eeg_editbench.jsonl \
--adapter path/to/my_adapter.py \
--adapter-config path/to/model_config.json \
--eeg-data path/to/things_eeg2_test_data \
--output results.jsonPrepared score matrices are also supported. See docs/evaluation.md for the input format and command.
Each run writes one results.json containing:
- 200-way Original accuracy;
- EP-Top1 accuracy;
- Identity, Attribute, Background, Removal, and Overall 2AFC accuracy.
Ties are counted as incorrect.
Install the optional construction dependencies:
python -m pip install -e '.[construction]'The construction pipeline is organized as explicit stages for source understanding, edit-target generation, edit-prompt generation, image editing, quality control, and final filtering:
source-manifest -> source-understanding -> *-targets -> *-prompts
-> images -> result-manifest -> quality-control -> filter
List all stages or inspect the options for one stage with:
eeg-editbench-build --help
eeg-editbench-build source-understanding --helpAll prompts are stored as editable Markdown files under prompts/, so users can inspect or replace them without changing the Python code.
EEG-EditBench is built on the THINGS-EEG2 test set. We thank the dataset authors for making the EEG recordings, stimulus images, and preprocessing resources publicly available.
We also thank the authors of the following models for releasing the implementations used in our evaluation:
The code in this repository is released under the MIT License. Please refer to the Hugging Face dataset page for the dataset release information.
@misc{zhang2026eegeditbench,
title = {EEG-EditBench: Probing Visual Information in EEG--Image Retrieval Models with Controlled Image Edits},
author = {Kaifan Zhang and Lihuo He and Yuqi Ji and Junjie Ke and Lukun Wu and Tianhao You and Xinbo Gao},
year = {2026},
eprint = {2607.27857},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.27857}
}