Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity

[AtomScott]

1. Overview (basic ideas)

The authors were able reconstruct 2D images from retinotopic mappings across the visual system.

2. Novelty

A fully convolutional image-to-image network with VGG feature loss and an adversarial regularizer improved video reconstruction.

3. Method (Technical details)

brain2pix has two major components:

1. Brain activity 2 pixel space:
   a receptive field mapping that transforms the brain activity of visual regions to a tensor in pixel space, exploiting the topographical organization of the visual cortex.
2. Pixel space to natural images:
   a pix2pix network that converts the brain responses in pixel space to realistic looking natural images.

4. Results

5. links to papers, codes, etc.

bioarxiv: https://www.biorxiv.org/content/10.1101/2021.02.02.429430v1

6. Thoughts, Comments

Results don't look to good, but I guess it's a start. Progress in computer vision is very fast, so I can't image where we'll be in ~10 years.

Also I have no idea how they got the data.

Not really sure of the baselines as well.

7. bibtex

@Article {Le2021.02.02.429430,
author = {Le, Lynn and Ambrogioni, Luca and Seeliger, Katja and G{"u}{\c c}l{"u}t{"u}rk, Ya{\u g}mur and van Gerven, Marcel and G{"u}{\c c}l{"u}, Umut},
title = {Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity},
elocation-id = {2021.02.02.429430},
year = {2021},
doi = {10.1101/2021.02.02.429430},
publisher = {Cold Spring Harbor Laboratory},
abstract = {Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent in the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments, we show that our method offers a significant improvement over existing video reconstruction techniques.Competing Interest StatementThe authors have declared no competing interest.},
URL = {https://www.biorxiv.org/content/early/2021/02/03/2021.02.02.429430},
eprint = {https://www.biorxiv.org/content/early/2021/02/03/2021.02.02.429430.full.pdf},
journal = {bioRxiv}
}

8. Related Papers

• pix2pix paper
• Papers from https://neuralcoding.nl/?