AI Model Developed by Israeli Scientists Reconstructs Visual Experiences from Brain Scans

A team of scientists from the Weizmann Institute has developed an artificial intelligence model capable of reconstructing images a person sees with high accuracy, using functional magnetic resonance imaging (fMRI) scans. The model utilizes high-resolution fMRI to obtain clear scans. The scan decoder operates on two branches: one analyzes brain activity to determine object location, contours, shape, orientation, and color distribution, while the other focuses on the semantic meaning of the image, understanding whether a person sees a pear or an apple. A diffusion neural network then generates the final picture.

To address the scarcity of fMRI scans for training, researchers created an additional encoder. By alternately training the encoder and decoder, they enabled the use of data without prior scanning. The training cycle involves feeding the model an image the subject has never seen, like a leopard. The encoder predicts what an fMRI scan would look like if a person viewed the leopard. The decoder then attempts to reconstruct the original image from this artificial scan. Initial reconstructions bear little resemblance to the original, but the system refines both the encoder and decoder by comparing the original image with the reconstructed one and identifying errors.

The research was presented at the Cognitive Computational Neuroscience conference. The developed "universal encoder" requires only about an hour of calibration for a new participant, a significant reduction from the previous 40 hours, making neurobiological research considerably cheaper and faster. The model already surpasses existing analogs in accuracy, though it occasionally makes mistakes, such as confusing a dog with a goat.

Future plans include reconstructing videos, audio, dreams, and imagined images from fMRI scans. The technology could potentially give a voice to fully paralyzed individuals. "Mind reading is a good name for what we are doing," noted co-author Michal Irani. However, the technology has raised concerns among neuroethics experts regarding the potential risk of unauthorized extraction of personal data and thoughts if adapted for portable EEG headsets. The creators are currently focused on the project's positive medical and scientific prospects.