Optimizing Images for the Brain: From fMRI Encoding Models to Brain-Guided Generation

In standard neural encoding, we train a model that maps an image to predicted brain activity. But once this model is differentiable, we can reverse the question: instead of asking “what brain response will this image produce?”, we can ask “what image would produce the strongest response in this brain region?”

Motivation: From Prediction to Intervention

What image would make a face-selective brain region respond as strongly as possible? What about a place-selective region, or a region that responds to written words?

Traditionally, neuroscience studies answer these questions by designing stimulus sets, showing them to participants in an fMRI scanner, and measuring which images activate which regions. But this process is inherently limited by the images we choose to test.

Recent work suggests a different approach. First, train an image-to-fMRI model: a neural network that receives an image and predicts the brain response it would evoke. Then, instead of only using this model for prediction, use it as an objective. Search for images that maximize the predicted activity of a target voxel, region of interest (ROI), or cortical pattern.

In machine learning terms, image to fMRI models turns the brain into a differentiable loss function. The image, or the latent code of a generative model, becomes the variable we optimize. The output is a synthetic image that the model predicts will strongly activate a chosen part of the visual cortex.

This idea connects three fields: neural encoding models, feature visualization, and generative modeling. Together, they allow us to move from asking “can we predict brain activity?” to asking “can we synthesize stimuli that reveal what a brain region represents?”

Background: Image-to-fMRI Encoding Models

In a previous post, I introduced the basic idea of image-to-fMRI encoding models: models that receive a visual stimulus, such as an image, and predict the brain response it would evoke.

A central dataset in this field is the **Natural Scenes Dataset (NSD)**. NSD contains high-resolution 7T fMRI responses from participants viewing thousands of natural images, creating paired examples of the image and the participant's fMRI activity. This provides supervised examples for learning image-to-brain encoding models.

The Natural Scenes Dataset. Participants view natural images inside an fMRI scanner, to get image-fMRI pairs.

Modern approaches usually do not train the visual representation from scratch. Instead, they use a pretrained vision model as a feature extractor. A common example is **CLIP2Brain**, which uses CLIP visual representations and learns a prediction head that maps those representations to fMRI responses. After training, the model can predict how strongly different brain regions are expected to respond to a new image.

Image-to-fMRI encoding with CLIP features. A pretrained CLIP vision model extracts an image representation, and a learned prediction head maps this representation to voxel-wise fMRI responses. These predictions can then be analyzed inside specific regions of interest, such as face-, body-, scene-, or early visual areas.

A full fMRI prediction can contain tens of thousands of voxel values. To make the output interpretable, researchers often focus on regions of interest (ROIs) — groups of voxels associated with a meaningful anatomical or functional area.

For example, in the visual cortex, we may care about:

• Face-selective regions (e.g., FFA)
• Place- or scene-selective regions (e.g., PPA)
• Body-selective regions
• Word-selective regions
• Early visual areas (e.g., V1, V2)

This ROI-level view is important because it lets us connect model predictions to semantic hypotheses about brain function. If an ROI is known to respond to faces, then images that strongly activate it should, ideally, contain faces or face-like features.

Before generating new images, a simple way to understand an ROI is to look at the natural images that activate it the most. Given a trained image-to-fMRI model, we can rank a dataset of images according to the average activation in a target ROI.

For example, we can take all images from NSD, compute the mean activation in a face-selective region, and retrieve the top images.

Top-activating images for a face-selective region. Ranking natural images by their measured or predicted activation can reveal what types of visual content drive a target ROI. In this example, many highly activating images contain people, faces, or animals.

This retrieval-based approach is intuitive and useful, but it has one major limitation: it can only show us images that already exist in the dataset. If the most informative stimulus is not present in NSD, retrieval will never find it.

Activation Maximization: Reversing the Flow

Once an encoding model is differentiable, we can use it in the opposite direction. This idea is known as activation maximization.

In classic deep learning interpretability, activation maximization is used to visualize what a neuron is sensitive to by iteratively updating the pixels of a random image so that the chosen neuron becomes more active. In the brain setting, the target is a predicted fMRI response.

Recent work on **Voxel-Weighted Activation Maximization** shows that this can be applied directly to encoding models of human visual cortex. Starting from a random image, the pixels are iteratively optimized to increase the predicted response of a target voxel.

Activation maximization. Starting from random noise, the image is iteratively optimized with gradient ascent to maximize the mean predicted activation of the voxels in a target ROI, producing an image predicted by the model to strongly drive that brain region.

For early visual areas (like V2), the optimized images contain localized high-contrast patches consistent with classic retinotopic mapping. In higher-level regions, the images become more semantic: face-selective areas (FFA) produce face-like structures, while place-selective areas (PPA) yield ground planes and buildings.

results from paper.

NeuroGen & BrainDiVE: Adding a Generative Prior

Direct pixel optimization has an important weakness: because the optimizer can change every pixel freely, it often finds adversarial patterns that strongly activate the model but look unnatural.

A natural next step is to optimize inside the latent space of a generative model. This is the idea behind NeuroGen. Instead of directly changing pixels, NeuroGen combines an fMRI-trained encoding model with a deep generative network. The generator provides a natural-image prior,and therefore restricts the search to images that are more likely to look like real visual stimuli.

BrainDiVE: Diffusion Based Generation

BrainDiVE — *Brain Diffusion for Visual Exploration* — takes this a step further by replacing older generators with a modern diffusion model. During generation, an estimated clean image is passed through a differentiable brain encoder that predicts voxel-wise fMRI activity. Gradients from this score are then used to guide the diffusion process.

BrainDiVE pipeline. Starting from noise, a diffusion model iteratively denoises an image representation. At each step, the current image is decoded and passed through a brain encoder, which predicts activation in a target brain region. The predicted activation is then used as a guidance signal, and gradients are backpropagated through the generation process to steer the image toward stimuli that are predicted to strongly activate the selected cortical region.

What makes BrainDiVE especially interesting is that the generated images are not random or arbitrary. When the authors target well-known category-selective regions, the generated images show semantic structure that matches what we already know about these areas. For example, images generated for face-selective regions contain face-like structure, images generated for place-selective regions resemble rooms, buildings, or spatial layouts, and images generated for body-, word-, and food-selective regions also reflect the expected visual categories.

This makes the results easy to interpret: the model is not only increasing a numerical activation score, but producing images that visually reveal what kind of content the target region is predicted to prefer. BrainDiVE validates this idea on known ROIs such as faces, places, bodies, food, and words, showing that brain-guided diffusion can synthesize preferred images with appropriate semantic specificity.

BrainDiVE results validating generation across known ROIs.

A technical intuition: how does the brain guide the diffusion model?

The subtle part of BrainDiVE is that the brain encoder expects a clean image, but diffusion generation happens through noisy intermediate latents. Early in the denoising process, the current latent (z_t) is not yet a real image. So how can we use the brain encoder to guide the generation?

BrainDiVE solves this by estimating, at each denoising step, what the final clean image might look like. In a diffusion model, the network predicts the noise in the current latent:

Using this prediction, we can estimate the clean latent:

This is not the final image, but it is an approximation of the image that the diffusion process is currently moving toward. BrainDiVE decodes this estimated clean latent into an image-like prediction and passes it into the image-to-fMRI encoder.

The brain encoder predicts voxel-wise activity, and for a target region, BrainDiVE averages the predicted activation of the target voxels. Then it computes the gradient of this score with respect to the current diffusion latent or to the diffusion process parameters. This gradient tells the sampler or the generator how to slightly change the denoising direction so that the predicted image will activate the target region more strongly:

This is the core idea. The diffusion model still provides the natural-image prior, while the brain encoder adds a guidance signal during sampling. Instead of guiding the generation toward a text prompt or a class label, BrainDiVE guides it toward a predicted brain response.

BrainExplore: From Generating Preferred Images to Discovering Brain Representations

While BrainDiVE uses a known target to synthesize preferred images, BrainExplore takes a hypothesis-free direction, asking: “What interpretable visual patterns exist across the visual cortex, and where are they represented?”

A useful motivation comes from the work A Highly Selective Response to Food in Human Visual Cortex Revealed by Hypothesis-Free Voxel Decomposition. Many classic visual neuroscience studies focus on predefined categories, such as faces, places, bodies, and words. In contrast, this work used data-driven voxel decomposition on fMRI responses to natural images. The method recovered known components, such as faces, places, bodies, and words, but also revealed a food-selective component. This is a nice example of discovery neuroscience: instead of only testing categories we already expected, the analysis exposed a meaningful response pattern that emerged from the data.

BrainExplore scales this idea into a larger automated framework. It starts from the Natural Scenes Dataset, where subjects viewed natural images from COCO while their fMRI activity was measured. To expand the image pool, BrainExplore also uses many additional COCO images that were not shown to the subjects. For these images, an image-to-fMRI encoder predicts the expected brain response. This creates a larger pool of image–fMRI pairs: some measured, some predicted.

The method can be understood in four stages.

1. Decompose — Activity within predefined brain regions is broken down into candidate patterns.

2. Visualize and explain. The images that most strongly activate each pattern are retrieved, and a vision-language pipeline generates a natural-language explanation of their shared content.

3. Upscale — To scale this process, BrainExplore builds a dictionary of candidate hypotheses and automatically tests pattern-hypothesis pairs.

4. Discover — High-scoring pattern-concept pairs are projected back onto the cortex, suggesting coherent functional subregions.

In this sense, BrainExplore moves beyond targeted generation. BrainDiVE uses a known target to synthesize preferred images. BrainExplore searches across many brain patterns to discover what visual concepts are represented, where they appear, and how reliable those interpretations are.

BrainExplore framework. The method starts with measured fMRI responses from NSD and expands the image pool using predicted fMRI responses for additional COCO images. It then decomposes activity within each brain region into candidate patterns. For each pattern, BrainExplore retrieves the images that activate it most strongly and uses a vision-language pipeline to explain their shared visual content. High-scoring pattern–concept pairs can then be projected back onto cortex, suggesting candidate functional subregions.

Conclusion

Encoding models started as prediction tools: given an image, predict the brain response. But once these models are differentiable, they become experimental tools. We can retrieve images, optimize images, guide generative models, and even search for new interpretable patterns across cortex. This changes the role of machine learning in neuroscience: it is no longer only a way to model brain activity, but also a way to design stimuli and generate hypotheses about what the visual brain represents.