Predicting Where People Look: Human Visual Attention with ResNet18

Introduction

When we look at an image, we do not pay attention to every pixel equally.

For example, if there is a person, an animal, food, or an object in the image, our eyes usually focus on those meaningful regions first. Some areas attract more attention, while background regions may be ignored.

This project is about predicting that attention.

The main question is:

Can a deep learning model look at an image and predict where humans are likely to focus?

Instead of predicting a class label like “cat” or “dog”, this model predicts a heatmap. A heatmap is an image-like output where brighter regions represent higher human attention.

So the task is:

Input: image Output: human attention heatmap

This is not a classification problem. It is a dense regression problem, because the model predicts continuous pixel values instead of a class label.

Why I Chose This Problem

This project is connected to my previous study on visual attention.

In my earlier work, I analyzed how people looked at AI-generated and real images. Participants clicked on image regions, and those clicks were converted into attention heatmaps. That project was mostly about analysis.

In this deep learning project, I wanted to turn that idea into a prediction problem.

The difference is:

Previous study: analyze where people looked This project: predict where people may look

So instead of only studying existing attention maps, I trained a neural network to generate attention maps from images.

What Is a Heatmap?

A heatmap is a visual representation of attention.

In this project:

• brighter regions mean higher predicted attention
• darker regions mean lower predicted attention

For example, if humans mostly look at a person, an animal, or an important object in an image, the ground truth heatmap becomes brighter around those regions.

The model tries to learn this pattern.

It receives an image and tries to produce a heatmap similar to the human fixation map.

Figure 1. Example prediction from the fine-tuned model. The model does not perfectly match the ground truth, but it captures coarse attention regions.

Dataset

For training, I used a subset of the SALICON dataset.

SALICON contains images and corresponding human attention / fixation maps.

For this project, I used:

• 500 training images
• 100 validation images
• corresponding saliency / fixation maps

I used a subset because the goal was not to train the best possible saliency model. The goal was to understand the architecture and training process clearly.

All images and heatmaps were resized to 224 × 224.

The input image shape is:

[B, 3, 224, 224]

The target heatmap shape is:

[B, 1, 224, 224]

Here:

• B is the batch size
• 3 means RGB image channels
• 1 means a single attention heatmap channel
• 224 × 224 is the spatial resolution

This confirms that the task is image-to-heatmap prediction, not classification.

Why ResNet18?

I used ResNet18 because it is a well-known convolutional neural network architecture.

Normally, ResNet18 is used for image classification.

The usual flow is:

Image → ResNet18 → Class label

For example:

Image → ResNet18 → “dog”

But my problem is not classification.

I do not want one class label. I want a full heatmap.

So I modified the architecture.

Turning ResNet18 into an Encoder

In my model, ResNet18 is used as an encoder.

An encoder takes an image and extracts useful visual features from it.

These features may represent:

• edges
• textures
• shapes
• object parts
• higher-level semantic regions

Instead of using the final classification layer of ResNet18, I removed the classification head.

So the model becomes:

Image → ResNet18 Encoder → Feature Map

The encoder compresses the image into a smaller but meaningful feature representation.

Adding a Decoder

After the encoder, I added a decoder.

The decoder does the opposite of the encoder.

The encoder reduces the image into feature maps. The decoder upsamples those feature maps back into an image-sized output.

In this project, the decoder produces a single-channel heatmap.

The full model is:

Image → ResNet18 Encoder → Upsampling Decoder → Attention Heatmap

The output shape is:

[B, 1, 224, 224]

This means the model outputs one heatmap channel for each image.

At the end of the model, I used a sigmoid activation function. I did not use softmax.

Softmax is usually used for classification because it produces class probabilities. In this project, there are no class labels. Each pixel is a continuous attention value between 0 and 1.

So sigmoid is more appropriate because it keeps each predicted heatmap pixel in the 0–1 range.

Experiment 1: Frozen Encoder

In the first experiment, I froze the ResNet18 encoder.

This means the encoder weights were not updated during training.

Only the decoder was trained.

Why did I do this?

Because ResNet18 was already pretrained on ImageNet. It already knows many useful visual patterns, such as edges, shapes, textures, and object-like regions.

So the first experiment asked:

Can pretrained ResNet18 features be useful for attention heatmap prediction?

In this setup:

• encoder: frozen
• decoder: trainable
• loss function: Mean Squared Error
• optimizer: Adam
• learning rate: 1e-3
• epochs: 5

The final validation loss was approximately:

Frozen Encoder Val Loss ≈ 0.0228

This showed that the decoder was able to learn a basic mapping from ResNet features to attention heatmaps.

Experiment 2: Fine-tuning the Last ResNet Block

In the second experiment, I continued from the frozen encoder checkpoint.

This is important.

I did not start everything from zero. First, I trained the decoder while the encoder was frozen. Then I loaded that checkpoint and unfroze only the last ResNet block.

This means:

• early ResNet layers stayed frozen
• the last ResNet block became trainable
• the decoder stayed trainable

Why only the last block?

Early CNN layers usually learn general features, such as edges and simple textures. These are useful for many vision tasks. Later layers learn more task-specific features.

So instead of updating the whole ResNet18, I only fine-tuned the last residual block.

This is a safer strategy when the dataset is small.

The fine-tuning setup was:

• last ResNet block: trainable
• decoder: trainable
• earlier ResNet layers: frozen
• optimizer: Adam
• learning rate: 1e-4
• epochs: 5

The final validation loss improved to approximately:

Fine-tuned Last Block Val Loss ≈ 0.0139

This means fine-tuning helped the model adapt better to the attention prediction task.

What Backpropagation Did Here

Backpropagation is how the model learns.

The model predicts a heatmap. Then the predicted heatmap is compared with the ground truth heatmap. The difference is calculated using Mean Squared Error loss. Then backpropagation updates the trainable parameters to reduce that error.

In the frozen encoder experiment, backpropagation updated only the decoder.

In the fine-tuning experiment, backpropagation updated:

• the decoder
• the last ResNet block

It did not update the earlier ResNet layers.

This helped me understand that “training a model” does not always mean updating every layer. We can choose which parts of the architecture should learn.

An Important Mistake I Caught

One important lesson was about heatmap scaling.

At first, I normalized each heatmap so that its sum was 1. This made the target values extremely small. The model output, however, was produced by a sigmoid layer and stayed between 0 and 1.

This created a scale mismatch.

The loss looked small, but the visual predictions were not meaningful.

After checking the predicted heatmaps visually, I realized the issue. Then I changed the heatmap preprocessing so that the target heatmaps stayed in the 0–1 range.

This was an important lesson:

Low loss does not always mean the model is producing meaningful results.

Visual inspection matters.

Results

The main comparison was:

Frozen ResNet18 Encoder + Trainable Decoder
vs.
Fine-tuned Last ResNet Block + Trainable Decoder

The validation loss decreased from approximately:

0.0228 → 0.0139

This is about a 39% reduction in validation MSE.

Figure 2. Validation MSE comparison. Fine-tuning the last ResNet block achieved lower validation loss than using the frozen encoder only.

This shows that limited fine-tuning improved the model.

The result does not mean the model is perfect. The goal was not to reach state-of-the-art performance. The goal was to understand how architecture changes, transfer learning, backpropagation, and fine-tuning work in a real dense prediction task.

Qualitative Results

The numerical loss shows improvement, but visual inspection is also important.

Below is another prediction example from the fine-tuned model.

Each example contains:

Input image → Ground truth heatmap → Fine-tuned prediction

Figure 3. Another qualitative result. The prediction is still blocky, but it captures some general attention regions.

The predicted heatmaps are not perfect. They are still somewhat blocky and less smooth than the ground truth maps.

This is probably because the decoder is simple and uses transposed convolution layers, which can create checkerboard-like or blocky artifacts.

However, the model learned the general idea of predicting spatial attention regions from images.

What I Learned

Through this project, I learned:

• how to adapt a classification CNN into an encoder-decoder model
• how ResNet18 can be used as a feature extractor
• what frozen encoder training means
• what fine-tuning means
• how backpropagation updates only trainable layers
• why heatmap scaling matters
• why visual results should be checked together with numerical loss
• how a decoder upsamples feature maps into spatial predictions

The biggest takeaway is that understanding the training pipeline is more important than only looking at final performance.

Limitations

This project has several limitations.

First, I used only a small subset of the dataset.

Second, the decoder architecture is simple.

Third, the prediction maps are still blocky.

Fourth, I used MSE loss, which may not be the best loss function for saliency prediction.

A stronger version of this project could use:

• a larger dataset
• skip connections
• a smoother decoder
• saliency-specific metrics
• more training epochs
• better visualization and post-processing

Conclusion

In this project, I built a ResNet18 encoder-decoder model for human visual attention heatmap prediction.

I first trained only the decoder with a frozen ResNet18 encoder. Then I fine-tuned the last ResNet block from the frozen checkpoint.

The fine-tuned model achieved lower validation loss, showing that adapting the last block helped the model learn the attention prediction task better.

The project was not about achieving state-of-the-art performance. It was about understanding how a neural network architecture can be modified, trained, fine-tuned, and evaluated for a dense prediction problem.