What Happens Inside an AI Vision Transformer Model When It Gets Tricked?

Artificial intelligence has become incredibly good at understanding images.

Your phone can recognize faces. Self-driving cars can identify pedestrians. Apps can detect objects in photos almost instantly.

But there’s something surprising about these systems.

Sometimes, you can make tiny, almost invisible changes to an image, and the AI suddenly gets completely confused.

Example of how adversarial perturbations can cause AI models to misclassify images even when the changes are barely visible.

A picture of a cat might suddenly be classified as an airplane.

To humans, the image still looks exactly like a cat.

To the AI, something went very wrong.

This strange behavior is called an adversarial attack, and it’s one of the biggest weaknesses in modern AI systems.

The Question That Started My Project

Most researchers study these attacks by measuring one thing:

How much the model’s accuracy drops.

But while working on my project, I started wondering something different.

Instead of only asking how wrong the model becomes, I wanted to ask:

What actually changes inside the model when it gets tricked?

Does the model start looking at the wrong parts of the image?

Does its “attention” shift in strange ways?

To explore this idea, I studied a newer type of AI model called a Vision Transformer and tried to understand how it behaves when someone tries to fool it.

How AI Models Look at Images

Older image recognition models work a bit like scanners.

They slowly analyze small parts of an image, piece by piece, until they understand what’s in it.

Vision Transformers work differently.

Instead of scanning small areas step-by-step, they look at the entire image at once and decide which parts are important.

Simplified pipeline used in the experiment to evaluate adversarial robustness and attention behavior in Vision Transformers.

You can imagine it like this:

When you look at a photo of a dog, your eyes naturally focus on certain areas, maybe the face or the body.

AI models do something similar. They place attention on different parts of the image to figure out what it is.

This made Vision Transformers perfect for my experiment.

Because if the model gets fooled, we can actually see how its attention changes.

The Experiment

To understand what happens during an attack, I ran a series of tests.

First, I trained AI models to recognize images from a dataset of everyday objects like cars, animals, and airplanes.

When the images were normal, the models worked extremely well, correctly identifying objects almost all the time.

But then I introduced adversarial attacks.

These attacks slightly modified the images in ways that humans would barely notice.

And suddenly, the AI started making mistakes.

Sometimes very confident mistakes.

Something Interesting Happened

The accuracy drop was expected.

But when I looked at the attention patterns inside the Vision Transformer, something fascinating appeared.

Normally, the model spreads its attention across the important parts of the object.

For example:

If the image contains a ship, the model focuses on the structure of the ship.

But when the image was attacked, the model’s attention shifted.

Instead of looking at the ship, it might focus on a random corner of the image or a meaningless patch of pixels.

Visualization showing how attention shifts (FGSM) when an adversarial attack is introduced.

Stronger attacks (PGD) cause attention collapse, where the model focuses on very small manipulated regions.

It was almost like the attack had hijacked the model’s focus.

And once the model started paying attention to the wrong areas, it made the wrong prediction.

Can We Fix This?

The next step was to see if the model could be trained to resist these tricks.

So I trained the model again but this time, I included attacked images during training.

This approach teaches the model to recognize and handle manipulated inputs.

After doing this, the results improved a lot.

Adversarial training helps stabilize attention patterns and reduces attention drift.

The model became much better at handling attacked images, even though it lost a small amount of accuracy on normal images.

More importantly, the attention patterns became more stable.

Instead of jumping to random parts of the image, the model kept focusing on the correct regions.

Why This Matters

Adversarial attacks might sound like an academic problem, but they have real-world implications.

Imagine:

• A self-driving car misreading a stop sign • A security system misidentifying a person • A medical AI misclassifying a scan

Understanding why models fail is just as important as making them accurate.

Attention stability improves after adversarial training, showing more consistent focus across layers.

And studying attention patterns gives us a window into the model’s reasoning process.

The Big Takeaway

What I learned from this project is simple:

AI models don’t just fail randomly.

When they get fooled, their focus shifts.

They start paying attention to the wrong things.

By studying those attention changes, we can better understand how AI systems behave and how to make them more reliable.

Looking Ahead

This research opens the door to some interesting possibilities.

In the future, AI systems might be able to detect attacks automatically by noticing strange attention patterns.

Or they might monitor their own attention to make sure they’re focusing on the right things.

If we can teach AI to keep its attention where it belongs, we might be able to build systems that are not only powerful but also trustworthy.

For Further Reading

For readers interested in the technical implementation, evaluation results, and detailed methodology, you can read our full paper and code here:

Attention-Driven-Adversarial-Defense-Framework-for-Vision-Transformers