How CNNs Learn to See What Radiologists See

A look at why convolutional neural networks are surprisingly well-suited for medical image analysis and what they’re actually detecting under the hood.

Architecture of CNN

When I first built a brain tumour classifier, I didn’t fully understand what the model was actually learning. I knew the architecture, I knew the accuracy, but I couldn’t answer the question“What spatial features is CNN actually detecting?”

That question pushed me to go deeper. And what I found changed how I think about CNNs entirely not as black boxes, but as systems that build a visual vocabulary layer by layer the same way a radiologist does over years of training.

Why You Can’t Just Flatten an Image

Before CNNs, the obvious approach to image classification was to flatten the image into a 1D vector and feed it into a regular neural network.

A 256×256 grayscale MRI scan becomes 65,536 input values. That’s already unwieldy, but the bigger problem is what you lose: spatial relationships.

When you flatten an image, the pixel at position (10, 10) and the pixel at position (11, 10) which are neighbours become just two arbitrary numbers in a long list. The model has no way of knowing they were ever next to each other.

For natural images that’s bad. For medical images, where where an anomaly sits and what surrounds it is clinically meaningful, it’s a fundamental failure. CNNs were designed specifically to fix this.

Filters: Small Windows Looking for Patterns

The core idea behind a CNN is simple instead of looking at the whole image at once use a small window called a filter or kernel that slides across the image and looks for one specific pattern.

A 3×3 filter might be trained to detect a horizontal edge. Another detects a vertical edge. Another picks up a specific texture — the irregular, grainy texture of abnormal tissue for example. At each position the filter visits it performs a dot product between its weights and the pixels underneath it producing a single number that says “how strongly is this pattern present here?” Slide the filter across the entire image and you get a feature map a spatial record of where that pattern showed up.

This is convolution. The network figures out which patterns are worth looking for.

How Layers Build a Visual Vocabulary

One filter looking for one pattern isn’t very useful. The power comes from stacking layers.

Early layers learn low level features edges corners intensity gradients. These are almost universal the first-layer filters of CNNs trained on completely different tasks often look similar.

Middle layers combine those low-level features into shapes curvesand circular boundaries. A middle layer neuron might activate strongly when it sees an edge curving into a roughly circular region.

Deep layers detect highlevel task specific structures. In a medical imaging model, these neurons learn to respond to things like an irregular asymmetric boundary and smooth well-defined border .

This is hierarchical feature learning. And it mirrors at a structural level how radiologists actually develop expertisefirst learning to see edges and densities then shapes and then identifying meaningful patterns built from those shapes.

VGG16: Why This Architecture

Why I Chose VGG16

Not all CNNs are built the same. Architectures like ResNet and Inception use complex branching structures and skip connections. VGG16 does something almost stubbornly simple it stacks sixteen layers of small 3×3 filters, one after another, going progressively deeper.

That simplicity is actually its strength for medical imaging.

Those small 3×3 filters mean the network is always looking at local patches tiny neighbourhoods of pixels at a time. As you go deeper, each layer of the image is built from the combined output of the layers before it. By layer 10, the network isn’t seeing raw pixels anymore — it’s seeing combinations of edges which form textures which form shapes which form structures.

For brain MRI scans specifically, this depth matters. The difference between a tumour and healthy tissue isn’t always obvious at the pixel level. It shows up in boundary irregularity, asymmetry, and texture gradients — features that only become detectable when you’ve built enough layers of abstraction.

VGG16 was originally trained on ImageNet — 1.2 million natural photographs. What makes transfer learning work here is that the low and mid-level features it learned there edges, curves and texture patterns are genuinely universal

The result a model that arrives at medical image analysis already knowing how to see structureand only needs to learn which structures matter clinically.

Why Medical Images Are a CNN Sweet Spot

This is the part I find genuinely interesting. CNNs aren’t just usable for medical images the structure of medical imaging data maps almost perfectly onto what CNNs are designed to exploit.

Local structure matters. A tumour occupies a region of the image not the whole thing. CNNs process local patches so they’re naturally focused on regions rather than global statistics.

Position shouldn’t matter. A tumour in the top-left of an MRI is clinically the same as a tumour in the bottom-right. It detects same pattern regardless of the position it appears.

A plain neural network sees 65,536 independent numbers. A CNN sees structure, locality, and hierarchy. For medical images that difference is everything.

What My Model Actually Learned

The heatmap wasn’t scattered randomly across the MRI. It was concentrated on the tumour region specifically on its boundary. The model had learned, without being told, that the edge characteristics of the region were the most diagnostically relevant feature.

That’s not just good accuracy. That’s the model learning something medically meaningful.