The Mathematics Behind LeNet & AlexNet: Understanding the equations that power Convolutional Neural Networks

Introduction

Convolutional Neural Networks (CNNs) have transformed computer vision by enabling machines to automatically learn features from images. Among the many CNN architectures developed over the years, LeNet and AlexNet stand out as two milestone innovations.

LeNet, proposed by Yann LeCun in 1998, introduced the fundamental principles of convolutional neural networks for handwritten digit recognition. More than a decade later, AlexNet demonstrated the enormous potential of deep learning by winning the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012.

While these architectures are often studied from a structural perspective, their true power comes from the mathematical operations performed at every layer. Understanding these mathematical foundations helps explain how CNNs extract features, learn patterns, and classify images with high accuracy.

This article explores the core mathematics behind LeNet and AlexNet, including convolution operations, kernel size selection, padding, stride, activation functions, parameter analysis, and computational complexity.

LeNet and AlexNet at a Glance

LeNet was designed primarily for grayscale handwritten digit recognition. It consists of seven layers and introduced concepts such as convolution, subsampling (pooling), and fully connected layers.

LeNet Architecture

AlexNet expanded these ideas into a much deeper architecture capable of handling large RGB images. It introduced several innovations including ReLU activation functions, dropout regularization, GPU-based training, and data augmentation.

AlexNet Architecture

Although separated by fourteen years, both networks rely on the same mathematical foundations.

1. Convolution Operation

The convolution operation is the heart of every CNN.

Rather than processing the entire image at once, CNNs use a small matrix called a kernel or filter that slides across the image. At each position, the kernel multiplies its values with the corresponding image pixels and sums the results.

The mathematical representation of convolution is:

Where:

• I represents the input image
• K represents the kernel
• x and y indicate pixel positions

This operation enables CNNs to detect important visual patterns such as:

• Edges
• Corners
• Textures
• Shapes

In LeNet, convolution layers detect simple digit features. In AlexNet, deeper convolution layers learn complex object structures such as animal faces, vehicle components, and background patterns.

2. Kernel Size Effects

A kernel determines the receptive field of a convolution layer.

Common kernel sizes include:

• 3×3
• 5×5
• 11×11

Smaller kernels focus on local features while larger kernels capture more contextual information.

Small Kernels (3×3)

Advantages:

• Fewer parameters
• Lower computational cost
• Better feature localization

Used extensively in modern CNNs.

Large Kernels (11×11)

Advantages:

• Capture larger image regions
• Faster reduction of spatial dimensions

AlexNet’s first convolution layer used an 11×11 kernel because it processed large 227×227 RGB images.

However, larger kernels require significantly more parameters and computations.

3. Padding Mathematics

Without padding, every convolution operation reduces image size.

For example:

Input = 32 × 32

Kernel = 5 × 5

Output becomes:

32 − 5 + 1 = 28

To preserve information near image borders, CNNs use padding.

The output size equation is:

Where:

• N = Input Size
• K = Kernel Size
• P = Padding
• S = Stride

Padding adds extra zeros around the image boundary, preventing excessive shrinkage of feature maps.

AlexNet uses padding in several convolution layers to preserve important spatial information.

4. Stride Mathematics

Stride determines how many pixels a kernel moves after each convolution.

Stride = 1

The kernel moves one pixel at a time.

Advantages:

• More detailed feature extraction
• Higher accuracy

Stride = 2

The kernel skips pixels.

Advantages:

• Smaller output dimensions
• Faster computation

Using the output formula:

For:

• Input = 32
• Kernel = 5
• Padding = 0
• Stride = 1

Output:

O=28

For Stride = 2:

O=14

This significantly reduces computational requirements.

5. ReLU Mathematics

One of AlexNet’s most important innovations was replacing sigmoid activations with the Rectified Linear Unit (ReLU).

The ReLU function is defined as:

Behavior:

• Negative values become zero
• Positive values remain unchanged

Why ReLU Matters

ReLu Activation function

Traditional sigmoid functions suffer from the vanishing gradient problem.

ReLU helps by:

• Increasing training speed
• Improving gradient flow
• Enabling deeper networks

This innovation was one of the key reasons behind AlexNet’s success.

6. Pooling Calculations

Pooling reduces feature map dimensions while retaining important information.

The most common method is Max Pooling.

Consider a 2×2 region:

5  2
8  1

Max Pooling selects:

8

Benefits:

• Reduces computations
• Reduces memory usage
• Improves translation invariance
• Helps prevent overfitting

Both LeNet and AlexNet use pooling layers after convolution layers

7. Parameter Count Analysis

The number of trainable parameters determines a model’s learning capacity.

LeNet

Approximately: 60,000 Parameters

Suitable for:

• Handwritten digit recognition
• Small datasets
• Limited computing resources

AlexNet

Approximately: 60 Million Parameter

Suitable for:

• Large-scale image classification
• Complex visual patterns
• High-dimensional datasets

AlexNet contains roughly 1000 times more parameters than LeNet, allowing it to learn significantly richer feature representations.

8. Computational Complexity

The computational cost of a convolution layer is:

Where:

• H = Output Height
• W = Output Width
• Cin = Input Channels
• Cout = Output Channels
• Kh = Kernel Height
• Kw = Kernel Width

From this equation we observe:

Increasing:

• Image size
• Kernel size
• Number of filters

directly increases computational cost.

This is why AlexNet required GPU acceleration while LeNet could be trained on comparatively modest hardware.

9. Impact and Legacy

The mathematical concepts introduced in LeNet and expanded in AlexNet continue to influence modern deep learning.

These ideas paved the way for:

• VGGNet
• GoogLeNet
• ResNet
• DenseNet
• EfficientNet
• Vision Transformers

Even today, convolution, ReLU activation, pooling, and parameter optimization remain fundamental concepts in computer vision systems.

The progression can be viewed as:

LeNet (1998) → AlexNet (2012) → VGGNet (2014) → ResNet (2015) →Vision Transformers (2020+)

Personal Learning Experience

While studying deep learning architectures, I found that understanding the mathematics behind CNNs provided much deeper insight than simply memorizing layer structures. Learning how convolution, ReLU, pooling, and parameter scaling work mathematically helped me appreciate why AlexNet became a breakthrough model and how modern computer vision architectures evolved from these foundational ideas.

Conclusion

LeNet and AlexNet are more than historical CNN architectures — they are mathematical frameworks that transformed computer vision.

From convolution operations and kernel design to ReLU activation and computational optimization, every component is built upon carefully designed mathematical principles. LeNet established the foundation, while AlexNet demonstrated how scaling these concepts could achieve unprecedented performance.

Understanding these mathematical foundations not only helps us appreciate the evolution of deep learning but also provides the knowledge necessary to design and improve future neural network architectures.

References

1. Yann LeCun et al. — Gradient-Based Learning Applied to Document Recognition (LeNet)
2. Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton — ImageNet Classification with Deep Convolutional Neural Networks
3. Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
4. Stanford CS231n: Convolutional Neural Networks for Visual Recognition
5. TensorFlow and PyTorch Documentation