Classic Vision Neural Networks : History of Neural Network Architecture

TImeline

If you haven’t checked out my previous blogs on this topic I would recommend checking them out:

• ***What are CNNs ?***

In this article we will talk about

1. LeNet-5
2. AlexNet
3. VGG
4. ResNet

General Introduction

LeNet-5 network which came from, I guess, in 1980s, AlexNet which is often cited in the VGG network. These are examples of pretty effective neural networks, and you see ideas from these papers that will probably be useful for your own work as well. Then I want to show you the ResNet or called residual network.

After seeing these neural networks, I think we have much better intuition about how to build effective convolutional neural networks. Even if you end up not working computer vision yourself, you find a lot of the ideas from some of these examples, such as ResNet, Inception network, many of these ideas are cross fertilizing, are making their way into other disciplines. Even if you don’t end up building computer vision applications yourself, I think you’ll find some of these ideas very interesting and helpful for your work.

LeNet-5

source : https://www.philschmid.de/getting-started-with-cnn-by-calculating-lenet-layer-manually

The idea of CNNs is intelligently adapt to the properties of images by reducing the dimension. Initially, the goal of LeNet-5 was to recognize handwritten digits, so maybe an image of a digits like that and LeNet-5 was trained on grayscale images, which is why it’s 32 by 32 by 1.To achieve this convolutional layer and pooling layer are used. Convolutional layers are reducing the dimensions by adding filters (kernel windows) to the Input. The dimension can reduce by applying kernel windows to calculate new outputs.

Back then when this paper was written in 1998, people use average pooling much more. If you’re building a modern variant, you probably use max pooling instead. But in this example, you average pool and with a filter width two and a stride of two, you wind up reducing the dimensions, the height and width by a factor of two, so we now end up with a 14 by 14 by 6 volume.

This neural network was small by modern standards, had about 60,000 parameters. And today, you often see neural networks with anywhere from 10 million to 100 million parameters, and it’s not unusual to see networks that are literally about a thousand times bigger than this network

Note :

One other pattern you see in this neural network that’s still often repeated today is that you might have some one or more convolution layers followed by pooling layer, and then one or sometimes more than one convolution layer followed by a pooling layer, and then some fully connected layers and then the outputs. So this type of arrangement of layers is quite common.

Link : Paper

AlexNet

AlexNet, named after Alex Krizhevsky, who was the first author of the paper describing this work.

Architecture Overview

AlexNet’s architecture consisted of five convolutional layers, followed by three fully connected layers. The network was significantly deeper than its predecessors, with a total of eight layers and millions of parameters. This depth, combined with the use of powerful GPU computing, allowed AlexNet to learn rich and complex feature representations from raw image data.

Key Innovations

1. Relu Activation: AlexNet introduced the use of the Rectified Linear Unit (ReLU) activation function, which helped mitigate the vanishing gradient problem faced by traditional activation functions like sigmoid and tanh. ReLU enabled faster training and improved performance.
2. Max Pooling: Instead of average pooling used in earlier architectures like LeNet, AlexNet employed max pooling, which captures the most salient features while reducing spatial dimensions.
3. Data Augmentation: To reduce overfitting and improve generalization, AlexNet employed data augmentation techniques, such as random image crops, flips, and color shifts, effectively expanding the training dataset.
4. Dropout Regularization: AlexNet incorporated dropout regularization, a technique that randomly drops units during training, preventing co-adaptation and reducing overfitting.
5. GPU Computing: Leveraging the parallel computing power of GPUs, AlexNet could be trained efficiently on large datasets, a capability that was not readily available before.

Impact and Legacy

So this neural network actually had a lot of similarities to LeNet, but it was much bigger. So whereas the LeNet-5 from previous slide had about 60,000 parameters, this AlexNet that had about 60 million parameters. And the fact that they could take pretty similar basic building blocks that have a lot more hidden units and training on a lot more data, they trained on the image that dataset that allowed it to have a just remarkable performance.

AlexNet’s performance on the ImageNet challenge, with a top-5 error rate of 15.3%, significantly outperformed the previous state-of-the-art, which had an error rate of around 26%. This remarkable achievement sparked a renewed interest in deep learning and CNNs, propelling their adoption in various computer vision applications, including object detection, image segmentation, and video analysis.

Today, AlexNet serves as a foundational architecture, inspiring numerous variations and advancements in CNN design. Its success paved the way for more sophisticated architectures like VGGNet, ResNet, and Inception, which have pushed the boundaries of computer vision even further.

While AlexNet may seem relatively simple by today’s standards, its impact on the field of computer vision and deep learning cannot be overstated. It ushered in a new era of deep learning-based approaches, revolutionizing how machines perceive and understand visual data.

VGG - 16

VGG Architecture

Introduced by researchers at the University of Oxford, Visual Geometry Group (VGG), the VGG-16 architecture made a significant impact on the field of computer vision and deep learning. Building upon the success of AlexNet, VGG-16 demonstrated the power of depth and simplicity in convolutional neural networks (CNNs) for image recognition tasks.

Architecture Overview

VGG-16 consists of 16 convolutional layers, organized into five blocks, followed by three fully connected layers. The convolutional layers use small 3x3 filters, which are convolved multiple times to capture larger receptive fields. This unique approach, combined with the network’s depth, allowed VGG-16 to learn increasingly complex and abstract representations of visual data.

Key Innovations

1. Depth and Simplicity: VGG-16 emphasized depth over complexity, stacking multiple 3x3 convolutional layers with a consistent architecture throughout the network. This simple yet powerful design demonstrated the effectiveness of depth in capturing intricate visual patterns.
2. Small Convolutional Filters: Unlike AlexNet, which used larger filters, VGG-16 employed small 3x3 convolutional filters. These smaller filters captured low-level features more efficiently and allowed for deeper stacking, enabling the network to learn increasingly complex representations.
3. Max Pooling: Similar to AlexNet, VGG-16 utilized max pooling layers to progressively reduce spatial dimensions and capture the most salient features.
4. Computational Efficiency: Despite its depth, VGG-16 maintained computational efficiency by using a fixed convolution filter size throughout the network, reducing the number of hyperparameters and facilitating easier training.

Impact and Legacy

VGG-16 achieved impressive results on the ImageNet dataset, with a top-5 error rate of 7.3%, significantly outperforming AlexNet’s performance. This success demonstrated the power of depth and simplicity in CNN architectures, inspiring subsequent developments in deep learning for computer vision.

Furthermore, the VGG architecture’s simplicity and modularity made it highly versatile, enabling its use as a feature extractor or as a base model for transfer learning in various computer vision tasks, such as object detection, image segmentation, and style transfer.

While VGG-16 has been surpassed in performance by more recent architectures, its impact on the field of deep learning cannot be overstated. It laid the foundation for deeper and more complex CNN architectures, paving the way for groundbreaking models like ResNet and Inception.

Today, VGG-16 remains a popular architecture for teaching and understanding the principles of deep convolutional neural networks, serving as a fundamental building block in the ever-evolving field of computer vision and deep learning.

Next, let’s go beyond these classic networks and look at some even more advanced, even more powerful neural network architectures.

Link to next article will be available soon ( Hint : It will be on ResNets )

What are ResNets ?

I feel extremely happy sharing all this knowledge and do let me know if this article has helped you.

Thank you for reading, I hope this article helped you

Aditya Raghuvanshi ( IIIT Hyderabad, INDIA )

Connect me on the following :

Github | linkedin | Medium | Gmail : tanalpha.aditya@gmail.com