An Introduction to ResNets

Image by author using Sora

Residual networks, or ResNets for short, have played an important role in the computer vision-oriented deep learning community. In this essay, you’ll get a conceptual understanding of these networks and the challenges they provided responses to. However, we will avoid the underlying mathematics and keep explanations as simple as possible.

After reading this tutorial, you will…

• Understand why neural networks should improve performance with increasing depth… in theory.
• Understand why the shattering gradients problem results in degraded performance with depth, a.k.a. the degradation problem.
• Understand how ResNets reduce shattering gradients and yield better performance and what they look like architecturally/component-wise.

Are you ready? Let’s take a look!

Adding more layers should improve performance…in theory

During the AlexNet deep learning breakthrough in 2012, which reignited interest in the technology, many people realized that deep neural networks are feature learners.

This is best illustrated by comparing them to more classic approaches like logistic regression and Support Vector Machines. In these model types, machine learning engineers or data scientists had to deliberately and manually design features (i.e., model input columns). In other words, considerable feature analysis and, in many cases, feature selection were required, such as using PCA. When relevant features were chosen, they had to be engineered—for example, by applying appropriate filters to make them machine learning-compatible.

This was no longer true for neural networks. With the addition of convolutional layers, neural networks were able to learn to detect critical aspects of images that were important to classification results, such as making the filters/kernels learnable.

When numerous convolutional layers are stacked, the feature mappings learned in each succeeding layer grow more sophisticated. For example, the figure below depicts a ConvNet with multiple Conv layers and two Dense layers for future prediction. When displaying the filters/kernels of these ConvNets, low-level concepts (such as eyeballs) remain evident inside the images. These patterns become more generic as one moves further downstream.

Source:https://irishtechnews.ie/neural-networks-delivery-robots-with-brains-that-write-their-own-code/

When training a neural network to distinguish between cats and dogs, you can imagine that as you progress downstream, a more generic representation of cats and dogs emerges. The more generic the representation, the more cats are identified as cats and dogs as dogs. In other words, it should then become much easier for the dense layers to distinguish between the classes.

Obviously, this increases the appeal of deep learning. The term "deep" refers to the depth of the neural networks used, with deeper networks typically performing better. In theory, these networks should be capable of learning and approximating any function. But can they? Let’s look at some of the challenges that neural network practitioners encountered in the early days of deep learning.

From vanishing and exploding gradients to the “degradation problem.”

While practitioners soon realized that they needed to use nonlinear activation functions to allow their neural networks to work, they were mostly familiar with Sigmoid and Tanh. Remember from the high-level machine learning process that after a forward pass, the loss is computed backwards throughout the network, resulting in a gradient for each trainable parameter, and the network then adjusts in the direction of each individual gradient.

Backpropagation is performed throughout the layer stack using the chain rule—in other words, gradient multiplications. For example, the gradients in the third layer of the hierarchy away from the loss are influenced by the loss as well as the gradients computed for the two layers preceding it. This influence is expressed by means of multiplication.

Let’s look at the image below, which shows the Sigmoid activation function and its first derivative. The derivative has a maximum value of ~0.25 and is less than 0.1 throughout most of the domain.

Remember that this derivative is used to compute the gradient. Now consider what happens when you multiply these numbers across layers. If your network has 10 layers, the gradient at each layer is impacted by a multiplication with values on the order of 0.25⁹, which is a relatively small quantity. When applying these conventional activation functions, the gradient at the most upstream layers is almost infinitesimal, leading to extremely sluggish (or perhaps impossible) training. It’s commonly referred to as the vanishing gradients problem.

Fortunately, the vanishing gradients problem can be resolved by means of the ReLU activation function.

The converse of small gradients is also possible; that is, your model may suffer from the exploding gradients problem. This problem occurs when your data is not properly standardized, causing excessively large gradients during optimization. Multiplying these massive gradients causes gradients in the most upstream layers to become extremely large, potentially exceeding the maximum values for the data type (e.g., float), at which point they become NaN. That, in turn, causes model instability and, once again, the impossibility of periodically converging to a solution.

Batch Normalization, when paired with ReLU, effectively resolves the vanishing and exploding gradient problems.

He et al. (2016), however, identified a different issue. If we look at a graph cited in their paper, we can notice the mistake immediately:

Source: He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

When neural networks, like the one shown above, with a stack of neural layers (referred to as “plain networks” by He et al.), go deeper, both training and testing performance suffer.

Clearly, the 56-layer network performs worse than the 20-layer network.

This is extremely surprising, as theory predicts that deeper networks are stronger feature learners and hence perform better. When they replicated this difficulty using "identity mappings" in the 20-vs-56-layer comparison, for example, by training a network with 20 layers and another with 20 layers and 36 identity mappings that return their inputs, they encountered the same issue. Because network performance degrades, they dubbed it the degradation problem.

So, there is a difference between what theory predicts and what practice demonstrates. But why? Let’s look at another paper that examines the gradients of increasingly deep networks.

Shattering gradients problem

Balduzzi et al. analyze why ResNets (you’ll learn about them in the following section) of a specific depth perform better than plain networks of the same or even less depth in their study entitled “The shattered gradients problem: If ResNets are the answer, then what is the question?”

They built a neural network that can learn to map scalars to scalars, like 2 to 2 (the input domain was [-2, 2]). Please note that they argue that while the network itself is unlikely to be beneficial in practice, it is an excellent “laboratory” option for thoroughly investigating the topic.

The following graphic depicts what happens in progressively deep networks. The gradients of each input value, as well as the covariance of the gradients between the inputs, demonstrate the presence of structure in a shallow network (leftmost column). In other words, because close inputs yield comparable gradients, the network can gradually but steadily converge on a locally or globally optimum solution.

When the layer is made much deeper (the (b) column is moved to the left), this structure disappears, and the similarity between gradients resembles white noise. In other words, because certain inputs create considerable gradients, determining the best solution becomes more difficult as depth increases. Balduzzi et al. refer to this phenomenon as the shattering gradients problem, and they argue that it is one of the primary causes of the previously described degradation problem.

You can also examine the results of the study for a 50-layer ResNet. Clearly, the similarity between gradients as measured by covariance is lower than that of the 1-layer “plain” network, but it is considerably higher than that of the 24-layer plain network—and it is twice as deep! Indeed, as demonstrated by displaying autocorrelation for each model type (paper, page 3), ResNet gradient similarity is similar to brown noise.

In other words, ResNet gradients remain reasonably comparable as depth increases, eliminating the shattering gradients problem and enabling far deeper networks to be trained. Now that we know why they can be better, let’s have a look at how they appear. :)

Source: Balduzzi, D., Frean, M., Leary, L., Lewis, J. P., Ma, K. W. D., & McWilliams, B. (2017, July). The shattered gradients problem: If resnets are the answer, then what is the question?. In International Conference on Machine Learning (pp. 342–350). PMLR.

Introducing residual networks (ResNets)

He et al. (2016) found that neural networks could be made deeper while remaining performant by designing them in a framework they call residual learning. For this reason, they are known as residual networks, or ResNets for short.

If a stack of neural layers (such as the two layers in the illustration below) has to learn a mapping H(x), they can accomplish it simply by stacking. This produces the “plain network” situation, which, as you already know, can be troublesome as depth increases. However, it may also be expressed in a different way… by splitting H(x) into discrete components.

For example, suppose we let the stack of neural layers learn another mapping, F(x), where F(x) = H(x) — x. The original mapping may then be constructed using the equation H(x) = F(x) + x.

Interestingly, we can simply apply that mapping in our neural network by utilizing the idea of a skip connection. The input to our stack learning F(x) will simply be added to the stack’s output, resulting in F(x) + x, or the original mapping H(x). He et al. chose to do the element-wise addition before the stack’s last ReLU activation.

Source: He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

By stacking several of these “residual blocks” on top of one another, you may build a neural network that learns a succession of these deconstructed mappings. According to the Balduzzi et al. study, doing so causes gradients to become more comparable as depth increases, which benefits convergence to some optimum and so allows for better feature learning.

Residual network architectures

In this part, we will look at the different ResNet topologies and how shortcut connections are employed in networks.

The shortcut connections between layers are plainly seen in the ResNet-34 network. If you look closely, you’ll notice two distinct lines for the shortcut connections: one solid and one dotted. The dotted lines represent an increase in dimensions. You might wonder, “Why compare the three models side by side?”

ResNet-34 architecture alongside VGG 19 and a plain 34-layer neural network (Source).

VGG 19 contains 19 layers in total and 19.6 billion FLOPs, which is a large quantity. The ordinary neural network comprises 34 layers and just 3.6 billion FLOPs. The ResNet-34 with shortcut links has 3.6 billion FLOPs. You see, shortcut connections do not add to the computation of a neural network, but they undoubtedly offer additional benefits.

Along with the 34-layered ResNet,there are ResNet variations for 18, 50, 101, and 152 layers. Take a look at the figure below to get a better sense.

Figure showing different ResNet architectures according to the number of layers. ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152 (Source).

The above figure has an intriguing detail. Even one of the biggest Residual Neural Network architectures, ResNet-101, has less FLOPs (11.3 billion) than VGG-19. This fact will become even more intriguing when we examine the findings of Residual Neural Networks in the next section.

Summary

In today’s post, we learned that ResNets are residual networks that revolutionized deep learning by enabling the training of much deeper neural networks. They were introduced by He et al. in 2016 to solve critical problems that arose when making neural networks deeper.

The key problems ResNets solve are two-fold: 1) the degradation problem — contrary to theory, simply making neural networks deeper actually made them perform worse, not better. Even 20-layer networks outperformed 56-layer networks. 2) The Shattering Gradients Problem — in very deep networks, gradients become chaotic and lose structure, resembling white noise rather than organized patterns. This makes it extremely difficult for the network to converge to optimal solutions.

We discussed how ResNets work. Instead of learning a direct mapping H(x), ResNets decompose it into F(x) = H(x) — x (learned by the neural layers) and H(x) = F(x) + x (reconstructed using skip connections).

The key innovation is skip connections — direct pathways that add the input to the output of neural layer stacks. This creates “residual blocks” that can be stacked to build very deep networks.

We learned that skip connections help maintain gradient similarity across layers, reducing the shattering gradients problem. This allows networks to be much deeper while still training effectively and achieving better performance.

ResNets remain widely used in computer vision today, demonstrating their lasting importance in enabling the practical training of very deep neural networks. I hope you’ve learned something from today’s post. In my next post, we will implement a ResNet from scratch using TensorFlow2 and Keras. Any comments, questions or suggestions are weclome. Thank you for reading!!

References

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

Balduzzi, D., Frean, M., Leary, L., Lewis, J. P., Ma, K. W. D., & McWilliams, B. (2017, July). The shattered gradients problem: If resnets are the answer, then what is the question?. In International Conference on Machine Learning (pp. 342–350). PMLR.

Krizhevsky, A. (n.d.). CIFAR-10 and CIFAR-100 datasets. Department of Computer Science, University of Toronto. https://www.cs.toronto.edu/~kriz/cifar.html

Jung, J. K. (2018, April 16). Extending Keras’ ImageDataGenerator to support random cropping. JK Jung’s blog. https://jkjung-avt.github.io/keras-image-cropping/

TensorFlow. (n.d.). Tf.keras.applications.resnet50.preprocess_input. https://www.tensorflow.org/api_docs/python/tf/keras/applications/resnet50/preprocess_input

A Message from AI Mind

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the AI Mind Publication
• 🧠 Improve your AI prompts effortlessly and FREE
• 🧰 Discover Intuitive AI Tools