Optimizing Your Deep Learning Journey: Understanding Optimizers

Authors : Dr. Rashmi Ashtagi, Bhushan Laddha, Bhushan Hatwar, Vedant Bhosle, Sakshi Bothe

In this article, we are focusing on the understanding of the topic “Deep Learning Optimizers”. We are discussing about Deep Learning, Gradient Descent, and all popular Deep Learning training algorithms. Let’s start the journey of Deep Learning.

Neural Network

What is Deep Learning?

A neural network with three or more layers is a deep learning, and deep learning is a subset of machine learning. Although they fall far short of being able to match the human brain’s capabilities, these neural networks make an effort to mimic its activity and enable it to “learn” from massive quantities of data. Despite the fact that a neural network with a single layer may still produce approximations, more hidden layers can assist to improve and optimize for accuracy.

Fig 1. Brain

Deep learning is the foundation for many artificial intelligence (AI) systems and services that enhance automation by carrying out mental and physical activities without the need for human participation.

Deep learning is the technology that powers both established goods and services (such as digital assistants, voice-activated TV remote controls, and credit card fraud detection) as well as cutting-edge innovations (like self-driving automobiles).

What Are Optimizers in Deep Learning?

In the realm of deep learning, optimizers are algorithms designed to fine-tune a model’s parameters during the training process, aiming to minimize a specified loss function. Their primary function is to facilitate the learning process of neural networks by iteratively adjusting weights and biases. Notable optimizers in this domain encompass Stochastic Gradient Descent (SGD), Adam, and RMSprop. Each optimizer is characterized by distinct update rules, learning rates, and momentum, all tailored to converge towards optimal model parameters, thereby enhancing overall performance.

Optimization algorithms, synonymous with optimizers, play a pivotal role in enhancing the performance of deep learning models. These algorithms significantly impact both the accuracy and speed of training for deep learning models.

What is Gradient Descent?

Gradient Descent is an optimization algorithm a technique used to learn weights.

Fig 2. Gradient Descent

The most crucial barometer used here is the loss function.

The algorithm tries to adjust weights which leads to finding the most suitable consequences. It works by adjusting the weights and checking the loss function, which helps in tuning the parameter.

Thus the accuracy of the model is guage with each iteration and the update of the parameter. This process, until the loss is 0 or equal to 0, or more iterations, leads to no improvements

This leads to the development of more accurate and robust deep-learning models.

But many times these processes could be a lot of resource-intensive processes, which increases difficulties in training the model.

To solve the above process, a number of optimization techniques have been proposed

Fig 3. Weights Learning step

Moment-based gradient descent

Let’s see an issue with earlier methods, In vanilla gradient descent, we only calculate the gradient at that point and eight update is decided by only two things learning rate and gradient at that point.

It leads to two major issues

The cost function’s gradient is minimal or zero at saddle points (plateaus), which results in little to no weight updating. As a result, learning ceases and the network becomes stagnant.

Even when running in mini-batch mode, Gradient Descent’s route is highly unsteady.

The above issues can very time limit us from reaching the global minimum.

To solve issues, Moment-based gradient descent is proposed.

For eg. any time while riding a bicycle, if there’s a downward slope, we do not need to do much work and the bicycle starts going down, but the slope ends and it reaches the plateau region even then it retains its momentum and for some distance, we continue to move forward without doing much work.

Moment-Based Gradient Descent works on similar principles.

Formula:

Fig 4. Moment-Based Gradient Descent

As per the formula in Fig 4, we can see we have brought momentum term, in the formula.

Here, we take an account of previous actions by taking the moving average over the past iterations. This allows it to have momentum, with the help of this, regions with high slopes would experience large updates, leading to faster convergence. In this method, more weightage is given to the recent weights, this allows it take make a difference between recent updates to earlier updates.

Nesterov-based gradient descent

The problem with Momentum-based gradient descent is such that, in the momentum, it many times overshoots the target region due to it being in momentum, which leads it to do oscillations a number of times.

Nesterov-based gradient descent proposes a solution to the above problem. It works on the principle loo before you leap.

As per the updated formula in Fig 4, it says, we are going to move at least γ.updatet-1 and then a bit more based on the value of the nΔwt gradient.

So instead of calculating the gradient at point t, we calculate the gradient at a lookahead point given by

Fig 5. Comparison of the gradient at w vs w lookahead

As per Fig 5, we can see how instead of calculating the gradient at point w, we calculate the gradient at point w lookahead, thus it helps us avoid moving farther from the target region.

Formula:

Fig 6. Nesterov-based gradient descent

This helps it by first moving in the direction of the momentum and then calculating the gradient at that look-ahead point.

This way it helps it to converge quicker than the moment-based gradient descent

Oscillations are smaller and chances of missing the target region also become smaller

Batch Gradient Descent

In batch Gradient Descent, all the data is taken into consideration before making an update. This leads to big complexity.

This is the most accurate, as this is the true gradient of the loss as it takes the average over all the points

Suppose we have a training of data of size 10 million, we calculate the gradient of every point and then make a change. This can be a lot of resource-intensive processes in plateau regions. Even if we are using earlier versions like moment-based and Nesterov-based gradient descent, updates in plateau regions are always slow.

Stochastic Gradient Descent

We have stochastic gradient descent to solve the above issue. In stochastic gradient descent (SGD), we only take into account one sample at a time. We take an example, provide it to the neural network, determine its gradient, update the weights using the gradient we computed, and then repeat these steps for all examples in our training dataset. But here, as we rely on one point for the gradient, it is not the true gradient but an estimation of the gradient.

You can think of it like tossing the coin only once and then on the basis of the result, telling the probability.

Sounds funny? Right

So, there is no guarantee that each step will decrease the loss

But the limitation here is that the cost will vary across the training instances and won’t definitely go down because we are just taking into account one example at a time. However, with time, you’ll notice that prices fluctuate less.

The cost will also never hit the minimum since it fluctuates so much, instead, it will keep circling around it. Stochastic Gradient Descent is applicable to bigger datasets. Large datasets result in more frequent parameter adjustments, which speeds up convergence.

Mini-Batch Gradient Descent

This algorithm solves the above problem which we see in the above case. There we rely only on one point, which provides an estimation, as all the points try to move in a favourable direction but that can harm others or the main objective. Here, to solve this problem instead of updating for every point we update it after a batch of points, this helps us to improve our estimation and thus provide a more suitable and robust optimization technique.

This optimization algorithm also gives rise to fluctuations, but they are less than stochastic gradient descent, thus making it more robust.

1 Epoch = 1 pass over training data

1 Step = One update of the parameters

N = Number of training points/ Size of training data

B = Mini-Batch size

Table 1. Update Rule

As per Table 1[3], we can see how parameter updates change as we move from batch gradient descent to mini-batch gradient descent.

Fig 7. Comparison of Fluctuations

As per fig 7[5], we can see that stochastic gradient descent has the most fluctuations while batch gradient descent has the least as here, we iterate over all points and then update the parameters, thus, we are doing this based on a true gradient as compared to the batch gradient descent where we update for each data point, thus only providing a poor estimate of gradient which may not hold good for all.

Stochastic Gradient Descent with momentum

2D representations of Cost vs weight

This method consistently outperforms the conventional Stochastic Gradient Descent (SGD) algorithm. The main issue with SGD is its susceptibility to oscillations, which hinders the ability to raise the learning rate, consequently prolonging the convergence process. In contrast, this approach leverages Exponentially Weighted Averages to calculate gradients, employing these gradients to update the parameters.

An equation[2] to update weights and bias in SGD with momentum.

Formula:

Fig 8. Update weights formula in SGD

In stochastic gradient descent (SGD) with momentum, we introduce a component that takes into account the past gradients when computing the current gradient. This means that the current gradient is influenced by its previous values, which effectively accelerates the convergence of the optimization process and helps to dampen oscillations in the parameter updates.

Fig 9. Convergence in SGD

Fig 10. Convergence in SGD with Momentum

The above Fig 9 and Fig 10, show how the convergence happens in SGD with momentum vs SGD without momentum. We can clearly see that Convergence in SGD with momentum is fast as compared to convergence in SGD.

Adagrad

Adagrad is a simpler optimizer than SGD with momentum because it doesn’t incorporate the concept of momentum. It operates on the principle of adapting individual learning rates for each parameter based on the number of iterations. The rationale for this approach is to assign higher learning rates to parameters associated with sparse features[2], as their infrequent occurrence necessitates a more substantial learning rate, in contrast to dense feature parameters.

Fig 11. Formula for Adagrad

The provided Adagrad optimizer equation is in Fig 11. demonstrates an inherent feature in which the learning rate undergoes automatic reduction. This happens because, with each time step, the summation of the past gradient squares consistently accumulates and tends to increase over time.

For better understanding purposes let’s take an example, consider a single neuron with 2 inputs and 1 output.

Adagrad will give-

Fig 12. Adagrad Formula

Stochastic Gradient Descent will be:

Fig 13. Stochastic Gradient Descent(SGD) Formula

Though the advantage of not needing manual learning rate tuning is clear, there is a significant drawback because of a monotonic decrease in learning rate at a certain point in the training process, the model’s learning rate becomes extremely small, causing it to effectively cease learning.

Adadelta

Adadelta is a modification of the Adagrad optimization algorithm designed to address the issue of rapidly decreasing learning rates over time. The core concept of Adadelta involves limiting the historical information used in computing the squared gradients. Instead of accumulating squared gradients from the beginning of training (1 to “t” time steps), Adadelta employs a moving window approach. This means that it only considers the squared gradients from the most recent, say, 10 time steps and averages them.

Fig 14. Adadelta Moving window approach Formula

The equation in Fig 14.[2] presented illustrates that as the number of time steps “t” increases, the summation of squared gradients denoted as “α” grows, causing a reduction in the learning rate “η.” To mitigate this exponential increase in “α,”

Fig 15. Adadelta Formula

Adadelta introduces the concept of exponentially weighted averages for squared gradients. As per Fig 15.[3], Unlike Adagrad, where “α” increases exponentially with each time step, Adadelta uses exponentially weighted averages to maintain better control over the growth of “Sdw.”

Adam Optimizer

The Adam optimizer is highly favored for its ability to combine the momentum principle from SGD with momentum” and the adaptive learning rate feature of “AdaDelta”.

Exponential Weighted Averages for past gradients:

Fig 16. Exponential Weighted Averages Formula

Exponential Weighted Averages for past squared gradients:

Fig 17. Exponential Weighted Averages for past squared gradients Formula

Using the above equation in Fig 17., the weight and bias updation formula looks like this:

Fig 18. Weight and Bias Update Formula

Bias correction in Exponential Weighted Averages is a technique aimed at improving initial estimates. However, many individuals opt not to employ bias correction because they prefer to allow some time to pass and reach a point where the bias error becomes negligible.

RMS Prop

RMS prop, an extension of the Rprop (Resilient Backpropagation) algorithm, has gained popularity in the deep learning community for its ability to address the issue of varying gradients. While not formally published, it is well-known for its effectiveness. Rprop, the predecessor of RMSprop, adjusts the step size individually for each weight by considering the sign of the gradient. However, it faces challenges with large datasets and mini-batch updates.

Fig 19. RMS Prop Formula1

RMSprop improves upon Rprop by maintaining a moving average of squared gradients for each weight and dividing the current gradient by the square root of this mean square. The algorithm aims to accelerate optimization by reducing the number of function evaluations needed to reach a local minimum. The formula for RMSprop involves a forgetting factor, gamma, and the weights are updated accordingly.

In practical terms, RMS prop proves advantageous when dealing with parameters causing significant oscillations in the cost function. For example, in a fish classification model where ‘color’ is a dominant factor leading to errors, RMSprop penalizes the ‘color’ parameter, encouraging reliance on other features. This prevents the algorithm from adapting too quickly to changes in ‘color’ compared to other parameters.

Fig 20. RMS Prop Formula2

Despite its benefits, RMS prop requires manual definition of the learning rate, and a universally suggested value may not be optimal for every application. Nevertheless, it offers faster convergence and demands less tuning compared to earlier gradient descent algorithms and their variants.

Conclusion

In summary, this blog has offered a thorough review of Deep Learning Optimizers, illuminating the key ideas of Deep Learning, Gradient Descent, and several widely used training methods. Numerous artificial intelligence applications, like personal assistants and self-driving cars, are powered by deep learning, which has numerous layers of neural networks.

With a focus on the relevance of the loss function and parameter tweaking in producing accurate and reliable models, the discussion of Gradient Descent presented the significance of optimization in deep learning. Numerous optimization techniques were investigated to address the problems with traditional gradient descent, including Moment-based gradient descent, Nesterov-based gradient descent, batch gradient descent, stochastic gradient descent, mini-batch gradient descent, and stochastic gradient descent with momentum.

Fig 21. Contour amp

As seen in above fig 21., we can see Adadelta, Adagrad are reaching convergence earlier. With research work being carried out, we can totally expect more optimizers to be developed which would make training of Large Language models more optimized. Here, each optimizer has its own pros and cons and one can decide which one to use based on the specific use case.

References

1.) https://www.ruder.io/optimizing-gradient-descent/

2.) https://medium.com/mlearning-ai/optimizers-in-deep-learning-7bf81fed78a0

3.) NPTEL IIT Ropar Deep Learning

4.) https://www.geeksforgeeks.org/ml-momentum-based-gradient-optimizer-introduction/

5.) https://www.analyticsvidhya.com/blog/2022/07/gradient-descent-and-its-types/