Deep Learning Series 09: Optimizers for Optimization

Previous Blog

[embed]Deep Learning Series 08: Gradient Descent Techniques And Limitations Gradient Descent is known as one of the most commonly used optimization algorithms to train machine learning models by…medium.com

In machine learning, an optimizer is an algorithm used to minimize (or maximize) an objective function, often referred to as a loss function or cost function. Optimizers play a crucial role in training models by adjusting the model’s parameters (weights and biases) to minimize the error between the model’s predictions and the true outputs.

Photo by NASA on Unsplash

In Deep learning, we can use different optimizers for a paricular use case, each optimizer techinque has its own limitation. let’s discuss briefly..

1. SGD With Momentum

• Momentum is an extension to the gradient descent optimization algorithm, often referred to as gradient descent with momentum.
• It is designed to accelerate the optimization process, e.g. decrease the number of function evaluations required to reach the optima, or to improve the capability of the optimization algorithm, e.g. result in a better final result.
• Momentum adds history to the parameter update equation based on the gradient encountered in the previous updates.

Formula looks like:-

In momentum we have a value called Vt−1,which is a history component. stores the history of data in order to move faster to the minimum.

Here we are adding derivative of weight and history of weight,then we are updating our weights by wt+1.(note wt+1 is new weight)

Momentum gradient descent oscillates in and out when reaches near to the minima,which means it will take many turns. Despite taking many turns still reach in minima faster than vanilla gradient descent.

Graphical Intuition of Momentum

In this graph you can see that weights updation is faster. but their is one big problem in this Algorthm where you are oscillating more frequently in order to reach minima.

so this is one disadvantage with Momentum Gradient Descent. One thing we can do in order to reach minima by fine tuning the parameter Gamma we can reduce the oscillation.

Nesterov Accelerated Gradient(NAG)

Nesterov Accelerated Gradient is a gradient descent technique, which is similar to the one which we have studied i,e Momentum. So only difference in this algorithm is we are taking a history component vt-1 in current epoch we are adding a history and then we calculate derivative of the weights, after we are taking step in direction of derivative, and finally we are

storing a history component for the next iteration.

Where formula looks like :

Loop

1. Add Vt-1 with a current weight Wt

2. Calculate the derivative of Wtemp

3. Update the weight(which means take another step) Wt+1

4. Calculate the new history component Vt

Graphical Intuition of NAG

NAG is efficient than Momentum because it actually reduces the number of oscillations as compared to Momentum.

It also have same problem we are facing in Momentum. we want directly to reach minima instead of oscillating.

Adagrad ( Adaptive Gradient Descent)

Adagrad, short for Adaptive Gradient Algorithm, is an optimization algorithm designed to adaptively adjust the learning rates of individual parameters during training in order to perform larger updates for infrequent parameters and smaller updates for frequent parameters.

Working Principle of Adagrad:

• Adaptive Learning Rates:
• Adagrad adjusts the learning rates of each parameter based on the historical gradients observed for that parameter.
• Accumulation of Squared Gradients:
• It maintains a per-parameter learning rate that is inversely proportional to the square root of the sum of the squares of past gradients for that parameter.
• Parameter Update:

where n is the global learning rate, ϵ is a small constant (typically added for numerical stability to avoid division by zero).

where vt is a square of gradient and history component

Advantages:

• Automatic Learning Rate Adjustment: Adagrad adapts the learning rates of individual parameters based on their historical gradients. It inherently performs larger updates for parameters with infrequent updates and smaller updates for frequently updated parameters.
• Sparse Feature Handling: It is beneficial for sparse data and settings where certain features are rarely present.

Limitations:

• Accumulation of Squared Gradients: As Adagrad accumulates the squared gradients over time, the learning rates can become extremely small for frequently occurring parameters, causing the algorithm to stagnate prematurely.
• Monotonic Learning Rate Decrease: The learning rates continually decrease, which can lead to very slow convergence or premature stopping in some cases.

Graphical Intuition of Adagrad

In this graph you can see AdaGrad is converge much faster than than vanilla gradient descent because it adjusts the learning rate based on input data given.But problem in adagrad is, in some cases there wont be any updation in parameter because gradients are high which affects the learning rate hence updation is close to zero which means almost no change.

To address some of Adagrad’s limitations, variants like RMSprop and Adam have been developed. RMSprop incorporates a decay term to limit the accumulation of past gradients, while Adam combines momentum with adaptive learning rates to improve performance.

RMSProp

RMSProp, short for Root Mean Square Propagation, is an adaptive learning rate optimization algorithm designed to address some limitations of Adagrad, specifically its aggressive and monotonically decreasing learning rates.

Working Principle of RMSProp:

• Adaptive Learning Rates:Like Adagrad, RMSProp also adapts the learning rates of individual parameters based on the magnitude of their gradients.
• Exponentially Weighted Moving Average (EWMA) of Squared Gradients:Instead of accumulating all past squared gradients as Adagrad does, RMSProp uses an exponentially weighted moving average to compute the average of the squared gradients.
• Parameter Update:

Importance of Current and Previous Weights:

• Exponential Moving Averages of Gradients:RMSProp maintains an EWMA of the squared gradients for each parameter. This moving average represents a smoothed-out estimation of the recent history of gradients.The EWMA gives more weight to recent gradients while discounting older gradients, effectively placing more emphasis on the current behavior of gradients rather than historical behavior.
• Adaptation to Current Gradients:Parameters associated with current gradients that are significant or changing rapidly will have a greater impact on the EWMA of squared gradients.If the current gradients for a parameter are consistently low, RMSProp’s EWMA calculation will ensure that the squared gradients for that parameter remain relatively low. This, in turn, would adjust the learning rate to be higher for such parameters, allowing for larger updates and faster learning.
• Balancing Historical Information:While RMSProp discounts older gradients by assigning less weight through the exponential decay, it still maintains some memory of past gradients.This balance allows RMSProp to adapt to changing gradients and allocate learning rates based on the recent behavior of gradients while not completely ignoring the historical trends.

Advantages of RMSProp:

• Mitigating Rapid Decrease in Learning Rates: RMSProp uses an exponentially weighted moving average, preventing the aggressive, monotonically decreasing learning rates seen in Adagrad.
• Adaptive Learning Rates: It adapts learning rates based on recent squared gradients, allowing for better convergence. it will give less importance for previous gradients.
• Memory Efficiency: Unlike Adagrad, RMSProp accumulates a decaying average of past gradients, which helps in handling the exploding and vanishing gradient problems more effectively.

Limitations:

Hyperparameter Sensitivity: Like many adaptive algorithms, RMSProp requires careful tuning of hyperparameters such as the learning rate, decay rate β, and the small constant ϵ.

ADAM:

Adam (short for Adaptive Moment Estimation) is an optimization algorithm that combines the benefits of both momentum and adaptive learning rates, aiming to overcome some limitations of other optimization techniques like Adagrad and RMSProp. It’s known for its efficiency in training deep neural networks and is widely used in machine learning and deep learning applications.

Working Principle of Adam:

• Adaptive Learning Rates:Like RMSProp, Adam computes individual adaptive learning rates for each parameter.
• Exponentially Weighted Moving Averages (EWMA) of Gradients and Their Squares:Adam keeps track of two exponentially decaying averages: the first moment (mean) of the gradients and the second moment (uncentered variance) of the gradients.The first moment is an EWMA of gradients, while the second moment is an EWMA of squared gradients.
• Bias Correction:Adam performs bias correction to adjust for the fact that the EWMA estimates are biased toward zero, especially during the initial time steps.
• Parameter Update:

Advantages of Adam:

• Adaptive Learning Rates: It adaptively adjusts the learning rates for individual parameters, offering better convergence performance in a variety of scenarios.
• Efficiency and Robustness: Adam combines the benefits of momentum and adaptive learning rates, making it robust, efficient, and well-suited for a wide range of problems and neural network architectures.
• Bias Correction: Incorporating bias correction in the calculation of EWMA estimates helps in the initial time steps when the estimates are biased towards zero.

Up Next

[embed]Deep Learning Series 10:- Weight Initialization Techniques In deep learning, weight initialization plays a crucial role in training neural networks effectively. Poor…medium.com