A High-Level Overview of Deep Learning Optimizers

What are Optimizers?

Optimizers are algorithms that update the weights and biases of a neural network during training to minimize the loss.

First, data passes through the network (forward pass), and the loss is computed. Then, during backpropagation, gradients (slopes) are calculated. These gradients tell us how much each weight is affecting the loss.

During backpropagation, we are basically checking how each variable influences the loss. Sometimes, the model may reach a point where the loss is low in a small region — this is called a local minimum.

A local minimum is a good solution, but it is not the best one. There may exist another point where the overall loss for the entire data is even lower — this is called the global minimum.

Because of this, we use different optimizers that help the model move better and try to reach the global minimum, where the loss is lowest for the whole dataset.

Local Minima vs Global Minima:

Local Minima: A local minimum is a point where the loss is lower than nearby points, but not the lowest overall.

Global Minima: A global minimum is the lowest point of loss in the entire graph.

1. Gradient Descent: In Gradient Descent we uses the entire dataset to train the model. It computes the gradients using all the data and updates the weights step by step to reduce the loss. Since it follows a smooth and fixed path, it may reach a minimum. However, it can get stuck in a local minimum and cannot easily escape because there is no randomness in its updates.

1. Convex Problem (one minima in fig1): Gradient Descent reaches the global minimum because there is only one minimum
2. Non-convex Problem (many minima in fig2): Gradient Descent may get stuck in a local minimum instead of reaching the global minimum. fig2

👉Major Problem: It might got stuck in local minima

2. Stochastic Gradient Descent: Since normal Gradient Descent has no randomness, we use Stochastic Gradient Descent. In this method, instead of using the entire dataset, we pass one random data point at a time. Because of this randomness, the model moves in different directions as it sees different slopes for different points. This makes the path noisy, which helps it avoid getting stuck in local minima, although it may take a less smooth path toward the global minimum.

👉Major Problem: It moves in different directions since we train on random points, so the convergence is not smooth and is more noisy.

3. Mini-Batch SGD: Here, instead of passing a single data point, we pass a small batch (subset) of data at a time. This reduces the noise compared to SGD and makes the updates more stable.

👉 Major Problem: The convergence is smoother than SGD, but still not completely smooth, and it can still oscillate.

4. SGD With Moment: In SGD with Momentum, we use past gradients along with the current gradient gt​. The value vt​ stores this past information and helps reduce oscillations and speed up learning

👉Major Problem: Because of momentum, the model may move too fast and miss the exact minimum.

5. AdaGrad: AdaGrad is an optimization method that adapts the learning rate for each parameter based on past gradients. As training continues, the learning rate keeps decreasing because gradients accumulate over time. This helps in handling sparse data, but one problem is that the learning rate can become too small, which may slow down or stop learning.

👉Major Problem: Because the learning rate keeps shrinking, the model takes very tiny steps and may stop learning

6. RMS Props: Root Mean Square Props

In RMSProp, instead of storing all past gradients like AdaGrad, we keep track of recent squared gradients. This helps control the learning rate so that it does not keep decreasing too much. Unlike SGD with momentum, which uses past gradients to smooth the direction of movement, RMSProp focuses on adjusting the step size (learning rate) based on recent gradients. This allows the model to take balanced steps and continue learning without slowing down too much.

👉Major Problem: It does not consider momentum (direction properly), so updates can still be unstable or inconsistent.

7. Adam (Adaptive Moment Estimation) combines the ideas of Momentum and RMSProp. It keeps track of past gradients (like momentum) to smooth the direction and also uses squared gradients (like RMSProp) to adjust the learning rate. Because of this, Adam updates weights in a more stable and efficient way, helping the model converge faster and work well in most cases.

👉Major Problem: Adam can sometimes converge to a suboptimal solution and may not generalize as well as simpler methods like SGD in some cases.

All:

Thanks for reading!!!

Note: These are my personal learning notes, written in a simple way to understand the concepts. If you spot any mistakes or improvements, feel free to share!