Optimizers in Deep Learning: From Gradient Descent to Adam

A practical, math-grounded tour of the algorithms that actually teach neural networks how to learn.

FIG.1

Why Do We Even Need Optimizers?

Training a neural network is, at its core, a search problem. The network has millions of parameters (weights and biases), and we want to find the combination of values that minimizes a loss function — a number that tells us how wrong the network’s predictions are.

The loss surface created by these millions of parameters is high-dimensional, bumpy, and full of valleys, plateaus, and saddle points. We can’t solve for the minimum analytically. Instead, we need an iterative strategy that nudges the parameters, step by step, toward lower loss.

That strategy is the optimizer. An optimizer decides:

• Which direction to move the parameters (using gradients of the loss).
• How big a step to take (the learning rate).
• How to use past information to make smarter, faster, more stable steps.

A bad optimizer can make training painfully slow, get stuck in poor regions, or diverge entirely. A good optimizer can mean the difference between a model that converges in hours versus days.

Gradient Descent is an iterative optimization algorithm used to minimize a cost function by adjusting model parameters in the direction of the steepest descent of the function’s gradient. In simple terms, it finds the optimal values of weights and biases by gradually reducing the error between predicted and actual outputs.

FIG.2

Suppose you’re at the top of a hill and your goal is to find the lowest point in the valley. You can’t see the entire valley from the top, but you can feel the slope under your feet.

1. Start at the Top: You begin at the top of the hill (this is like starting with random guesses for the model’s parameters).
2. Feel the Slope: You look around to find out which direction the ground is sloping down. This is like calculating the gradient, which tells you the steepest way downhill.
3. Take a Step Down: Move in the direction where the slope is steepest (this is adjusting the model’s parameters). The bigger the slope, the bigger the step you take.
4. Repeat: You keep repeating the process feeling the slope and moving downhill until you reach the bottom of the valley (this is when the model has learned and minimized the error).

The key idea is that, just like walking down a hill, Gradient Descent moves towards the “bottom” or minimum of the loss function, which represents the error in predictions.

What is Learning Rate?

Learning rate is a important hyperparameter in gradient descent that controls how big or small the steps should be when going downwards in gradient for updating models parameters. It is essential to determines how quickly or slowly the algorithm converges toward minimum of cost function.

1. If Learning rate is too small: The algorithm will take tiny steps during iteration and converge very slowly. This can significantly increases training time and computational cost especially for large datasets.

FIG.3

2. If Learning rate is too big: The algorithm may take huge steps leading overshooting the minimum of cost function without settling. It fail to converge causing the algorithm to oscillate. This process is termed as exploding gradient problem.

FIG.4

In image we can see point got oscillated from right to left with converging to minimum gradient value.

To address these problems we have some technique that can be used:

• Weights Regularzations: The initialization of weights can be adjusted to ensure that they are in an appropriate range. Using a different activation function such as the Rectified Linear Unit (ReLU) can help us to mitigate the vanishing gradient problem.
• Gradient clipping: Restrict the gradients to a predefined range to prevent them from becoming excessively large or small.
• Batch normalization: It can also help to address these problems by normalizing the input of each layer to prevent activation function from saturating and hence reducing vanishing and exploding gradient problems.

Choosing right learning rate can leads to fast and stable convergence improving the efficiency of the training process but sometimes vanishing and exploding gradient problem is unavoidable and to address these we have some techniques that we will discuss further in the article.

1. Gradient Descent (GD)

FIG.5

Gradient Descent is the foundation of almost every optimizer used today. The idea is simple: compute the gradient (slope) of the loss function with respect to every parameter, and move in the opposite direction of that gradient — because the gradient points toward the steepest increase, and we want to decrease the loss.

Fig.6

Update rule:

θ = θ - η · ∇L(θ)

Where:

• θ = model parameters
• η = learning rate
• ∇L(θ) = gradient of the loss with respect to θ, computed using the entire training dataset

How it works:

1. Pass the full dataset through the network.
2. Compute the loss and its gradient with respect to every weight.
3. Update all weights once.
4. Repeat for many epochs.

Advantages

• Gradient estimate is accurate and stable since it uses the full dataset.
• Convergence is smooth — no noisy zig-zagging.

Limitations

• Extremely slow for large datasets — one full pass over millions of samples is needed just for a single update.
• Memory-intensive: the entire dataset must be loaded to compute one gradient.
• Can get stuck in sharp local minima or saddle points because there’s no noise to help escape them.

2.Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent (SGD) is a variation of G/radient Descent that updates the model parameters using one training example at a time instead of the entire dataset. Rather than waiting to process all samples before updating the weights, SGD performs an update after each individual training example.

This makes learning much faster and allows the model to start improving immediately, especially when dealing with large datasets.

FIG.7

Update Rule:

θ=θ−η⋅∇Li​(θ)

Where:

• θ= Model parameters (weights and biases)
• η = Learning rate
• ∇Li​(θ) = Gradient of the loss computed using a single training example iii

How it Works

1. Select one training example from the dataset.
2. Perform a forward pass and compute the loss.
3. Calculate the gradient of the loss with respect to the model parameters.
4. Update the weights immediately.
5. Move to the next training example and repeat.
6. Continue until all samples are processed (one epoch).
7. Repeat for multiple epochs until convergence.

Example

Suppose a dataset contains 10,000 samples.

• Gradient Descent updates weights once per epoch.
• Stochastic Gradient Descent updates weights 10,000 times per epoch.

This frequent updating often leads to much faster learning.

Advantages

Faster Training

Since updates are performed after every training example, SGD can begin learning immediately without waiting for the entire dataset.

Memory Efficient

Only one sample needs to be processed at a time, making SGD suitable for very large datasets.

Can Escape Local Minima

The noisy updates introduce randomness that can help the optimizer escape local minima and saddle points.

Suitable for Online Learning

SGD can continuously learn from new incoming data without retraining on the entire dataset.

Limitations

Noisy Updates

Because gradients are computed from a single sample, updates fluctuate significantly and may not always move directly toward the optimum.

Unstable Convergence

The loss function often oscillates around the minimum instead of smoothly converging.

Sensitive to Learning Rate

A learning rate that is too high can cause divergence, while a very small learning rate can make training extremely slow.

May Require More Epochs

Although each update is fast, SGD may require more training iterations to reach the optimal solution.

When to Use SGD?

SGD is commonly used when:

• The dataset is very large.
• Memory resources are limited.
• Online or streaming learning is required.
• Fast initial learning is more important than perfectly stable updates.

Because of its noisy nature, SGD is often enhanced with advanced optimizers such as Momentum, RMSProp, and Adam, which improve convergence speed and stability.

3. Mini-Batch Gradient Descent

Instead of using the whole dataset (too slow) or a single example (too noisy — this extreme case is called Stochastic Gradient Descent, SGD), Mini-Batch GD strikes a balance: it computes the gradient over small batches of data (typically 32, 64, 128, or 256 samples).

fig.8

Update rule:

θ = θ - η · ∇L_batch(θ)

Here ∇L_batch(θ) is the gradient computed from a small randomly sampled batch instead of the full dataset.

How it works:

1. Shuffle the dataset and split it into mini-batches.
2. For each batch, compute the gradient and update the weights.
3. After going through all batches once, that’s one epoch.

Advantages

• Much faster than full-batch GD since updates happen frequently.
• Introduces helpful noise, which can help escape shallow local minima/saddle points.
• Efficiently parallelizable on GPUs.

Limitations

• Noisier convergence path — loss can fluctuate instead of decreasing smoothly.
• Choosing the right batch size and learning rate becomes an extra hyperparameter tuning task.
• Still uses a single fixed global learning rate for all parameters.

4. Momentum-Based Gradient Descent

FIG.9

Plain Mini-Batch GD can zig-zag badly, especially across narrow valleys in the loss surface (imagine a steep ravine — gradient descent oscillates from wall to wall instead of moving straight down the ravine). Momentum fixes this by giving the optimizer “memory” of past gradients, similar to a ball rolling downhill and building up speed.

FIG.10

Update rule:

v_t = β · v_(t-1) + (1 - β) · ∇L(θ)
θ   = θ - η · v_t

Where:

• v_t = velocity (an exponentially weighted moving average of past gradients)
• β = momentum coefficient (commonly 0.9)

Intuition: gradients that consistently point in the same direction accumulate and accelerate the update, while oscillating components (that flip sign frequently) cancel out.

Advantages

• Speeds up convergence, especially in ravine-like loss surfaces.
• Dampens oscillations, leading to a smoother path.
• Helps escape shallow local minima due to accumulated velocity.

Limitations

• Can overshoot the minimum due to built-up momentum, causing oscillation around the optimum.
• Adds another hyperparameter (β) to tune.

A popular variant, Nesterov Accelerated Gradient (NAG), looks ahead by computing the gradient at the projected future position rather than the current one, giving even faster and more accurate corrections.

5. AdaGrad (Adaptive Gradient Descent)

So far, every parameter shares the same learning rate. But some parameters (like rarely-activated features in sparse data) need bigger updates, while others need smaller, more cautious updates. AdaGrad adapts the learning rate per parameter, based on how large its historical gradients have been.

FIG.11

Update rule:

G_t = G_(t-1) + (∇L(θ))²
θ   = θ - (η / √(G_t + ε)) · ∇L(θ)

Where:

• G_t = sum of squares of all past gradients for that parameter
• ε = small constant to avoid division by zero (e.g., 1e-8)

Intuition: parameters with consistently large gradients accumulate a large G_t, which shrinks their effective learning rate. Parameters with small/infrequent gradients keep a relatively larger effective learning rate.

FIG.12

Advantages

• Excellent for sparse data (e.g., NLP, recommendation systems) where some features appear rarely.
• No manual learning-rate decay schedule needed — it adapts automatically.

Limitations

• G_t only grows — it never shrinks. Over time, the effective learning rate keeps shrinking and can become so small that learning effectively stops, even before reaching a good minimum.

6. RMSProp (Root Mean Square Propagation)

FIG.13

RMSProp was designed specifically to fix AdaGrad’s “learning rate vanishes too fast” problem. Instead of accumulating all past squared gradients forever, it keeps an exponentially decaying average of squared gradients — so old gradients gradually “forgotten” rather than permanently inflating the denominator.

FIG.14

Update rule:

E[g²]_t = β · E[g²]_(t-1) + (1 - β) · (∇L(θ))²
θ       = θ - (η / √(E[g²]_t + ε)) · ∇L(θ)

Where β is typically 0.9 (decay rate).

Advantages

• Solves AdaGrad’s vanishing learning rate problem.
• Works well for non-stationary objectives (e.g., RNNs, online learning) since it adapts to recent gradient behavior.
• Widely used and reliable in practice.

Limitations

• Still requires manual tuning of the base learning rate η.
• Doesn’t incorporate momentum on its own — pure RMSProp can still be slow to build directional speed.

7. Adam Optimizer (Adaptive Moment Estimation)

Adam is essentially Momentum + RMSProp combined. It keeps track of both:

1. An exponentially decaying average of past gradients (like Momentum) — the first moment.
2. An exponentially decaying average of past squared gradients (like RMSProp) — the second moment.

FIG.15

Update rule:

m_t = β1 · m_(t-1) + (1 - β1) · ∇L(θ)          (1st moment - mean)
v_t = β2 · v_(t-1) + (1 - β2) · (∇L(θ))²         (2nd moment - variance)

m̂_t = m_t / (1 - β1^t)     (bias correction)
v̂_t = v_t / (1 - β2^t)     (bias correction)

θ   = θ - η · m̂_t / (√v̂_t + ε)

Defaults: β1 = 0.9, β2 = 0.999, ε = 1e-8.

The bias correction step matters: since m_t and v_t are initialized at zero, early estimates are biased toward zero. Dividing by (1 - β^t) corrects this, especially important in the first few steps of training.

Advantages

• Combines the speed of Momentum with the per-parameter adaptivity of RMSProp.
• Converges fast and reliably across a very wide range of problems.
• Relatively insensitive to hyperparameter choices — works well “out of the box.”
• The de-facto default optimizer for most deep learning tasks today (CNNs, Transformers, RNNs).

FIG.16

Limitations

• Can sometimes generalize slightly worse than well-tuned SGD with momentum on certain tasks (notably some computer vision benchmarks).
• More memory required (it must store two moving averages per parameter, instead of one).
• Variants like AdamW (which decouples weight decay from the adaptive update) are now often preferred for training large models such as Transformers.

Putting It All Together: Visual Intuition

Picture the loss surface as a valley:

FIG.17

• Gradient Descent walks straight down using the full map, but takes forever per step.
• Mini-Batch GD takes quick, noisy steps based on small samples of the map.
• Momentum behaves like a ball rolling downhill, building speed along consistent directions and smoothing out zig-zags.
• AdaGrad slows down progressively per-direction the more it has already moved that way — great for sparse terrain, risky for long journeys.
• RMSProp does the same per-direction adaptation but with a short memory, so it doesn’t grind to a halt.
• Adam combines a “rolling ball” (momentum) with “smart per-direction step sizing” (RMSProp), making it fast and adaptive.

FIG.18

Comparison Table

FIG.19

Batch-Based Optimizers vs Advanced Optimizers

Throughout this article, we explored several optimization algorithms, but it’s important to understand that they belong to two different categories.

Batch-Based Optimizers focus on how gradients are computed, whereas Advanced Optimizers focus on how those gradients are used to update the model parameters.

Batch-Based Optimizers

The first family of optimizers determines how much training data is used to calculate the gradient.

• Gradient Descent (GD): Uses the entire dataset to compute a single gradient update.
• Stochastic Gradient Descent (SGD): Uses one training example at a time.
• Mini-Batch Gradient Descent: Uses a small subset of data (batch) for each update.

These methods primarily differ in the trade-off between computational efficiency and gradient accuracy.

Advanced Optimizers

Advanced optimizers build on top of gradient descent and introduce additional mechanisms to improve training.

• Momentum: Uses previous gradients to accelerate learning and reduce oscillations.
• RMSProp: Adapts the learning rate for each parameter individually.
• Adam: Combines the strengths of Momentum and RMSProp to achieve fast and stable convergence.

These optimizers focus on improving the parameter update process rather than changing how gradients are calculated.

FIG.20

Final Takeaway

There’s no single “best” optimizer for every situation, but a practical rule of thumb:

• Start with Adam (or AdamW) as your default — it works well across most architectures with minimal tuning.
• If you’re chasing the absolute best generalization on a well-understood problem (like image classification with CNNs) and have time to tune hyperparameters, SGD with Momentum can sometimes outperform Adam.
• If your data is sparse (text, recommendation systems), AdaGrad or RMSProp are worth experimenting with.
• Always pair your optimizer choice with a sensible learning rate schedule — even the best optimizer can fail with a poorly chosen learning rate.

Understanding these optimizers isn’t just academic — it directly shapes how fast your models train, how stable that training is, and ultimately, how good your final model performs.

If you found this useful, give it a clap 👏 and follow for more deep learning breakdowns.