AXIS: From Error to Improvement: How Neural Networks Actually Learn — Gradient Descent, Intuitively

Back-propagation tells us what went wrong. Gradient descent tells us how to fix it.

Gradient Descent in Real Life

This is part of the AXIS series (AI eXplained through Intuitive Systems), where we break AI systems into intuitive, visual steps to make it super easy for anyone to learn the magic of AI in the simplest way possible.

In this point we will continue from the back propagation to the gradient descent.

1. Quick Recap (From Back-propagation Post)

In that post, we broke down back-propagation:

• We made a prediction
• Measured how wrong it was (loss)
• Computed how each parameter (w, b) contributed to the error (gradients)

That answered: What caused the mistake?

Now what?

2. Learning Analogy and Why It Isn’t Magic

Imagine trying to hit a tennis serve into the box. (Don’t worry if you haven’t played tennis ever, you can imagine any other sports or target you are trying to hit)

You miss. Slightly long. What do you do next?

• Swing randomly and hope it lands in?
• Or adjust just a little — maybe less power, slightly different angle?

While swinging randomly may give you a few success over numerous trials, adjusting a little, trying again and then adjusting a little again has higher probability of success.

That tiny correction is nothing but learning.

This is exactly how machines learn. Through small, informed corrections one step at a time.

3. The Real Goal

Let’s tie things back to a neural network domain. Our goal is to reduce the loss or error. Why?

Because if the neural network can predict with less error, it is learning how to predict with less error, i.e. how to predict with higher accuracy.

4. How does the loss function look visually?

So, we have three things that are in the interplay here — parameter w (weights), parameter b (biases) and loss L.

Now, let’s visualize the loss function. Below is a graph showing the loss surface (L) plotted along with parameters (w and b).

This is called loss landscape. As you can see it takes the shape of a surface.

Height = Loss = How wrong the model is or how big the loss is

Position = What’s specific values of w and b are for a particular height (or loss)

Now imagine this:

You’re dropped somewhere on this surface… blindfolded.

How do you reach the bottom?

5. The Key Insight

You don’t need to see the whole landscape.

This is very important, as neural networks do not know everything when they start. Remember that they start with randomly assigned weights and biases.

You just need to know:

Which direction goes downhill?

That is standing on the surface of the loss which direction will take you to low error area.

That’s what the gradient (aka slope) gives you.

• Gradient = direction of steepest increase
• So the opposite direction = steepest decrease

6. Worked example of the key insight

Below is a worked example of the direction of gradient on a simple curve

y = x²

If we are at point x = 2, should we increase x or decrease x to reduce the loss? From the graph it’s quite evident that we need to decrease x to lower the loss, but is there a consistent way of formalizing it so the math works across all the points on the graph?

The answer is resounding yes. At any given point we will move in the “opposite” direction of the gradient. This is because gradient gives us the direction where the curve has the steepest increase and if we go in the opposite direction we will go down the curve and up the curve.

*x_new = x_old — learningRate gradient ← UpdateEquation**

Learning rate is the rate at which we update the parameter values and it’s a fraction. For our example we will use 0.001 as the learning rate.

So, assuming the learning rate to be 0.001,

6.1 Update of x at 2

At x = 2,

y = Loss = (2)² = 4

gradient = 4 (from the worked example in the image above)

Using UpdateEquation from above

x_new= 2 — (0.001)*4 = 1.996 ← Using UpdateEquation a

And loss at the new value = (1.996)² = 3.984,

So by doing this adjustment of x=2 to x=1.996 we are moving in the direction of the smaller loss or smaller error. This is aligned with our goal of reducing error.

6.2 Similartly Update of x at -2

At x = -2,

loss = (-2)² = 4

gradient = -4 (Again, from the worked example in the image above)

Using UpdateEquation from above

x_new = (-2) — 0.001 * (-4) = -1.996

Loss at the new value = (-1.996)² = 3.984

Again the loss is smaller after the adjust of x at x = -2.

7. Gradient Descent in Plain English

Here’s the entire algorithm in simple steps:

1. Start with random parameters (random values to w, b)
2. Make a prediction (get y_pred from the forward pass on the network desribed in the backpropagation post)
3. Measure error (aka loss)
4. Use backpropagation to compute gradients
5. Adjust the parameters of the network to move slightly in the opposite direction
6. Repeat

That’s it. No magic. Just:

Measure → Blame -> Adjust → Repeat

8. The One Knob That Matters: Learning Rate

As described in th UpdateEquation above, the learning rate controls how big each step can be.

So, how big should each step be?

Visually you can see the impact of the steps below:

Learning rate’s impact on steps

1. Too Small

• Tiny steps
• Takes forever

Safe to arrive at the lowest point on the loss surface, but painfully slow

2. Too Large

• You overshoot the valley
• Bounce around

Aggressive and steps are bigger, but it leads to unstable run as you may just keep bouncing around the curve.

3. Just Right

• Smooth descent
• Converges efficiently

Small corrections, compounded over time ends up in a good run towards the lowest point on the surface.

9. Takeaway Mental Model

If you remember one thing, remember this:

Gradient Descent = Error → Direction → Small Correction → Repeat

Or even simpler: Adjust slightly in the right direction and keep doing it until you reach your goal.

10. One Image to Sum It All Up

Now you are ready to take a look at the picture below and make sense of it. Hope it helps you understand visually what’s happening under the hood.

Gradient Descent Summary in One Image