What is Backpropagation?

When newbies just started learning Neural Networks, they often hear “Back Propagation” but have no idea what it is. Believe me. I was once in these shoes. I was even asked in one of my data Science Intern interviews about what backpropagation is and, yes, I failed miserably. After watching the free deep learning course 6S191 series offered by MIT, which I highly recommend you to watch, I came to the conclusion that Back Propagation is no more than

the process of computing the gradient line and obstaining better weights to adjust the model.

backpropagation is the green arrow

A NN with no backpropagation is like a car without an engine — it lacks the essential mechanism required for learning and improving its performance.

But to understand why we need backpropagation, we first need to understand gradient descent. Gradient Descent is an algorithm used to minimize the loss function of a model. It works by iteratively adjusting the parameters of the model in the direction of the steepest descent of the loss function.

Think of the loss function as a metric for how good or bad your model is. The bigger the loss the worse our model is because our predictions deviate from the true values. In the graph above, J(w0,w1) represents the loss of the model with weights w0, w1. For every set of weights, loss can be calculated. How to go from top of the loss hill to the bottom? Use gradient descent.

In every step of gradient descent, we use some derivative and chain rule magic to compute the gradient: a direction to the steepest uphill direction. So we move weights in opposite directions, and voila, simple as that.

In summary, backpropagation is the process of applying a gradient descent algorithm. It finds better weights and gives an awesome model by computing the direction it should go in the loss mountains iteratively. Without backpropagation, Neural Networks would never evolve, and predictions would be inaccurate.