Tangent Line Working in Gradient Descent

As we all know, convergence algorithms rely on tangent lines to move in the direction of the steepest descent, ultimately aiming to find the optimal minima. In this blog, I’ll walk you through how tangent lines are formed and how they function in the context of gradient descent.

P.S.: Don’t confuse the slope of the best-fit line with the slope of a tangent line .

Let’s begin by understanding what a tangent line is.

The definition of a tangent line has evolved over time.

Initially, it was defined as a line that touches a circle at exactly one point.

Later, it was generalized as a line that touches any curve at a single point without crossing it.

Eventually, the definition became more mathematical: a tangent line is one that, when slightly altered, becomes a secant line (a line that intersects the curve at two or more points). In calculus terms, the derivative at a point exists if the left-hand limit is equal to the right-hand limit.

To draw a tangent line, the equation is simple:

where m is the slope of the tangent at point (x1,y1).

Now, let’s understand the purpose of a tangent line. A tangent line gives the rate of change of a curve at a specific point. In other words, it tells us how steep the curve is at that point.

Let’s take a real-world example. Consider a graph that shows the relationship between year and world population. If we want to find the rate of change of the world population at a particular year, we draw a tangent line at that point on the curve. The slope of that tangent line will represent the instantaneous rate of population change in that specific year.

Tangent Line in Gradient Descent

Now, let’s say we have a best-fit line with a slope θ1=0.2. For simplicity, let’s assume θ0=0 (i.e., the intercept is zero). So the equation of our line becomes:

Next, we need to compute the cost function, which in our case is the Mean Squared Error (MSE). The cost function is given by:

Usually, when we visualize gradient descent, we plot the cost function J(θ1) against θ1​. This curve represents how the cost changes for different values of θ1​.

Lets take a single training point and update the equation,

Lets perform derivation

If we substitute the respective θ1​ value, we will get the slope of the tangent line. Now we have the slope of the tangent line, and we also have the point (θ1,J(θ1)),

where ,

θ1 is the slope of the best-fit line and

J(θ1) is the corresponding cost.

Now we can compute the equation of the tangent line at that point on the gradient descent curve.

This forms the basis of the convergence algorithm.