How Squared Gradients Amplify Overfitting — And Why Absolute Gradients Can Help Feature Selection

In this blog, let’s walk through why we say Ridge Regression helps in avoiding overfitting, while LASSO Regression helps in eliminating independent features that are weakly correlated with the output. At first glance, the math seems quite similar — in Ridge, we add the sum of squared weights as a regularization term, whereas in LASSO, we add the sum of the absolute values of the weights. So how does this one change — from squares to absolutes — drastically alter the behavior of the algorithm?

Let’s explore that by diving into the derivation.

Ridge regression cost function is

Now, when we take the derivative of this cost with respect to θ, we get:

Now, in gradient descent:

Let’s rewrite this:

In Ridge Regression, the regularization term is proportional to the square of the slope (i.e., λθ²). When we compute the derivative of this term with respect to the parameter θ, we get 2λ. This derivative depends on the current value of θ, meaning:

• As θ becomes smaller, the regularization term itself becomes smaller.
• As a result, the gradient update becomes smaller too — the “pull” toward zero gets weaker the closer you get.
• This means Ridge regression shrinks parameters but rarely drives them exactly to zero.
• It reduces the magnitude of the weights, helping to prevent overfitting, but usually retains all features.

In LASSO Regression, the regularization term is based on the absolute value of the slope (i.e., λ∣θ∣). The derivative of this term is:

So the update becomes:

This derivative is constant, the regularization term still applies a constant force to reduce it. Eventually, if the force from the data’s gradient is weaker than λ, the sign of the total gradient flips. When that happens, the optimizer drives θ exactly to 0, and stops updating it.

Note: λ acts like a scaling factor — it determines what proportion of the slope to subtract. But the actual update to θ at each step depends on the current value of θ itself, not just λ. In most gradient graphs, λ is shown as the main driver, which can be misleading. Yes, increasing λ reduces the slope, but that’s because it increases the proportion of the slope being subtracted. So the slope reduction is determined by both λ and θ — not by λ alone.

This is why LASSO leads to sparse solutions, effectively performing feature selection by eliminating some coefficients.