Regularization: Taming Variance for Better Generalization

In the previous post, we discovered the bias–variance tradeoff. We saw how:

• Simple models (like a linear fit) suffer from high bias — they miss the true structure.
• Complex models (like a 10th-degree polynomial) suffer from high variance — they fit the noise as much as the signal.

This tension gave us the iconic U-shaped curve of error, showing that neither extreme is ideal. Generalization lives in the balance.

But here’s the challenge: as we increase model complexity, variance often explodes much faster than bias shrinks. A model that looks powerful on training data collapses when tested on new inputs.

👉 How can we stop variance from exploding without forcing ourselves back into overly simple, biased models?

This question is what gave birth to regularization.

Why Not Just Use a Simpler Model?

It might seem tempting to say: “If variance is the problem, let’s just keep models simple.” But this approach fails in practice:

• Real-world data often requires complex models to capture nonlinear relationships.
• Simplicity reduces variance but explodes bias — and underfitting is just as bad as overfitting.

Regularization offers a smarter solution: allow complexity, but constrain it. Instead of choosing between a simple underfitting model or a complex overfitting one, regularization lets us use powerful models responsibly.

The Core Idea of Regularization

At its heart, regularization is simple:

• Ordinary training tries to minimize prediction error on the training data.
• Regularization says: “Don’t just minimize error — also keep your model weights under control.”

Mathematically:

• Loss(Data): how well the model fits the training data (e.g., squared error).
• Penalty(weights)\text{Penalty(weights)}Penalty(weights): a function that grows when weights become large or complex.
• λ: the tradeoff knob — it controls how much we care about simplicity vs. fitting the data.

A Simple Analogy: Spending Money

Imagine you’re shopping with a limited budget.

• Without constraints, you might buy everything that looks useful, even if some items don’t add much value.
• With a budget limit, you must prioritize — you can’t let one expensive item dominate, so you spend more wisely.

Regularization is the budget constraint for your model’s weights.

• A small λ = generous budget → model is freer, risk of overfitting.
• A large λ = tight budget → model is heavily constrained, risk of underfitting.
• The right λ = enough flexibility to fit the data, but not enough to go wild on noise.

Ridge Regression (L2 Regularization)

Let’s start simple: linear regression with one feature and no intercept.

Ordinary Linear Regression

*We want to fit: yᵢ​ ≈ wxᵢ​ — where w is the parameter we want to learn.*

❓ Question: Why is this unstable?

If ∑xᵢ²​ is very small (e.g., features are correlated, or data is noisy), the denominator approaches zero. That makes www blow up, leading to high variance and poor generalization.

Ridge Regression: Adding the Penalty

To prevent weights from blowing up, we impose a constraint:

This says: “Yes, fit the data — but don’t let weights become too large.”

Instead of solving this constrained problem directly, we use the Lagrangian method, which converts it into a penalized optimization problem:

Here, λ is the Lagrange multiplier, controlling how strictly we enforce the “keep weights small” constraint.

• If λ = 0 → we’re back to OLS (no penalty).
• If λ → ∞ → weights shrink toward zero (underfitting).
• For moderate λ → balance between data fit and simplicity.

Ridge Solution

❓ Question: Why does λ appear in the denominator?

Because it stabilizes the solution. Even if ∑xᵢ²​ ​is tiny, the denominator can never vanish — λ guarantees stability. Variance is reduced dramatically.

❓ Question: Why does Ridge shrink weights but never make them zero?

Geometric Intuition

In higher dimensions:

• The Ridge penalty λ∥w∥₂²​ defines a circular constraint in weight space.
• The solution is where the error contours touch this circle.
• Because the constraint is circular, weights are shrunk proportionally but rarely set to zero.

Analogy: Tightened Guitar Strings

Without regularization, some weights vibrate wildly in response to noise. Ridge is like tightening all guitar strings — they still vibrate, but within stable limits. The harmony (generalization) improves.

👉 Ridge reduces variance at the cost of a little bias. The result is a more stable model that generalizes better.

But sometimes, we don’t just want stability — we want sparsity, where irrelevant features are completely ignored. That’s where Lasso Regression (L1) comes in.

Lasso Regression (L1 Regularization)

Like Ridge, Lasso modifies the loss function to constrain weights. But instead of penalizing the squared magnitude of weights, it penalizes the absolute value of weights.

In the previous section, we derived the solution for ordinary least squares (OLS). The key problem was instability: weights could blow up if the denominator was small. Ridge fixed this by penalizing the squared weights (∥w∥₂²​), which shrinks them smoothly but never eliminates them.

But what if we also want sparsity — the ability to automatically set some weights to zero and ignore irrelevant features? That’s where Lasso (L1 Regularization) comes in.

Lasso: Adding the L1 Penalty

Instead of penalizing squared weights, Lasso adds the absolute value of the weights:

❓ Question: Why absolute values instead of squares?

Because the geometry changes:

• L2 → circular constraint → smooth shrinkage.
• L1 → diamond-shaped constraint → sharp corners.

Those corners tend to land exactly on the axes, forcing some weights to zero.

Solving Lasso in 1D (Soft-Thresholding)

The derivative of the absolute value isn’t defined at 0, so we use subgradients:

❓ Question: Why does Lasso set weights exactly to zero?

Because if the correlation ∑xᵢyᵢ​ is too small relative to λ, the solution gets thresholded to zero. 👉 Small, noisy, or irrelevant features are automatically eliminated.

• Ridge → shrinkage only.
• Lasso → shrinkage and sparsity.

❓ Question: Why is sparsity useful?

1. Feature Selection: Irrelevant features are dropped automatically.
2. Interpretability: Only the important predictors remain.
3. Stability: Eliminating noisy features reduces variance.

Geometric Intuition

• Ridge’s circle shrinks weights but never hits zero.
• Lasso’s diamond has sharp corners that align with axes, so the optimal solution often lies exactly on an axis → some coefficients = 0.

Analogy: Budget Cuts

• Ridge = every department gets a smaller budget (everyone contributes).
• Lasso = some departments are shut down completely, leaving only the essentials.

👉 Lasso doesn’t just stabilize models like Ridge. It also makes them simpler and more interpretable by eliminating irrelevant features.

But what if we want the best of both worlds: Ridge’s stability + Lasso’s sparsity? That leads us to Elastic Net.

Elastic Net (Hybrid Regularization)

So far, we’ve seen two different approaches:

• Ridge (L2): Shrinks weights smoothly, stabilizes solutions, but never sets them to zero.
• Lasso (L1): Shrinks and can eliminate weights entirely, producing sparse models, but can be unstable when features are highly correlated.

👉 Elastic Net combines the best of both worlds.

❓ Question: Why do we need both?

• Lasso works well when there are many irrelevant features. But when features are highly correlated, it tends to pick one and ignore the rest (unstable).
• Ridge handles correlation better but keeps all features, even unimportant ones.

Elastic Net = Ridge + Lasso → stable and sparse.

Geometric Intuition

• Ridge constraint = circle.
• Lasso constraint = diamond.
• Elastic Net constraint = a rounded diamond — sharp enough to encourage zeros, but curved enough to share weights across correlated features.

So:

• Like Lasso, it can set coefficients to zero.
• Like Ridge, it can keep groups of correlated features instead of arbitrarily choosing one.

Derivation in Simple Terms

❓ Question: When would Elastic Net be better than Ridge or Lasso?

• If features are uncorrelated and sparse → Lasso usually works best.
• If features are highly correlated → Ridge usually works best.
• If the dataset is large, noisy, and correlated → Elastic Net is often the most reliable choice.

That’s why in practice, many machine learning libraries (like scikit-learn’s ElasticNet) default to using it.

Analogy: Balanced Diet

• Ridge = you eat a little of everything (stability, no zeros).
• Lasso = you cut entire food groups (sparsity).
• Elastic Net = you eat balanced: cut down junk entirely, but also keep variety across healthy options.

👉 Elastic Net is the middle path: stable like Ridge, sparse like Lasso. It often gives the best generalization when data is messy and features are correlated.

Conclusion: Why Regularization Matters

The bias–variance tradeoff taught us that generalization lives in the balance. Too simple, and bias dominates; too complex, and variance explodes. Regularization is the tool that keeps this balance under control.

• Ridge (L2): Shrinks weights smoothly, stabilizes the solution, but never eliminates features.
• Lasso (L1): Shrinks and can set weights exactly to zero, automatically performing feature selection.
• Elastic Net (L1 + L2): Combines stability and sparsity, making it the most versatile choice when features are correlated or data is noisy.

At its core, regularization works by saying:

Don’t just minimize training error — also control model complexity.

The parameter λ is the dial that lets us decide how much we care about simplicity versus fitting power.

👉 Regularization introduces a small amount of bias, but in exchange, it dramatically reduces variance — leading to models that are more stable, interpretable, and better at generalizing.