OpenLearn | Regularization- The Art of Balancing Bias and Variance

“A journey of a thousand miles begins with a single line of code.” — Someone smart in our cohort group chat, probably

Moto of OpenLearn Initiative

👋 Welcome Back, Learner!

Yesterday, you were introduced to the Linear Regression family — simple, elegant, and straight to the point. But let’s be honest… sometimes, our models get a little too cozy with the training data. Yep, we’re talking about Overfitting — that clingy ex who just won’t let go.

But worry not! Today, we’re diving into some machine learning magic spells — Ridge, Lasso, and Elastic Net — the superheroes of regularization. These techniques keep our models humble, focused, and way better at handling real-world data.

Today’s Mission

Today you’ll: 1. Understand what regularization is and why it matters 2. Explore Ridge, Lasso, and Elastic Net regression 3. Learn the math behind each technique 4. Write Python code to implement them 5. Visualize the effect of regularization 6. Know when to use each method

Bonus : Understanding concepts like underfitting, overfitting, Bias-Variance trade-off

Let’s Break It Down

“You don’t really understand something unless you can explain it simply.” — Einstein (or was it your ML mentor?)

Imagine you’re trying to draw a line through a bunch of scattered dots — these dots are your data points. Now, your line has three ways to behave:

• It might try too hard to touch every single point, twisting and turning like it’s doing yoga — this is called overfitting.
• It might be too lazy, barely making an effort to follow the pattern — that’s underfitting.
• Or, it might find a sweet spot, capturing the main trend without obsessing over every dot — that’s generalization.

😵‍💫 But wait… what is this overfitting and underfitting thing anyway?

Visuals for understanding Overfitting and Underfitting

• Overfitting: The model becomes too smart for its own good. It learns the training data perfectly — even the noise and random quirks. While it gets an A+ on the training data, it fails miserably on new data. It’s like a student who memorizes every word of a textbook but freezes during real-life questions.
• Underfitting: The model is too simple to capture the pattern. It doesn’t learn enough, even from the training data. Imagine trying to explain rocket science using crayons — it’s not gonna work.
• Generalization: This is the goal! Your model learns the core patterns and ignores the noise. It performs well on both training and new data. It’s like a wise old owl 🦉 — it knows what matters and what doesn’t.

To help our model aim for that sweet spot, we use regularization — a way to gently say: “Hey buddy, don’t overcomplicate things!”

Regularization works by penalizing complexity, so your model doesn’t get carried away trying to memorize every detail.

What is the Bias-Variance Trade-off?

Imagine you’re learning archery. You shoot arrows at a target, and the goal is to hit the bullseye.

• Bias: How far off your arrows are from the bullseye on average.
• Variance: How spread out your arrows are — are you consistent?

Now apply this to machine learning:

Bias-Variance Trade off

What is Regularization?

Regularization is like a leash for your model — it keeps it from going wild. It stops the model from putting too much importance on any one feature, which helps it focus on what really matters (and prevents the model from learning noise) and perform better on new, unseen data.

💡 Pro Tip:

Regularization ≠ removing features. It’s about controlling how much importance (weight) each feature can carry.

Essential Mathematical Insights

Let’s break down the math progressively. All techniques below build on top of Linear Regression.

Linear Regression Loss

We try to minimize the following cost function:

cost function

Here’s what everything means:

• yᵢ ​: The actual value for the iᵗʰ data point.
• ŷᵢ : The predicted value for the iᵗʰ data point.
• n : Total number of data points
• ∑ : Summation (we’re adding up the squared errors for all data points)

✄ Lasso Regression (L1 Regularization)

The term Lasso stands for:

Least Absolute Shrinkage and Selection Operator (Fancy, huh? But don’t worry, we’ll break it down.)

Lasso helps prevent overfitting by punishing the model if it tries to rely too much on certain features (especially the useless ones).

What happens behind the scenes ?

The lasso model minimizes this :

• Error Term : This is the usual error — how far is the prediction from the actual value.
• Penalty Term : This adds a penalty for large values of the model’s coefficients. (sum of the absolute values of the coefficients.)
• λ is the tuning parameter that controls the strength of the penalty. As λ increases more coefficients are pushed towards zero

As you increase λ:

• The model becomes simpler (less chance of overfitting).
• But if λ is too high, the model might become too simple and start underfitting.

So, there’s a trade-off:

Low λ = overfit risk High λ = underfit risk Just right λ = chef’s kiss for generalization

Intuition Behind the Added Penalty:

The larger the feature’s weight -> larger is it’s penalty (as λ is multiplied by the weight in penalty term BUT Why penalize large coefficients ?

Because large coefficients → the model is relying too much on certain features. This makes the model too sensitive to changes or noise in the input.

Regularization adds a cost for large coefficients. Now, the model has to balance:

• Fitting the data well (low prediction error)
• Keeping coefficients small (low penalty)

So, even if a big weight makes the error smaller, it comes with a cost. The model may choose smaller weights to minimize the total loss.

How is Lasso able to make the coefficients 0 ?

• The absolute value |Wᵢ| is not differentiable at 0.
• This leads to sharp edges (kinks) in the loss function.
• During optimization (e.g., gradient descent or coordinate descent), these sharp corners create a “pull” toward exactly zero Because optimization algos try to minimize the value and if 0 is possible than why not to take it…

When to use Lasso Regression

Lasso Regression is useful in the following situations:

1. Feature Selection: It automatically selects most important features by reducing the coefficients of less significant features to zero.
2. Collinearity: When there is multicollinearity it can help us by reducing the coefficients of correlated variables and selecting only one of them.
3. Regularization: It helps preventing overfitting by penalizing large coefficients which is useful when the number of predictors is large.
4. Interpretability: Compared to traditional linear regression models that have all features lasso regression generates a model with fewer non-zero coefficients making model simpler to understand.

Fun Fact : Lasso is called L1 Regularization because it uses the L1 norm of the coefficients. Curious about what is L1 Norm search and find it 🧑🏻‍💻

But what if we don’t want to drop features? Enter: L2 Regularization (Ridge)

So, you just met L1 regularization (Lasso) — the minimalist friend who loves zeroing out coefficients and trimming the fat. But sometimes, we don’t want to eliminate features. Instead, we just want to gently control them — like keeping them on a diet, not cutting them off entirely.

What does Ridge do?

Ridge (L2) regularization doesn’t try to remove features like Lasso. Instead, it says:

“Okay coefficients, you can stay — but don’t get too big.”

It shrinks all the coefficients towards zero, but never exactly zero.

So if Lasso is the strict friend who says: “You’re either in or you’re out!”

Ridge is the chill one who says: “You can stay, but keep a low profile.”

That’s where Ridge regression steps in!

How does Ridge work? — The only difference is the penalty term used

So instead of using the absolute value like L1 does, L2 uses the square of each coefficient.

But why square it?

Because squaring:

• Grows faster as the coefficient increases.
• Keeps the function smooth and differentiable everywhere (especially at 0). -> bowl like structure
• Doesn’t create a kink — so it gently pulls coefficients closer to zero but never exactly zero.

Key Effect of the Squared Term:

• If a coefficient is large, |Wᵢ|² is much larger than |Wᵢ|, so it gets penalized more heavily.
• If a coefficient is small, the square becomes even smaller, so it’s not harshly punished.
• This leads to gradual shrinkage of all coefficients — like putting all of them on a “diet” — but none are completely removed.

When to Use Ridge Regression

1. Multicollinearity Exists : Handles correlated predictors by distributing weights instead of eliminating features.
2. All Features Might Be Relevant : Retains all features with small but non-zero coefficients.
3. You Want to Avoid Overfitting : Uses L2 penalty to shrink large coefficients and improve generalization.
4. High-Dimensional Data (m > n) : Works well when number of features exceeds number of observations.
5. Smooth Coefficient Shrinkage Preferred : Shrinks coefficients gradually without setting them exactly to zero.
6. You Don’t Need Feature Selection : Unlike Lasso, Ridge does not perform automatic variable elimination.

So… Lasso or Ridge?

Why not both? Meet Elastic Net — the best of both worlds!

Elastic Net Loss Function :

or more neatly -

Here:

• λ : controls how strong the penalty is overall.
• α : controls how much L1 vs. L2 you apply.

What is α (aka L1 Ratio in formula)?

In Elastic Net, l1_ratio lets you blend:

• L1 (Lasso) penalty — makes some coefficients exactly zero
• L2 (Ridge) penalty — shrinks all coefficients smoothly

So, if :

• l1_ratio = 0 ➜ 100 % Ridge (L2)
• l1_ratio = 0.5 ➜ Half Ridge, Half Lasso (Balanced)
• l1_ratio = 1.0 ➜ 100% Lasso (L1)

IMPORTANT NOTE : In this formula:

• **λ = alpha in code**
• **α = l1_ratio in code**

Why use Elastic Net?

• Like Lasso, it can drop irrelevant features.
• Like Ridge, it can handle correlated features better (Lasso struggles here).
• It’s more flexible — you tune both λ\lambdaλ and α\alphaα.

When to use Elastic Net?

• You have many features, possibly more than data points.
• Some features are correlated.
• You’re not sure if Lasso or Ridge will work better — so you blend them and let cross-validation pick the best mix.

So, the overall intuition is:

• Overfitting happens because minimizing loss alone lets the model use extreme coefficients to explain the data.
• Regularization adds a cost for large coefficients.
• This shrinks the solution space, keeping weights smaller, reducing sensitivity to noise, and hence reducing overfitting.
• L1 encourages sparsity (zeros), L2 encourages smallness (smoothness).

Code Lab: Your Hands-on Playground

Objective: Try Ridge, Lasso, and Elastic Net on the California Housing dataset and visualize how coefficients behave.

Let’s Implement

Step 1: Import the Necessary Libraries

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt
import numpy as np

Step 2 : Load the California Housing Dataset

# Load the data (features and target)
data = fetch_california_housing()
X, y = data.data, data.target

Step 3: Split the data

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 4: Initialize the Models

ridge = Ridge(alpha=1.0)
lasso = Lasso(alpha=0.1)
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5)

Step 5: Train the Models & Evaluate

models = {
    'Ridge': ridge,
    'Lasso': lasso,
    'ElasticNet': elastic
}

for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print(f"{name} MSE: {mean_squared_error(y_test, y_pred):.2f}")

Sample OUTPUT :

📊 Visualizing Coefficients

for name, model in models.items():
    plt.plot(model.coef_, label=name)

plt.title("Model Coefficients Comparison")
plt.xlabel("Feature Index")
plt.ylabel("Coefficient Value")
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.show()

Interpretation from output Graph:

➢ Ridge Regression (Blue Line):

• Adds L2 regularization (squares of the coefficients).
• Tends to shrink coefficients, but does not eliminate them (i.e., they rarely become exactly zero).
• As seen in the plot, Ridge assigns non-zero values to all features, even when some are small or negative.

➢ Lasso Regression (Orange Line):

• Adds L1 regularization (absolute values of the coefficients).
• Can shrink some coefficients to zero, effectively performing feature selection.
• In the graph, features at index 3 and 4 are assigned exactly zero, and several others are very close to zero.

➢ ElasticNet (Green Line):

• Combines both L1 and L2 regularization (controlled by the L1 ratio).
• Offers a balance between Ridge and Lasso.
• The Elastic Net line is smoother than Lasso but closer to zero than Ridge, combining the stability of Ridge and sparsity of Lasso.

🎮 Play with Regularization!

Try out interactive Streamlit playground

Points to remember

• Ridge shrinks coefficients but keeps all
• Lasso shrinks some to zero (feature selection!)
• Elastic Net = Best of both worlds

Today’s Task

Train all three models (Lasso, Ridge, Elastic Net) with different values of alpha (e.g., 0.01, 0.1, 1, 10) and visualize how coefficients change.

Bonus Resources

• Ridge vs Lasso Docs
• Visual Introduction to Lasso Regression
• Elastic Net Intuition Video

Previous & Next

• previous: *Linear Regression*
• next: *A Beginner’s Guide to Logistic Regression*

✨ Stay Curious, Stay Consistent “The best ML engineers are just consistent learners who got curious often.”

See you tomorrow, brain builder!