Bias Variance Tradeoff and Double Descent

In Bias-Variance Tradeoff they teach us that we have to find the “sweet spot” between bias and variance where the test error is minimized.

Fundamental changes in Machine Learning

In the past few years there has been tests and observation showing that running the model past the interpolation threshold that the model could descend a second time. This was especially observed in deep learning. This paper was the first to describe this phenomenon [1].

Understanding Double Descent

To understand double descent, let’s check out a simple example that relates to the double descent: natural cubic splines.

What’s a spline?

A spline is a way to fit a smooth curve to a set of data points. Imagine you have some points on a graph and you want to draw a smooth line that goes through or near these points. A spline helps you do that by using several small, connected polynomial pieces to make a smooth curve.

• Unlike simple models that assume a specific shape for the curve (like a straight line), a spline doesn’t assume a particular shape. It can bend and flex to fit the data.
• The spline is made up of small polynomial segments (little curves) joined together at specific points called knots. These segments fit together smoothly, making the overall curve look nice and continuous.
• Think of basis functions as building blocks or small pieces of the curve. The number of these pieces determines how flexible the spline is. More pieces mean the spline can bend more to fit the data closely.
• The number of basis functions is also called the degrees of freedom (DF). More DF means a more flexible curve.
• To fit the spline to your data, we use a method called least squares. This means we adjust the spline so that the overall difference between the data points and the spline’s curve is as small as possible.

Suppose we have 20 pairs of data points, where each pair consists of an X value and a Y value. We want to estimate the true relationship between X and Y using a smooth curve called a spline. In this case, the true relationship is Y=sin⁡(X) + episilon.

So, in the graph:

• Gray dots represent the 20 data points.
• The black curve represents the true function f(X)=sin⁡(X).
• The light blue curve represents the fitted spline with 4 degrees of freedom.

Now lets do it with 20 data points, and we try to fit a spline with 20 degrees of freedom. This means the spline is very flexible and can adjust to every single data point.

Why This is a Bad Idea:

• To fit a spline with 20 degrees of freedom, we use a method called least squares with 20 features.
• This approach will result in a spline that passes through every single data point perfectly, leading to zero training error. This is known as interpolation.
• However, this will also mean the spline is fitting the noise in the data, not just the underlying pattern. As a result, the spline will perform poorly on new, unseen data, resulting in high test error.

Now what if we fit a spline with 36 DF. This is bad news and would lead to overfitting since there are only 20 observations.

The solution won’t be unique! To select among the infinite number of solutions, I choose the “minimum” norm fit: the one with the smallest sum of squared coefficients.

Here’s what you get:

Let’s compare the results with 20 DF to 36 DF. It looks like 36 degrees of freedom did a little better.

We can take a peek at the training and test error: Why did the test error decrease when p>n? Isn’t that literally the opposite of what the bias-variance trade-off says should happen?

Understanding Double Descent with Splines

Fitting a Spline with 20 Degrees of Freedom (DF):

• If we have 20 data points and use a spline with 20 degrees of freedom, the model will fit the data perfectly, resulting in zero training error. This means the spline will pass through every data point exactly.
• However, this perfect fit includes all the noise in the data, leading to a very wiggly spline and poor performance on new data (high test error).

Increasing DF to More than 20:

• When we increase the degrees of freedom beyond the number of data points (p > n), there are many possible ways to fit the data perfectly.
• Among these many fits, we can choose the “minimum norm” fit, which is the least wiggly and most stable one.
• Surprisingly, this minimum norm fit is less wiggly and more stable than the spline with exactly 20 degrees of freedom (p = n).

Double Descent:

• This phenomenon is known as “double descent.” It happens because, as we increase the degrees of freedom, the model first overfits (too wiggly) and then, with even more flexibility, finds a smoother, more stable fit.
• In deep learning, when we use gradient descent to train neural networks, we often end up with this minimum norm solution, similar to the least wiggly spline.

Key Takeaway:

Double descent is a real phenomenon. It can be understood through the bias-variance trade-off in statistics. The more flexible models first overfit, then find smoother solutions as flexibility increases further.

In summary, double descent shows that more flexibility can sometimes lead to better generalization, and this is not magic but a concept grounded in statistical learning theory.

References

[1] https://arxiv.org/pdf/1812.11118

https://arjunahuja.medium.com/double-descent-8f92dfdc442f

https://x.com/daniela_witten/status/1292293125440847873