Your Optimizer Spends Its Whole Life One Step From Exploding. On Purpose.

There is a precise speed limit above which gradient descent flies apart. You would expect a well-behaved optimizer to stay safely below it.

Instead it drives straight up to the limit and then balances on the edge for the entire run, perpetually one nudge from disaster.

Start with the smallest possible example, because it contains everything.

Take the simplest bowl-shaped loss: one parameter, error equal to one-half times some curvature times the parameter squared.

Call the curvature λ. Gradient descent with step size η updates the parameter by multiplying it, each step, by the factor (1 − ηλ).

Now ask when this blows up.

If the size of (1 − ηλ) is less than one, the parameter shrinks toward the bottom — stable.

If it is bigger than one, the parameter grows each step — it explodes. The boundary sits at exactly:

λ = 2 / η

Below that curvature, you converge. Above it, you diverge.

Two over the learning rate is the cliff edge, and it is not a metaphor — it falls straight out of the arithmetic of multiplying by (1 − ηλ).

Real loss landscapes are not single bowls, but near any point they have a sharpest direction, with its own curvature

— the largest eigenvalue of the Hessian, the “sharpness.”

The same speed limit applies. If the sharpness exceeds two over the learning rate, the sharpest direction blows up.

So a sensible optimizer should keep the sharpness comfortably below the limit.

Here is what actually happens, documented by Cohen and colleagues in 2021.

Early in training, the sharpness climbs. The landscape gets steeper as the model learns — they called this progressive sharpening.

It climbs and climbs, heading straight for the speed limit. And when it arrives at exactly two over the learning rate, it does not stop short and it does not break through. It hovers.

For the rest of training the sharpness sits right at the threshold, the optimizer balanced on the knife’s edge of its own stability.

They named it the edge of stability. The model trains in the one regime classical optimization says it should avoid — right at the boundary where the next step could, in principle, send the sharpest direction flying.

Why would it choose to live there? The picture that emerges is a kind of self-correction.

When the model drifts a hair over the limit, the sharpest direction starts to blow up — and that very blow-up bounces the model into a flatter, less sharp region, pulling the sharpness back down below the limit.

Then progressive sharpening pushes it back up. The result is a constant, gentle oscillation across the boundary, the optimizer surfing the edge, never quite falling off.

And there may be a hidden gift in it. Flatter regions of the landscape tend to generalize better.

By repeatedly bouncing off the stability boundary, the optimizer is nudged, again and again, away from sharp minima and toward flat ones.

The danger and the benefit are the same mechanism. Living at the edge is what keeps pushing it toward the good kind of solution.

This rewrites the mental image most people carry. We picture training as a ball rolling smoothly down into a valley and settling.

The reality is jumpier and stranger — a system that accelerates until it is right at the threshold of instability, then rides that threshold, trading tiny controlled near-explosions for a slow drift toward flatness.

So when your loss is descending and everything looks calm, consider what the calm is made of. Not safety. A sharpness pinned to the exact value where one more step would diverge, held there by the same instability it is flirting with.

The question is almost vertiginous: if the smooth descent you see is really a balance on the edge of blowing up — what is holding it there, step after step, except the blowing up itself?

Source: Cohen, Kaur, Li, Kolter, Talwalkar — “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability” (2021), arXiv:2103.00065.