Ilya’s 30 Papers, Part 9: ResNets (or blending in your leftovers)

In my experience, the most revolutionary ideas are the simplest ones, the kind that make you think, “Cool… why didn’t I do that?”

ResNets are one of those ideas.

Back in 2015, when the ResNet (Residual Network) paper came out, people were struggling to train really deep neural networks. Once you got past a few dozen layers, things tended to get messy, including overfitting and other training headaches (there are great explanations of overfitting out there, so I’ll spare us both).

ResNet’s big contribution was a small change with huge impact: residual connections, often called skip connections now because, honestly, that name is way easier to say.

The leftovers analogy

Think of a bowl of leftover soup. You microwave it, and it’s always too hot. You could wait… or you could do something clever.

You split the soup into two bowls:

• Bowl A stays out of the microwave and it’s too cold.
• Bowl B goes in the microwave, and after cooking, it’s too hot.

Then you recombine them. The cold soup cools the hot soup, and now the whole thing is the right temperature.

That’s a residual connection.

• The soup is the input vector. (Be it known: I have just coined the term “soup vector”.)
• The microwave is the layer.
• The microwaved soup is the layer output.
• Mixing the two bowls is adding the input back into the output.

The core idea

You start with a list of numbers (a vector) and feed it into a layer. The layer transforms it into a new vector.

[.1, .2, .3, .4] → (layer) → [.2, .4, .6, .8]

In a normal network, you’d take that output and pass it straight into the next layer:

[.1, .2, .3, .4] → (layer 1) → [.2, .4, .6, .8] → (layer 2) → ...

ResNets change one thing: before sending the output to the next layer, you add the original input back in.

So instead you do this:

1. Run the layer:

[.1, .2, .3, .4] → (layer 1) → [.2, .4, .6, .8]

2. Add the original input back, element by element:

[.1, .2, .3, .4] + [.2, .4, .6, .8] = [.3, .6, .9, 1.2]

3. Feed that into the next layer:

[.3, .6, .9, 1.2] → (layer 2) → ...

That’s it. That’s the trick. We’re just doing element-wise addition: “take what you learned” plus “don’t forget what you started with.”

It’s worth noting that in 2015, other research teams had tried something similar: they allowed adding the input to the output vectors, but the input vectors were still weighted using a learned parameter. They didn’t get the amazing results that ResNets got. The other research teams were thinking too hard and never tried the absolute simplest solution: just straight adding the inputs to the outputs.

Also worth noting: transformers, the architecture behind ChatGPT, uses residual connections.

DeepSeek just came out with something they call Manifold-constrained hyper-connections. It’s a lot like residual connections but with extra steps. Instead of just adding the input to the output, the input gets transformed by a learned weight matrix. That weight matrix is constrained so that all the rows and columns must add up to 1, and no numbers can be negative.

“Wait… what if the vectors aren’t the same size?”

Yep. I said element-wise addition, which only works cleanly if the vector going into the layer has the same shape as the vector coming out.

But in real neural nets, sizes often change, channels expand, dimensions shrink, and everything refuses to stay nicely compatible.

So how do ResNets handle that?

More on that in the next post in this series…

BTW I am writing a book about AI for people who don’t know how to code. It will be out on Amazon later next week.