Weight Initialization in Neural Network: Part 1

One of those topics that sounds boring until you realize a badly initialized network won’t train at all, no matter how good your data, optimizer, or architecture is.

This is Part 1 of two. Here in Part 1, the goal is intuition: what initialization even is, why it matters, and what it feels like when it goes wrong. No formulas, no method names, just the underlying idea. In Part 2, we get into the actual methods like Xavier, Kaiming, orthogonal, and the math that makes them work. If you finish this post thinking okay, I get why scale matters, but how do people actually pick the right scale? that’s exactly the question Part 2 picks up.

I was learning PyTorch and reading about weight initialization, and the tutorials kept throwing formulas like Xavier this, Kaiming that, “std = √(2/fan_in)”. I could follow the math but I had no idea why any of it mattered. I’d never thought about initialization before. The model gets some random starting weights, training fixes them, who cares?

Then I asked the question I should have asked from the start:

What does “good” vs “bad” initialization actually mean, and why does it matter at all?

This post is the slow answer to that question, the one that finally made it click for me.

What initialization even is?

Before training starts, the weights of a neural network have to start as some numbers. They can’t all be zero (that breaks things and we’ll come back to it). So we pick random numbers. The real question is: random numbers from where?

If I tell you pick a random weight, you might pick:

• 0.000003 (tiny number)
• 0.3 (medium number)
• 1123(huge number)

All of these are random. But they’re very different in scale.

Initialization is just the choice of what scale of random numbers to start with.

Why does scale matter?

Pick a number — say 2. Multiply it by itself 10 times. You get 1024. Multiply 20 times. You get a million. Multiply 30 times. You get a billion. The number explodes.

Now pick 0.5. Multiply it by itself 10 times. You get 0.001. Multiply 20 times. You get a millionth. Multiply 30 times, a billionth. The number vanishes toward zero.

Now pick 1.0. Multiply by itself any number of times. You get 1. Stable forever.

A neural network is essentially a long chain of multiplications. A 10-layer network multiplies your input by 10 weight matrices in a row. If those weights are a bit too big on average, the signal traveling through the network explodes. If they’re a bit too small, the signal vanishes to nothing. If they’re just right, the signal stays at a reasonable scale all the way through.

That’s the whole concept. Initialization is the choice of starting scale, and the network is sensitive to it because of the multiplication chain.

What bad initialization actually looks like?

Let me give you two concrete failure modes.

Bad case 1: weights too big, assume input is something normal like [0.5, 0.3, -0.2, …]. After layer 1, the values are [3.2, -1.8, 4.1, …]. After layer 2, [27, -41, 33, …]. After layer 5, they’re in the millions. After layer 10, you get inf, then NaN. Your loss is NaN. Training is dead before it started.

Bad case 2: weights too small, assume input is [0.5, 0.3, -0.2, …]. After layer 1, values are [0.02, 0.01, -0.008, …]. After layer 2, [0.0003, 0.0001, …]. After layer 10, the activations are essentially zero. The model outputs the same prediction for every input, because internally, everything has been crushed to nothing. Loss goes flat. The model can’t learn because the signal that should be propagating through the network has died.

You’ve probably seen both if you’ve trained anything:

• A NaN loss is almost always a weights too big problem somewhere.

• A loss that stays flat at the random-guess value is often a weights too small or weights all the same problem.

These aren’t exotic edge cases. They’re the two most common reasons a network refuses to train, and they’re both initialization issues.

What “good initialization” means

Good initialization keeps the signal at a sensible scale all the way through the network. The activations at layer 10 should be roughly the same size as the activations at layer 1. Not exploding, not vanishing. Just stable.

It’s a Goldilocks thing: weights need to be just big enough that the signal doesn’t die, but just small enough that the signal doesn’t blow up.

The exact just right value depends on two things:

• How many connections feed into each neuron: A layer with 1,000 inputs needs smaller weights than a layer with 10 inputs, because more numbers are getting added together.

• What activation function you’re using: Some activations shrink the signal(tanh, sigmoid) some clip half of it to zero (ReLU). The choice changes the right scale.

People figured out the formulas for this in 2010 and 2015 — that’s what Xavier and Kaiming initialization are. They’re just principled ways of picking the right scale based on those two factors.

Why we don’t just initialize to zero

A tempting question: if the issue is “are the weights too big or too small,” why not start with zero — the perfectly neutral middle?

Because then every neuron in a layer is doing exactly the same thing as every other neuron in that layer. They all start identical, they all get identical gradients during backprop, they all update identically, and they stay identical forever. A layer of 256 neurons collapses into effectively one neuron. You’ve thrown away most of your model.

That’s why we use random initialization instead of zero: to break the symmetry between neurons so they can learn different features. The randomness isn’t the magic — it’s just a way to make sure neurons start out different from each other. The scale of the randomness is the actual magic.

Conclusion

Good initialization: the signal stays at a reasonable scale across all layers, so the network can actually train.

Bad initialization: the signal either explodes (NaN) or vanishes (flat loss), and the network can’t learn.

PyTorch handles this for you with good defaults on standard layers. You only need to think about it manually when you build something unusual, when training fails for no obvious reason, or when you’re using an architecture that needs specific values.

But knowing why the defaults exist is one of those things that quietly changes how you read papers and how you debug. The next time someone mentions “vanishing gradients” or “exploding activations” or “why deep networks were hard to train before 2015,” you’ll know exactly what they mean. It’s all one phenomenon — the chain of multiplications wanting to either run away or die out, and decades of clever fixes (better init, BatchNorm, residual connections) all addressing the same fundamental problem.