Weight Initialization in Neural Network: Part 2

In Part 1, we said good initialization keeps the signal at a stable scale across all layers, and bad initialization either explodes or vanishes the signal. We left it at the intuition.

Now the natural next question: how do we actually pick the right scale? If just right depends on fan-in(number of input to the layer) and the activation function, what’s the actual recipe? And why are there multiple methods like Xavier, Kaiming, orthogonal?

Introduction

A single layer in a neural network does this:

output = activation(W @ input + b)

W is the weight matrix. The input has some scale, let's say its values are roughly the size of standard random numbers, somewhere around magnitude 1. After multiplying by W, the output has some new scale that depends on W. Then we apply the activation, and pass it to the next layer.

If we keep going through 10 layers, that scale evolves at each step. Our goal: choose W so that the output of each layer has the same scale as the input.

If we can do that for one layer, we’ve done it for 10 or 100, because each layer preserves the scale. So the entire problem reduces to one question:

For one layer, what scale of weights keeps the output scale equal to the input scale?

Solve it for one layer, you’ve solved it for the whole network.

What do we even mean by “scale”?

Before we can answer that question with math, we have to pin down what “scale” actually means for a vector of numbers.

When we say “the signal at layer 5 is too big,” we’re talking about a vector of 256 numbers like [0.23, -0.45, 1.12, ...]. Saying it's too big must mean something about those 256 numbers collectively. We need a single number that summarizes, how big are these typically?

There are a few candidates. Could be the mean of the values. Could be the mean of absolute values. Could be the maximum. Could be the variance, which is the average squared distance from the mean.

The mean is useless here, because in well-initialized networks the activations are roughly zero on average — positives and negatives cancel. A signal of [1000, -1000, 999, -1001] has mean zero. So does [0.0001, -0.0002, 0.0001, -0.0001]. Mean tells you nothing about whether the signal is alive or dead.

Mean of absolute values would actually capture scale fine. So would the maximum. So why don’t we use those?

Because of the math. When you ask what happens to absolute values when you sum random numbers, the answer is messy. It depends on how often the signs cancel versus reinforce. There’s no clean formula. You can’t easily say, the mean absolute value of a sum is X times the mean absolute value of the inputs.

Variance, on the other hand, has two beautifully clean rules:

• Var(a + b) = Var(a) + Var(b) when a and b are independent
• Var(a · b) = Var(a) · Var(b) when a and b are independent and have mean zero

That’s it. Variances add when you sum independent things. Variances multiply when you multiply independent things. Both rules are exact.

And here’s the thing: neural networks are made of exactly two operations: multiplication and summation. A neuron multiplies inputs by weights, then sums them up. Multiply, sum. Multiply, sum. Layer after layer.

So we have two rules that tell us exactly what variance does under multiplication and summation, and a network that does nothing but multiplication and summation. Variance is the only measure of scale where we can actually predict what each layer does to the signal. Every other measure , mean absolute value, maximum, anything else, would leave us stuck.

This is why the whole field tracks variance. Not because it’s the most intuitive measure of, how big are these numbers, but because it’s the only one whose behavior is mathematically tractable through the operations we care about. Once you have a quantity you can track cleanly through layers, you can derive formulas. Without it, you’re just guessing.

So from here on, scale means variance. When we say keep the scale stable across layers, we mean keep the variance roughly equal to 1 at every layer. That’s the precise version of the intuition from Part 1.

The one-neuron picture

Now the math falls out naturally. Forget matrices for a moment — let’s look at one output neuron.

A single neuron computes:

y = w₁·x₁ + w₂·x₂ + w₃·x₃ + ... + wₙ·xₙ

where n is the fan-in, the number of inputs feeding into this neuron. If the previous layer has 256 neurons, fan-in is 256.

We want Var(y) = 1 if Var(x) = 1. Let's apply our two rules.

First, each term wᵢ·xᵢ is a product of two independent zero-mean things, so by the multiplication rule:

Var(wᵢ · xᵢ) = Var(wᵢ) · Var(xᵢ) = Var(w) · 1 = Var(w)

(assuming inputs have variance 1, and all weights are drawn from the same distribution).

Now y is a sum of n such terms, so by the summation rule:

Var(y) = n · Var(w)

We want Var(y) = 1, so:

n · Var(w) = 1
Var(w) = 1/n

Or equivalently, **std(w) = 1/√n** where n is fan-in.

That’s the whole derivation. Larger fan-in mean smaller weights. The formula isn’t arbitrary, it falls directly out of the two variance rules applied to the two operations a neuron does.

Take a moment with that. Everything else builds on it.

Method 1: Xavier / Glorot initialization (2010)

This was the first principled answer. Xavier Glorot and Yoshua Bengio published a paper in 2010 that said, use std = 1/√n. (Their actual recommendation was slightly differentstd = √(2/(fan_in + fan_out)) - accounting for both the forward pass and the backward pass, but the spirit is the same.)

Xavier was a huge deal in 2010. Before this paper, people initialized weights with arbitrary tiny random numbers (often U(-0.01, 0.01)), and deep networks just wouldn't train. After Xavier, networks 5-10 layers deep became trainable.

But Xavier assumes the activation function is roughly linear near zero, which is true for tanh and sigmoid. Then ReLU showed up, and the picture changed.

Method 2: Kaiming / He initialization (2015)

ReLU does something Xavier didn’t account for: it zeroes out half of its inputs on average. Negative inputs become zero; positive inputs pass through unchanged.

In variance terms: if you have 256 inputs flowing into a neuron and half of them get zeroed by ReLU, you effectively only have 128 contributing to the sum. The variance of the output is half of what Xavier predicted.

So you need to double the variance of your weights to compensate:

std = √(2/n)    instead of    std = √(1/n)

That extra factor of 2 came from Kaiming He’s 2015 paper, and it’s why “Kaiming init” exists separately from Xavier. Same idea, just adjusted for the fact that ReLU eats half the signal.

This is the default for ReLU-based networks. And it’s the reason ResNet-50 (a 50-layer CNN) could train at all in 2015, the original paper specifically credits Kaiming init alongside residual connections as the two things that made deep training possible.

Same idea as Xavier, one correction term. That’s it.

A subtle point: what about the backward pass?

Everything so far is about the forward pass, keeping the signal stable as it flows from input to output. But during training, gradients flow in the opposite direction: from the loss back to the inputs. Those gradients also go through a long chain of multiplications, and they can also explode or vanish.

For gradient stability, the relevant quantity is fan-out (how many neurons in the next layer receive this neuron’s output) rather than fan-in. And it turns out you can’t optimize for both simultaneously unless fan_in == fan_out, which usually isn't the case.

This is why the original Xavier paper used std = √(2/(fan_in + fan_out)) - it's a compromise between forward stability and backward stability.

Method 3: Orthogonal initialization (for RNNs)

For most networks the variance argument is enough. But RNNs have a special problem: they apply the same weight matrix at every time step. If you unroll an RNN over 100 time steps, you’re effectively multiplying by W a hundred times in a row.

Even with perfect Kaiming initialization for one application, multiplying by W 100 times will almost certainly explode or vanish, because random matrices, even well-scaled ones, have eigenvalues that aren't exactly 1. Eigenvalues > 1 explode; eigenvalues < 1 vanish.

The fix: initialize W to be an orthogonal matrix. Orthogonal matrices have all eigenvalues with magnitude exactly 1, so multiplying by them preserves vector lengths exactly, no matter how many times you apply them.

This is the standard trick for the recurrent weight matrix in RNNs. Same goal as before (keep the signal scale stable), different mechanism (use eigenvalue-1 matrices instead of variance arithmetic). The clever bit is recognizing that the recurrence structure changes the problem and needs a different solution.

When initialization isn’t enough

Here’s the honest limitation of all these methods: they only guarantee stability at the start of training. As soon as you take a gradient step, the weights change. After a few thousand steps, your beautifully-initialized weights drift, and the signal scale at layer 10 might no longer match the scale at layer 1.

Two architectural innovations were invented partly to address this:

Batch normalization (2015) doesn’t prevent drift , it actively renormalizes activations at every layer during the forward pass. Even if your weights wander into a bad regime, BN drags the activations back to a stable scale. It’s a continuous correction instead of a one-time setup.

Residual connections (2015) sidestep the problem entirely. Instead of h_new = f(h), they compute h_new = h + f(h). The + h part is an identity shortcut: even if f(h) produces garbage, the original signal h flows through unchanged. Suddenly it doesn't matter as much whether f preserves scale - the network has a clean signal path no matter what.

Both of these are partial solutions to a problem that good initialization started us toward. Initialization gets the network into a stable regime; BatchNorm and residuals keep it there during training.

That’s actually the trilogy of innovations that made deep learning possible: better init (start in a good place), batch norm (stay there), residual connections (have a safety net). Networks before 2015 were maxing out around 20 layers. After 2015, 100+ layers became routine.

The pattern across all methods

If you step back, every initialization method follows the same recipe:

1. Pick variance as the measure of signal scale.
2. Look at what the layer does to variance (multiplication adds variance, summation amplifies it, ReLU halves it).
3. Figure out the math: “for input variance to equal output variance, the weights need to have variance X.”
4. Draw random weights from a distribution with that variance.

Xavier did it for tanh networks. Kaiming did it for ReLU networks (one correction term). Orthogonal did it for RNNs (different mechanism for a different problem). Transformer init does it with empirically-tuned constants for transformer-specific quirks.

It’s not magic. It’s just careful variance accounting for what each architecture does to the signal.

Conclusion

Every initialization method is one idea applied carefully: pick variance as the measure of scale, then figure out what variance keeps it stable across layers, then draw weights from a distribution with that variance.

Xavier picks the right variance for tanh and sigmoid. Kaiming corrects it for ReLU. Orthogonal handles the special case of RNNs by using eigenvalue-1 matrices. Transformer init uses empirically-tuned constants. Three methods, three variations on the same recipe, all answering the question we started with in Part 1: how do we keep the signal alive across many layers?