Transformers & LLMs — Part 6: Layer Normalization, the Silent Stabilizer

In the previous parts of this series, we focused on what Transformers do: how attention replaces recurrence, how information flows across tokens, and how massive parallelism becomes possible.

In this part, we focus on something less visible, but just as critical:

Why Transformers don’t collapse numerically while doing all of that.

The answer is Layer Normalization.

This article is mainly inspired by Stanford’s CME295 lecture on LMMs & Transformers, where LayerNorm is framed not as a minor trick, but as a core structural component that enables depth and stability (Stanford CME295, 2025).

The Core Problem: Activations That Drift

Let’s start with a simple example.

Suppose a token is represented by the following hidden vector:

x = [0.6, 1.1, -0.4, 2.0]

This vector passes through:

• Self-attention (the token looks at other tokens in the sequence and gathers relevant contextual information)
• A residual connection (the original token representation is added back to preserve existing information)
• A feed-forward network (each token is independently transformed to refine and combine its features)
• Another residual connection (the refined features are added to the previous representation to avoid information loss)

After several layers, that same token might look like:

x' = [18.7, -12.3, 4.9, 31.2]

Nothing is “wrong” mathematically (the equations are still valid) — but numerically, things are getting unstable (values drift to extreme scales).

• Gradients explode or vanish (error signals become too large or too small to be useful)
• Learning rates must shrink (training becomes sensitive to step size)
• Training becomes fragile (small changes can break convergence)

In deep Transformers, this problem compounds quickly. Layer Normalization exists to keep every token’s representation under control, layer after layer.

What Layer Normalization Actually Does

Layer Normalization operates within a single token.

Take this hidden state:

x = [2.0, 4.0, 6.0, 8.0]

Step 1: Compute statistics across features

• Mean = 5.0
• Standard deviation ≈ 2.24

Step 2: Normalize

(x - mean) / std ≈ [-1.34, -0.45, 0.45, 1.34]

Now:

• The vector has zero mean
• Unit variance
• No dimension dominates

Step 3: Restore flexibility

After normalization, LayerNorm applies two learnable parameters to every feature:

• γ (gamma) → scale
• β (beta) → shift

So the full operation becomes:

LayerNorm(x) = γ · x̂ + β

where x̂ is the normalized vector (zero mean, unit variance).

This step is crucial.

At first glance, normalization seems restrictive — after all, we just forced the vector to have a fixed scale. So why doesn’t this limit what the model can represent?

The key idea is this:

Normalization standardizes the input, but γ and β let the model decide how much of that standardization it wants to keep.

What Does “Reintroduce Scale” Actually Mean?

Let’s look at a concrete example.

After normalization, a token might look like this:

x̂ = [-1.2, -0.3, 0.4, 1.1]

Now suppose the model learns:

γ = [5.0, 5.0, 5.0, 5.0]
β = [0.0, 0.0, 0.0, 0.0]

Then the output becomes:

γ · x̂ + β = [-6.0, -1.5, 2.0, 5.5]

Notice what just happened:

• Normalization did not permanently shrink the signal
• The model simply scaled it back up in a controlled way
• The only difference is that the scaling is now learned, not accidental

This is what we mean by reintroducing scale.

Why This Does NOT Reduce Model Capacity

If the model wants:

• Large activations → γ becomes large
• Small activations → γ becomes small
• Shifted distributions → β moves them

In the extreme case, the model could even learn parameters that almost undo normalization entirely.

So from a representational standpoint:

LayerNorm does not restrict what the model can express — it only stabilizes how the model gets there during training.

This is why Ba et al. emphasize that Layer Normalization improves optimization without reducing expressive power.

An Intuitive Analogy

Think of normalization like adjusting all audio tracks to the same volume before mixing.

• This prevents clipping and distortion
• But each track still has its own volume knob afterward

γ and β are those volume knobs.

The model first ensures everything is well-behaved, then decides how loud or quiet each feature should actually be.

Why This Matters in Practice

Without γ and β:

• Normalization would force every layer to look statistically identical
• The model would lose flexibility

With γ and β:

• Training stays stable
• Representations stay expressive
• Deep Transformers become possible

📘 Reference:

• Ba et al., 2016. Layer Normalization. arXiv PDF

Why Not Batch Normalization?

Batch Normalization normalizes activations across the batch, meaning it relies on statistics computed from multiple examples at once.

This works well in computer vision, where:

• Batch sizes are large
• Inputs have fixed structure
• Training and inference look similar

In NLP, this assumption breaks down.

During autoregressive decoding, a language model often runs with:

• Batch size = 1
• Tokens generated one at a time
• No meaningful batch statistics to normalize against

In this setting, BatchNorm becomes unreliable or unusable.

Layer Normalization avoids this problem entirely.

Instead of looking across the batch, LayerNorm:

• Normalizes within each token
• Behaves identically during training and inference
• Handles variable-length sequences naturally

This is why LayerNorm replaced BatchNorm in RNNs and became the standard choice for Transformers.

📘 Reference:

• Ba et al., 2016. Layer Normalization. arXiv PDF

LayerNorm Inside a Transformer Block

Let’s walk through a Transformer sublayer step by step.

1. Start with token representation x
2. Apply self-attention → Attention(x)
3. Add the residual connection:

y = x + Attention(x)

4. Normalize

z = LayerNorm(y)

Why normalize after the residual?

Because residual connections accumulate signals. Even small increases in variance stack across layers.

LayerNorm acts as a stabilizing checkpoint after each addition.

📘 Reference:

• Vaswani et al., 2017. Attention Is All You Need. NeurIPS PDF

Post-Norm vs. Pre-Norm: A Small Change with Big Impact

Post-Norm (Original Transformer)

The original Transformer applied normalization after the residual connection:

LayerNorm(x + Sublayer(x))

This worked well for relatively shallow models. However, as Transformers became deeper, training became unstable.

Why?

Because gradients had to pass through LayerNorm at every layer, which weakened learning signals and made optimization difficult in very deep networks.

Pre-Norm (Modern Transformers)

Modern architectures flipped the order:

x + Sublayer(LayerNorm(x))

This simple change makes a big difference.

With Pre-Norm:

• The residual path becomes a clean gradient highway
• Gradients can flow directly through the identity connection
• Even if a sublayer behaves poorly early in training, learning remains stable

As a result, deep Transformers become practical and reliable to train.

📘 Reference:

• Nguyen & Salazar, 2019. Transformers without Tears. arXiv PDF

An Intuition from Signal Processing

Think of residual connections as adding sound waves.

Each sublayer produces a new wave and adds it to the existing signal. Without normalization, these waves accumulate, the volume keeps increasing, and the signal eventually clips.

Layer Normalization acts like automatic volume control: it stabilizes the signal before amplification, ensuring that each new contribution is added on a well-behaved baseline.

Because this control is applied independently to every token, the overall signal remains stable — even as the network grows deeper.

RMSNorm: A Leaner Alternative

RMSNorm simplifies LayerNorm by removing mean subtraction.

Instead of:

(x - mean) / std

It uses:

x / RMS(x)

Surprisingly, this often works just as well:

• Similar performance
• Fewer operations
• Fewer parameters

This makes RMSNorm attractive for large-scale language models.

📘 Reference:

• Zhang & Sennrich, 2019. Root Mean Square Layer Normalization. arXiv PDF

Why Layer Normalization Matters

LayerNorm doesn’t help a model understand language better. It helps the model stay numerically sane while learning.

Without it:

• Residual connections destabilize
• Attention amplifies variance
• Depth becomes a liability

With it:

• Optimization becomes predictable
• Training scales to hundreds of layers
• Transformers become practical

LayerNorm is quiet — but without it, modern LLMs wouldn’t exist. Here you can find the infographic of the LayerNorm:

What’s Next

So far, we’ve focused on stability across depth. But there’s another problem waiting as sequences grow longer:

Full self-attention scales quadratically.

In the next part, we’ll explore Sparse Attention — how restricting who attends to whom makes long-context Transformers possible.