🚀 Layer Normalization Guide

Stabilise the neural network with a simple technique!

Photo by Google DeepMind on Unsplash

Introduction

There are several challenges to efficiently training a deep neural network due to various moving elements within the network, particularly the activation units in the hidden layer.

Key challenges include:

• Internal Covariate Shift — During training, as the parameters (weights and biases) of each layer are updated, the statistical distribution of the inputs to subsequent layers changes. This constant shifting of input distributions for each layer is Internal Covariate Shift.
• Exploding and vanishing gradient — Abrupt changes to parameter values.

What is Layer Normalization?

Introduced in 2016 by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton, Layer Normalization, also known as LayerNorm. It normalizes the inputs within a single training example, across all of its features (or hidden units) in a given layer.

Blog

This per-sample normalization has key advantages:

• Batch Size Independence: LayerNorm’s calculations are performed independently for each sample, making it effective regardless of batch size. This is particularly beneficial for RNNs and Transformer models.
• Consistent Behavior: It operates identically during both training and inference phases, simplifying deployment and reducing potential discrepancies.

Why Layer Normalization?

Layer Normalization acts as a stabilizing mechanism, bringing consistency to these internal dynamics and enabling more effective training. This technique is not merely an incremental improvement but a crucial component that has facilitated the development of the powerful and deep architectures prevalent today.

The Layer Normalization Process

LayerNorm performs the following steps:

1. Calculate Mean (μ): The average of all feature activations for that single sample within the current layer is computed. This represents the central tendency of features for that instance. μ = sum_of_activations / number_of_features
2. Calculate Standard Deviation (σ): The standard deviation of these activations from their mean is calculated. This measures the dispersion or scale of features for the sample. σ = sqrt(average_of_squared_differences_from_μ + ϵ) where ϵ is a small constant for numerical stability.
3. Normalize: Each feature activation is then re-centered and re-scaled by subtracting the mean (μ) and dividing by the standard deviation (σ). This standardizes the activations for the sample to have approximately zero mean and unit variance across its features. normalized_feature = (original_feature - μ) / σ
4. Scale and Shift: To preserve the network’s representational capacity, an affine transformation is applied using two learnable parameter vectors per layer:

• Gain (g, or γ): A scaling factor.
• Bias (b, or β): A shifting factor. The final output is: final_output = g * normalized_feature + b These parameters are learned during training, allowing the network to modulate the normalized values if the strict zero-mean, unit-variance constraint is suboptimal for a particular layer.

This method of normalizing across features for each sample is the defining characteristic of LayerNorm.

🌟 Sample 3x3 Layer Normalization Walkthrough

Let’s assume the input matrix A (3 rows, 3 columns) is the following:

The Layer Normalization is applied independently to each row (sample) of the matrix.

Row 1: (1, 2, 3)

The normalization steps (Mean, Variance, Normalize) are applied to the elements A_1 = (1, 2, 3).

1. Calculate Mean (mu):

2. Calculate Variance (sigma²):

3. Calculate Standard Deviation (sigma): Ignoring the small epsilon=1e-7 for simplicity.

4. Normalize (Output Row B_1)

Row 2: (10, 11, 12)

The normalization steps are applied to the elements A_2 = (10, 11, 12).

1. Calculate Mean (mu):

2. Calculate Variance (sigma²):

3. Calculate Standard Deviation (sigma):

4. Normalization (Output Row B_2):

Row 3: (0, 0.5, 1)

The normalization steps are applied to the elements A_3 = (0, 0.5, 1).

1. Calculate Mean (mu):

2. Calculate Variance (sigma²):

3. Calculate Standard Deviation (sigma):

4. Normalization (Output Row B_3):

Final Input and Output

Sample Input Matrix A:

Normalized Output Matrix B:

Observation: Notice that although the input rows had vastly different magnitudes (1 to 3 vs. 10 to 12), the output rows have the same mean (0) and standard deviation (1), demonstrating how Layer Normalization scales the data to a uniform distribution within each row.

Wrapping Up

Layer Normalization is a fundamental technique in modern deep learning. By providing stability and consistency to the training process, it enables the development and effective training of the deep and complex neural network architectures that are pushing the frontiers of artificial intelligence.

Thanks for reading!

Connect with the author here:

LinkedIn | YouTube | Threads | Twitter | Instagram | Facebook