Weight initialization in Neural Networks

Xavier/Glorot initialization and He initialization in depth

In the realm of neural networks, weight initialization plays a pivotal role in shaping the learning process and overall performance of the model. At its core, weight initialization refers to the process of setting initial values for the weights of connections between neurons in a neural network. These initial values serve as the starting point for the iterative optimization process during training.

Before we go deep into this, lets understand the weights.Imagine you have a recipe book for baking cakes. In this book, there’s a special ingredient called “weights.” These weights decide how much of each ingredient, like flour, sugar, and eggs, to use in your cake recipe.

Now, when you start making a new cake, you don’t know the perfect amount of each ingredient yet as this is your first time. That’s where weight initialization comes in! It’s like guessing the right starting amounts for all the ingredients before you actually start baking.

[embed]

If you guess too little flour, your cake might be too flat and gooey. But if you guess too much flour, it could be too dry and hard. So, getting the right initial guesses for the weights is super important to make sure your cake (or in this case, your neural network) turns out just right!

[embed]

**Importance of Weight Initialization **The choice of weight initialization method can significantly impact the convergence speed, training stability, and generalization ability of neural networks. Properly initialized weights can help prevent issues such as vanishing gradients, exploding gradients, and slow convergence, leading to more efficient and effective learning. Remember weights will go as inputs yo activation function

First of all, you might think why just not initialize them all to zero or mat be with same value? It turns out that’s a terrible idea because if you remember from the chain rule, then the net input value coming to the neuron is just the input times the weight [xi wi].And since the weights are all zeros, then you’d get a net input value to neuron is zero.And at the end you’ll get an output of zero and your weights will never be adjusted and thus no learning will occur.*

Mathematics behind initialization with Zero

As you could see above , my all gradients become equal which leads us to “symmetry breaking problem”

Because all neurons are updating in the same way, they end up representing the same feature.This lack of diversity means the network can't learn different aspects of the data.

This means that no matter how long you train the network, all the neurons in each layer will behave the same, and they will effectively represent the same feature. Hence, the network could not learn the non-linearity and complex patterns in the data.

So now we know that we shouldn’t initialize them to zero. So maybe just initialize them to random numbers. Well, it’s close but still not correct. It turns out that’s also not a good idea because you might end up with the vanishing gradient problem or the exploding gradient problem. And also another thing that you should understand is, different outputs will have different ranges.Because there’s no range you’re initializing your values, your weights from, you’re just throwing random numbers and therefore these standard numbers will have a specific range. And so therefore their output will also have specific range.And that’s definitely not good for the neural network.

So what’s the solution?

We sill initialize them with random numbers, however, we take from a specific range, And that turns out to be one solution actually.So in just instead of just throwing randomly initialized values, then we take these values from a specific range. We take them from a normal distribution with a mean of zero and a standard deviation of one.

There are better methods to initialized the weights is so that could control the variance of the outputs. There are several techniques for weight initialization, each designed to address specific challenges and optimize training performance. Common methods include

• Random initialization
• Xavier/Glorot initialization
• He initialization …. and more.

Note — Our end goal is the outputs are produced by the neurons have a similar distribution

Where nin in the number of input neurons coming towards the neuron

With ReLU activation → He initialization

The main characteristic of the ReLU activation function is that it outputs zero for all negative input values and a linear function for positive input values. However, ReLU can lead to a problem known as “dying ReLU neurons,” where some neurons become inactive (outputting zero) for all inputs during training. This can happen if the initial weights are set such that the neurons always receive negative inputs, causing them to always output zero.

He initialization addresses this issue by scaling the initial weights based on the number of input connections to each neuron. It sets the initial weights using a normal distribution with a mean of zero and a variance of 2number of input connections number of input connections2​ (or equivalently, a standard deviation of 2number of input connections number of input connections2​​). This scaling factor helps prevent neurons from starting in the saturated (zero output) region of the ReLU function, promoting better learning and reducing the risk of dying ReLU neurons.

In summary, He initialization is preferred for ReLU activation functions because it helps mitigate the issue of dying ReLU neurons and encourages better training dynamics in deep neural networks with ReLU activations.

With Sigmoid activation → Xavier initialization

Xavier/Glorot initialization is commonly used for weight initialization when the layers have sigmoid activation functions (including tanh, which is a scaled version of the sigmoid function). The rationale behind using Xavier/Glorot initialization with sigmoid-like activation functions includes the following key points:

1. Mitigating Vanishing/Exploding Gradients: Sigmoid activation functions tend to saturate (flatten out) for large positive or negative inputs, which can lead to vanishing gradients during training. Xavier/Glorot initialization is designed to scale the initial weights in a way that balances the gradients and activations, reducing the likelihood of vanishing gradients and promoting stable learning.
2. Balanced Initialization: Xavier/Glorot initialization sets the initial weights using a normal distribution with a mean of zero and a variance that depends on the number of input and output connections to each neuron. This balanced initialization helps ensure that the activations and gradients are neither too small (leading to vanishing gradients) nor too large (leading to exploding gradients) at the start of training.
3. Non-Zero Mean Activation: Sigmoid and tanh activation functions have non-zero mean values, unlike some other activation functions like ReLU. Xavier/Glorot initialization takes this into account by adjusting the variance of the initial weights based on the activation function’s characteristics, which helps prevent saturation and promotes efficient learning.

Print the weights in each layers

[embed]

Overall, using Xavier/Glorot initialization with sigmoid activation functions helps address challenges such as vanishing gradients, encourages stable training dynamics, and contributes to better learning performance in deep neural networks.

You might ask why we initialized the weights [both in He and Xavier/Glorot] with small values.

[embed]

1. The window of any activation function and its derivative is very small. This means that when the input to an activation function is large, the derivative of the function becomes extremely small. By starting with smaller weights, we reduce the likelihood of large inputs, which helps prevent the activation function from saturating and becoming insensitive to changes in input.

2. Weight updates, which involve matrix multiplication, happen in parallel. In other words, all weights must be updated simultaneously. Using smaller weights helps in efficient parallel processing, especially on GPU (Graphics Processing Unit) devices. The GPU’s VRAM (Video Random Access Memory) and cores can handle smaller values more effectively, leading to faster and more efficient weight updates during training.

Conclusion

In conclusion, weight initialization is a critical aspect of training neural networks effectively. By setting the initial values of weights appropriately, we can prevent common issues like vanishing or exploding gradients, promote stable training dynamics, and improve the overall learning efficiency of the network.

Throughout this journey into the weighty world of neural networks, we’ve explored various initialization methods such as random initialization, Xavier/Glorot initialization, and He initialization. Each method has its advantages and is tailored to work best with specific activation functions, ensuring that the network learns efficiently and generalizes well to unseen data.

As we continue to delve deeper into the complexities of neural networks, understanding the nuances of weight initialization remains a key pillar in building robust and high-performing models. So remember, the right weights set the stage for neural networks to bake up success in solving a wide range of tasks, from image classification to natural language processing and beyond.

By ensuring that weights are initialized to small random values, we allow each neuron to learn independently, capturing a wide range of features and patterns within the data, thus leveraging the full potential of neural networks.

References :- Srikanth Varma sir Harshit Gupta The Complete Neural Networks Bootcamp: Theory, Applications