What is vanishing gradient problem?

What happens?

• During **backpropagation**, gradients are passed backward from output to input layers
• In deep networks, these gradients get multiplied many times
• If they are small (< 1), they shrink exponentially

Result: Earlier layers (closer to input) get almost no updates

Why it’s a problem

1. Slow or no learning

• Early layers stop** learning** because gradients ≈ 0

1. Poor performance

• Network fails to capture important features

1. Training becomes unstable

• Deep networks become hard to train

Main causes

• Using activation functions like sigmoid or tanh
• Deep architectures (many layers)
• Poor weight initialization

Example intuition

Imagine passing a small number like 0.5 through **many multiplications**:

0.5 × 0.5 × 0.5 × … → becomes almost 0

That’s what happens to gradients.

How to solve it

• Use ReLU (Rectified Linear Unit) instead of sigmoid/tanh
• Proper weight initialization (e.g., Xavier, He initialization)
• Use Batch Normalization
• Use architectures like:
• LSTM / GRU (for sequences)
• Residual Networks (ResNet)

Simple idea

The vanishing gradient problem means:

“The network forgets how to learn in earlier layers.”