Beyond the Goldfish Memory: How I Stopped My Neural Networks from Forgetting

Human beings are naturally gifted at building knowledge over time. If you learn to play the piano and then learn to play the guitar, you don’t suddenly forget how to play the piano. In the world of Artificial Intelligence, however, this is a massive hurdle known as catastrophic forgetting.

In this article, I share my implementation and hyperparameter evaluation of a landmark solution to this problem: Elastic Weight Consolidation (EWC).

The Research Context

The core of this project is based on the 2017 paper “Overcoming catastrophic forgetting in neural networks” by James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, and a team of researchers at DeepMind and Imperial College London.

The researchers noted that while standard neural networks abruptly lose knowledge of Task A when trained on Task B, biological brains avoid this by protecting previously strengthened synapses. They translated this biological mechanism into an algorithm that slows down learning on weights that are “important” for previous tasks.

How EWC Works

EWC doesn’t just lock weights in place; it treats them like they are attached to a spring.

1. Fisher Information Matrix (F): After training on Task A, the algorithm calculates which weights were most critical to its success. This is done using the diagonal of the Fisher Information Matrix, which represents the second derivative of the loss near a minimum.
2. The Penalty Loss: When training on Task B, EWC adds a quadratic penalty to the loss function.

EWC Loss Formula

In this equation, L_B is the new task’s loss, and the second term is the “pull” from the old task. The variable λ sets how much the network should prioritize the old task over the new one.

My Implementation: Permuted MNIST

To test the limits of EWC, I followed the paper’s experimental setup using Permuted MNIST.

In this setup:

• Task A is a standard MNIST classification.
• Task B is the same MNIST dataset, but every pixel has been randomly shuffled (permuted).

For a standard network, Task B would destroy the accuracy of Task A. However, by using EWC, I was able to maintain high performance on both.

The Search for the Best λ

The paper emphasizes that λ (the Fisher multiplier) is critical. In my implementation, I focused on finding the best value for the EWC lambda to balance plasticity (learning the new) and stability (keeping the old).

• When λ is too low: The network behaves like plain SGD, and Task A’s performance crashes.
• When λ is too high: The weights are too rigid, and the network fails to learn Task B effectively.

I observed the best value for λ to be 0.05 for the permuted MNIST dataset.

Results & Repository

My experiments validated the paper’s findings: EWC allows a single network with fixed capacity to learn multiple tasks in sequence. By intelligently “squeezing” new functionality into less important weights, the network can share representations and preserve knowledge.

You can view my full code, the evaluation logic, and my findings on the optimal lambda values here:

Github : https://github.com/vedant-spatil/EWC-CatastrophicForgetting

Conclusion

EWC is a brilliant example of how biological inspiration can solve engineering bottlenecks in AI. It proves that the “lack of algorithms to support continual learning” is a barrier we are rapidly breaking down.

If you found this implementation helpful, please check out the repository and feel free to reach out with questions!

References: Kirkpatrick, J., et al. (2017). “Overcoming catastrophic forgetting in neural networks.” Proceedings of the National Academy of Sciences (PNAS).