Stop Messing with the Loss Function: The Smarter Way to Fix Catastrophic Forgetting in LLMs

Fine-tuning large language models to adapt to new tasks has become standard operating procedure for almost every AI team. But the inevitable hangover is catastrophic forgetting — the moment the model picks up new knowledge, some of its previously mastered capabilities start to degrade.

The mainstream fix has always been messing around with the loss function. Think suppressing high-loss tokens or using Elastic Weight Consolidation (EWC) to constrain weights that are critical to the original tasks.

But there is a massive paradox here: high-loss tokens are often exactly the “hard examples” you actually need the model to learn. This is especially true when you are adapting a model to low-resource languages or injecting specialized domain knowledge that barely existed in the pre-training data. If you just chop off the gradients for these tokens (like the WaRE approach does), the model will never truly master the new task.

This is where a new paper steps in with a brilliant shift in perspective. The authors mathematically prove that the amount of forgetting at each training step is approximately upper-bounded by a specific metric: the learning rate multiplied by the square root of the current training loss.

The intuition behind this bound is a game-changer. When the batch loss is sky-high, the model’s updates are massive and chaotic, making it incredibly easy to overwrite old knowledge. Conversely, when the loss is already low (meaning the model is converging), even a slightly larger learning rate is relatively safe.

In other words, the root of the problem isn’t which tokens you penalize in the loss function; it’s when and how you apply your learning rate.

The authors turned this theory into a practical framework called FINCH, which is essentially a loss-adaptive learning rate scheduler. It monitors the current batch loss in real-time: if the loss spikes, it dials back the learning rate; if the loss drops, it dynamically bumps it up.

The beauty of FINCH lies in its simplicity. It leaves the training objective completely untouched and requires zero data replay (no need to mix in old task data during training). It simply and dynamically recalibrates the step size of every single update.

The results speak for themselves. Across three major benchmarks — knowledge injection, scientific reasoning, and low-resource language adaptation — FINCH slashed forgetting by an average of 93%, all while maintaining identical performance on the new tasks compared to standard fine-tuning.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)