Why Reinforcement Learning, Not Just Math, Is Powering the Next Leap in Language Models

For the past year, if you’ve been watching the progress of large language models (LLMs), you’ve probably noticed the math obsession. Leaderboards flooded with new records on MATH and AIME benchmarks. Chain-of-thought prompts everywhere. Math, it seems, became the gold standard for reasoning.

But while LLMs were getting better at solving equations, many weren’t getting better at… anything else.

A new paper from researchers at CMU ( Carnegie Mellon University ), UPenn, and others asked the exact question I’ve been wondering: Does tuning LLMs for math actually make them better in general?

Their answer is, Yes but only if you use Reinforcement Learning (RL), not Supervised Fine-Tuning (SFT). And the difference is bigger than you’d expect.

Not All Fine-Tuning Is Equal

To test how math-focused training affects broader skills, the researchers fine-tuned identical base models in two ways:

• SFT (Supervised Fine-Tuning): Feed the model expert solutions, make it imitate.
• RL (Reinforcement Learning): Let the model explore, then reward only correct final answers.

Both methods used the same high-quality math dataset. The results?

• RL-tuned models improved on math AND generalized to other domains like science QA, agent planning, and even normal instruction following.
• SFT-tuned models often got worse outside math. Worse than the original base model in some cases.

The phenomenon is known as catastrophic forgetting: when a model overfits to one skill and forgets others. And SFT seems especially prone to it.

RL Doesn’t Just Learn Math — It Learns How to Learn

What makes RL so much better?

It turns out that RL doesn’t rewrite the entire brain of the model. It selectively nudges just the right parts. The study shows:

• RL changes fewer tokens — and mostly ones that actually matter for logic and structure.
• SFT floods outputs with irrelevant fluff like “Let’s check” or “Hmm, maybe…” even on tasks that don’t need reasoning.
• RL keeps the model’s internal representations stable, while SFT distorts them, especially on non-math tasks.

The result? RL-trained models become better thinkers without forgetting how to talk.

So if you’re training models for general use, you don’t want to rely solely on SFT, especially if you’re pushing reasoning skills. Math alone isn’t the magic. It’s how you train on math that matters.

Basically what you need to consider is the difference between training a model to repeat answers. and training it to explore, correct itself, and stay grounded. And that’s something I find exciting. We’re not just making models smarter in one narrow way — we’re learning how to help them grow without forgetting who they are.

A bit of technical dive

The researchers ran PCA on hidden states before and after training. The SFT models showed large Euclidean shifts in representation space — especially on non-reasoning tasks. The RL models? Far more stable. This suggests SFT warps the model’s internal geometry, hurting generalization.

Token Distribution Shift (KL Divergence & Rank Analysis)

KL divergence between base and fine-tuned models’ output distributions was much higher for SFT models — especially in non-math tasks like CoQA or HalluEval. Meanwhile, RL models maintained a closer alignment. That’s a big deal if you care about preserving a model’s “original voice” and skills.

Word clouds showing significantly shifted tokens between UniReason-Qwen3–14B-RL (left) and UniReason-Qwen3–14B-SFT-think (right) models on mathematical reasoning tasks

Even more telling: SFT caused hundreds of token rank shifts across entire sequences. RL changed just a handful, and mostly task-relevant ones. Think of it as the difference between surgical precision and rewriting paragraphs with a crayon.

Transferability Index (TI)

The authors introduced a metric called the Transferability Index to measure how gains in math translated to:

• Other reasoning tasks (like code generation or planning)
• Non-reasoning tasks (like instruction following)

Across the board, RL-tuned models had positive TI scores. SFT-tuned models often had negative scores, especially in non-reasoning domains. That’s a red flag if your model serves general users.

Controlled Experiment with Qwen3–14B

The paper’s standout experiment involved two versions of Qwen3–14B:

• One trained with SFT on 47k math problems (using CoT traces from a stronger model).
• One trained with RL on the same data (using correct answers as rewards).

The RL model not only matched or beat SFT in math but also significantly outperformed it on general tasks — despite being trained on the exact same data.

if you like to read the full paper, here is a PDF version of it: https://arxiv.org/pdf/2507.00432

if you like this post, it worth a world to me if you clap and share it with your friends and network and 🔔 Follow me on : Medium | Youtube | Linkedin | Github