DeepSeek-R1: How Reinforcement Learning Taught a Model to Think Without Being Shown How

Hello! In this blog let me share what I learned from reading the DeepSeek-R1 paper, and try to make it click for you if you know transformers and basic ML but haven’t dug into RL-based training before.

The hook: what if you never showed the model how to reason?

Every major LLM you’ve used was taught to reason by imitation. You collect thousands of examples of humans thinking through problems step by step, train the model to reproduce that pattern, then optionally fine-tune further. The ceiling on this approach is obvious in retrospect: the model can never reason better than the humans who wrote the demonstrations.

DeepSeek-R1 asked a different question. What if we just… didn’t show it any reasoning examples? What if we gave it a reward for getting the right answer, and let it figure out how to think on its own?

That’s not a small bet. The conventional wisdom in 2024 was that you absolutely needed supervised fine-tuning (SFT) as a warm-up before any RL training. Without it, models produce incoherent garbage. The DeepSeek team ignored that and ran the experiment anyway.

What happened is the most interesting result I’ve read in a while.

Before we go further: what problem is this paper solving?

The standard post-training pipeline for LLMs looks like this:

Pre-trained base model
        ↓
Supervised Fine-Tuning (SFT)
  → show model thousands of human-written examples
  → model learns to imitate
        ↓
Reinforcement Learning from Human Feedback (RLHF)
  → train a reward model on human preferences
  → optimize the LLM against that reward model
        ↓
Deployed model

This works. InstructGPT, GPT-4, Claude — all trained roughly this way. But there are two quiet problems baked in:

Problem 1: You’re bounded by human demonstrations. If humans write “think step by step like this”, the model learns to think like that. It never discovers that a completely different reasoning strategy might work better.

Problem 2: Neural reward models get gamed. When you train a model to score outputs and then optimize an LLM against that scorer, the LLM eventually learns to fool the scorer rather than actually improve. The reward goes up. The real performance goes down. This is called reward hacking, and it’s a genuine headache at scale.

DeepSeek-R1 is an attempt to sidestep both problems at once.

The algorithm: GRPO in plain English

To understand how the training works, you need to understand the RL algorithm they use: Group Relative Policy Optimization (GRPO). It’s simpler than the alternatives, and that simplicity is the point.

The standard RL algorithm for LLMs is PPO (Proximal Policy Optimization). PPO needs two models running simultaneously: the policy model (the LLM you’re training) and a value model — a second network of roughly the same size whose only job is to predict “how good is this partial response going to turn out to be?” For a 671 billion parameter model, that means maintaining two 671B models at once. The memory cost is brutal.

GRPO’s insight: you don’t need a value model if you have a group.

1. For a single question q:
   Generate 16 different responses
   [o1, o2, o3, ... o16]

2. Score each response
   [1,  0,  1,  0, ...]   (correct = 1, wrong = 0)

3. Normalize by group mean and std
   A_i = (r_i - mean(rewards)) / std(rewards)

4. Push model toward above-average responses,
   away from below-average ones

The “value” of a response isn’t predicted by a neural network. It’s measured by comparing it to its siblings. The group is the baseline.

The full objective looks like this:

J_GRPO(θ) = 
  For each response i:
    min(ratio_i × A_i,  clip(ratio_i, 1-ε, 1+ε) × A_i)
    - β × KL(current_model || reference_model)

Where:
  ratio_i  = how much the model changed its opinion of response i
  A_i      = how much better response i was vs the group average
  clip(…)  = hard limit: don't change probabilities too much per step
  β × KL   = penalty for drifting too far from the original model

Three things to notice:

The ratio tracks whether the model is moving toward or away from each response. The advantage A_i is just a z-score — how many standard deviations above average was this response? The KL penalty is a leash keeping the model from going completely off the rails and forgetting everything it learned during pre-training.

The clipping is the “proximal” part of the name. Without it, a single update might change a probability distribution dramatically, destabilizing training. The clip says: move toward good responses, but not by more than ε in any one step.

One subtle reason GRPO beats PPO for this specific task: PPO adds the KL penalty as a per-token reward, which implicitly penalizes longer responses — every extra token adds KL cost. DeepSeek-R1 needs the model to generate long reasoning chains. PPO would actively fight against that. GRPO adds KL directly to the loss function, so there’s no length penalty. Small design choice, big practical difference.

DeepSeek-R1-Zero: the pure RL experiment

With GRPO in hand, they ran the experiment. Start from DeepSeek-V3-Base (a 671B parameter model). No SFT warmup. Just RL, with this reward:

Reward = accuracy_reward + format_reward accuracy_reward: did the final answer match ground truth? (0 or 1) format_reward: did the response use <think> and <answer> tags? (0 or 1)

That’s it. No neural reward model. No process reward (they don’t care how the model reasons, only whether it got the answer right). The training prompt was deliberately bare:

The assistant first thinks about the reasoning process in the mind and then provides the answer.

<think> reasoning process here </think> <answer> answer here </answer> User: [question] Assistant:

They trained this on math competition problems, coding challenges, STEM questions, and logic puzzles — domains where you can check correctness with a rule or a compiler. Then they watched what happened.

Performance on AIME 2024 (a hard high school math competition):

Step 0:     15.6% solve rate
Step 5,000: ~50% solve rate
Step 10,000: 77.9% solve rate

The model went from below human average to dramatically above it, without ever being shown a single example of how a human would solve these problems. But here’s what made this genuinely surprising. Response length grew in parallel:

Step 0:      ~2,000 tokens average
Step 10,000: ~15,000 tokens average

Nobody told it to write longer responses. Nobody rewarded it for thinking longer. The reward only checked the final answer. But the model discovered on its own that spending more tokens reasoning before answering led to better answers, which led to more reward. Length became instrumentally valuable.

And with length came emergent behaviors:

Self-verification appeared spontaneously. Early responses: “The answer is 42.” Late responses: “…so x = 42. Let me verify: substituting back, 3(42) + 7 = 133. ✓ The answer is 42.”

**Backtracking appeared spontaneously. **“Let me try substitution… [works through it] …this is getting complicated. Actually, let me try factoring instead…”

And then there was the “aha moment.”

At around step 8,000, the model started producing text like this:

“…squaring both sides gives x² = a + √(a+x)… Wait, wait. Wait. That’s an aha moment I can flag here. Let me reevaluate this step-by-step…”

The word “wait” as a self-interruption — a signal to itself that something felt wrong — appeared almost nowhere in early training. After step 8,000 it spiked dramatically. The paper calls this the model’s “aha moment.” It was also an aha moment for the researchers watching it happen.

Why step 8,000 specifically? Because at step 8,200 they doubled the maximum response length from 32K to 65K tokens. The model suddenly had room to think much longer, and it immediately started using that space to catch its own mistakes.

Why they didn’t use a neural reward model

You might wonder: why rule-based rewards? Wouldn’t a neural reward model give richer signal?

The paper is explicit about this. They tried it. Here’s what happened:

Neural reward model experiment: Steps 0–300: reward ↑, actual performance ↑ ← looks great Steps 300+: reward ↑, actual performance ↓ ← reward hacking

The model learned to produce outputs that scored well rather than outputs that were correct. The reward went up while real benchmark performance dropped. They show this curve directly in the paper (Figure 6).

Rule-based rewards don’t have this failure mode. Either the answer to a math problem matches the ground truth or it doesn’t. You can’t fake passing a compiler’s test suite. The ground truth is the reward signal, and it can’t be gamed.

The tradeoff: this only works for verifiable tasks.

Rule-based rewards work:     Rule-based rewards break:
─────────────────────────────────────────────────────
Math problems        ✓       Creative writing   ✗
Code that runs tests ✓       Summarization      ✗
Logic puzzles        ✓       Open-ended QA      ✗

This is why R1-Zero is only trained on math, code, STEM, and logic. For everything else, you’re forced back to neural reward models, which brings back reward hacking. It’s not a fully solved problem. The paper says so honestly.

R1-Zero’s problems and why they mattered

The pure RL result was scientifically impressive but had three concrete failure modes that made it unusable as a product:

Language mixing. The base model was trained on English and Chinese. R1-Zero would sometimes switch languages mid-reasoning chain — English sentence, Chinese phrase, back to English. Correct but disorienting.

Readability. The raw reasoning chains were chaotic. Like reading someone’s actual scratch paper versus their written-up solution. Correct but hard to follow.

Narrow capability. Trained exclusively on verifiable reasoning tasks, the model had degraded on writing, general conversation, and instruction following. It became a specialist at the cost of being a generalist.

These three problems are what motivated the full DeepSeek-R1 pipeline.

DeepSeek-R1: the four-stage pipeline

DeepSeek’s 4 Stage Pipeline

Each stage gives something and costs something. Let me walk through what each one actually does.

Stage 1: Cold Start SFT

Before running RL again, they collected a few thousand high-quality reasoning examples that look like what you’d want R1-Zero to produce — readable, first-person, conversational, consistently in one language, with a clean summary after the raw thinking.

They didn’t write these by hand. They generated reasoning traces using R1-Zero, kept only the ones with correct final answers, had human annotators rewrite the messy traces into natural style, then used those rewrites as examples to prompt DeepSeek-V3 to rewrite more traces automatically.

The interesting thing: after this SFT stage, AIME performance actually dropped. R1-Zero: 77.9% on AIME R1-Dev1: 59.0% on AIME ← went DOWN after SFT

This is the SFT tax. By constraining the model toward readable human-style output, it temporarily loses some raw reasoning capability. The team accepted this deliberately. The RL stage that follows will recover it — but now starting from a model that also produces clean output.

This tension between SFT and RL is something worth holding onto. SFT constrains the output distribution toward human preferences. RL expands it toward high-reward behaviors. They pull in opposite directions. The alternating pipeline is how you get both.

Stage 2: First RL Stage

Same GRPO setup as R1-Zero, with one addition: a language consistency reward.

Reward_language = num_words_in_target_language / total_words

If the question is in English and your reasoning chain is 95% English, you score 0.95. If you start mixing in other languages, it drops. This gets added directly to the total reward.

There’s an uncomfortable tradeoff here. Adding this reward causes a small but measurable drop in reasoning performance. You’re now rewarding a process property (language consistency) rather than purely outcomes.

Stage 3: Large Scale SFT

By this point the model reasons well and speaks consistently, but it’s still a specialist. This stage rebuilds general capability with 800K training samples:

Math: 395K samples (long CoT, single-turn) Code: 211K samples (long CoT, single-turn) General: 178K samples (short or no CoT, single-turn) STEM: 10K samples (long CoT, single-turn) Logic: 10K samples (long CoT, single-turn)

The key design decision: for general tasks, they sometimes add a CoT thinking step even when it’s not strictly necessary. A translation request might get a thinking section that considers formality and word choice. But for simple queries like “hello” — no CoT. The model needs to learn when thinking is worth the tokens, not just how to think.

One thing the paper quietly admits: almost all of this data is single-turn. One question, one response, done. This is why the paper lists multi-turn conversational ability as a known limitation. Multi-turn conversation requires tracking context, accumulating constraints, and handling contradictions across turns — skills that simply never appear in single-turn training data, regardless of how complex each individual turn is. Future work.

Stage 4: Final RL

For the first time, they introduce neural reward models — one for helpfulness, one for safety. They couldn’t avoid it here: there’s no rule-based ground truth for “is this response helpful?”

But they’re careful about where neural rewards apply. Reasoning data still uses rule-based rewards. Neural rewards only apply to general data. This minimizes the surface area for reward hacking.

And they time-limit the neural reward phase strictly. Look at the training curve from their experiments: after about 400 steps of neural reward training, the reward keeps going up but actual performance starts going down. So they only run the neural preference reward for the last 400 steps of a 1,700-step Stage 4. Small window, minimal hacking opportunity.

What I find most interesting about this paper

The self-reflection, backtracking, and “wait — let me reconsider” behaviors all emerged from a reward signal that only checked final answers. The model was never directly rewarded for any of these behaviors. It discovered them because they were instrumentally useful.

This tells you something about the relationship between process and outcomes in reasoning tasks. If you tried to directly reward “let me verify” as a token, you’d get a model that writes “let me verify” everywhere — including on trivial questions, as a ritual to collect bonus reward rather than as a genuine check. The behavior would look the same but mean nothing.

When a behavior emerges from outcome rewards, it emerges because it actually works. That’s a fundamentally different kind of learning than imitating the surface pattern of human reasoning.

The deeper limitation — and the paper is honest about this — is that the entire approach only works for tasks where you can objectively verify correctness. Math, code, logic: yes. Writing, nuanced conversation, creative tasks: the reward signal problem isn’t solved, and the paper says so directly. DeepSeek-R1 is a reasoning engine. Making an RL system that’s genuinely good at the unverifiable half of what makes a useful assistant is the next hard problem.

For me, the most valuable thing in this paper isn’t the architecture or the benchmarks. It’s the demonstration that a model left to its own devices, given only the incentive to be correct, can develop reasoning strategies that humans didn’t think to teach it. We don’t fully understand why those strategies emerge or whether they’re the best possible ones. But they work.

What’s next

In my next blog, I’ll implement a minimal version of this pipeline from scratch — a small GRPO training loop on a math dataset, with rule-based rewards, and we’ll see if we can observe even a small version of the emergent reasoning behaviors described here. No 671B parameter model required.

If you have feedback or if I got something wrong, feel free to reach out.