Reinforcement Learning for Large Reasoning Models: A Complete Technical Deep-Dive

Based on: “A Survey of Reinforcement Learning for Large Reasoning Models”

I’ve been closely tracking the shift happening in AI post-training from the alignment-focused RLHF pipelines that gave us ChatGPT, to the new wave of reasoning-first systems like DeepSeek-R1, o1, and Qwen3 that are redefining what language models are actually capable of. When this 120-page survey dropped from 40+ researchers at Tsinghua University and Shanghai AI Laboratory, I spent considerable time going through every section the math, the algorithm comparisons, the open debates, the infrastructure details and decided it deserved a proper technical breakdown rather than a surface-level summary.

In this article, I’ve dissected the entire paper into something you can actually use as a practitioner. You’ll find the full MDP formulation of how LLMs are framed as RL agents, a deep technical breakdown of all four reward paradigms (verifiable, generative, dense, and unsupervised), working Python implementations of GRPO and related algorithms, a structured comparison of 20+ policy optimization algorithms, an honest look at the five biggest unsettled debates in the field (including whether RL actually “discovers” new reasoning or just “sharpens” what the model already knows), a comparison of production RL infrastructure frameworks, and coverage of applications across code, agentic tasks, multimodal reasoning, and medicine closing with nine frontier research directions.

This is not a high-level overview. If you’re a data scientist, ML engineer, or AI researcher who wants to actually understand how these systems are built and where the field is heading this is written for you.

Table of Contents

1. Why RL Now?
2. LLMs as RL Agents
3. The Timeline of Reasoning
4. Foundational Component #1: Reward Design
5. Foundational Component #2: Policy Optimization
6. Foundational Component #3: Sampling Strategies
7. The Five Foundational Debates
8. Data, Environments & Infrastructure
9. Applications Across Domains
10. Future Directions
11. Practical Takeaways for Data Scientists & Researchers

1. Why RL Now?

If you’ve been following AI developments in 2024–2025, you’ve noticed a seismic shift: models don’t just answer questions anymore, they think through them. OpenAI’s o1, DeepSeek-R1, Claude 3.7 Sonnet, Gemini 2.5, all of these represent a new class of system where long chains of reasoning emerge from a training paradigm rooted in Reinforcement Learning.

This is not the RLHF (Reinforcement Learning from Human Feedback) of ChatGPT fame. This is something new: RLVR (Reinforcement Learning with Verifiable Rewards). The idea is deceptively simple: give the model a hard problem, let it generate many candidate solutions, and reward the correct ones. Do this at massive scale, and something remarkable happen, the model learns to plan, reflect, self-correct, and discover novel reasoning paths.

This survey by 40+ researchers from Tsinghua University, Shanghai AI Laboratory, and other leading institutions is the most comprehensive technical synthesis of this rapidly evolving field. Let’s dissect it properly.

The Evolution: RLHF → DPO → RLVR → Open-Ended RL

2022 ──── RLHF ────────────────────────────────────────────────→
          (GPT-3.5, GPT-4)
          Reward-based, human preference data
          Purpose: Alignment (helpful, honest, harmless)

2023 ──── DPO ─────────────────────────────────────────────────→
          (Llama 3, Qwen 2.5)
          Reward-free, offline preference optimization
          Purpose: Simpler alignment, no reward model needed

2025 ──── RLVR ────────────────────────────────────────────────→
          (o1, DeepSeek-R1)
          Rule-based verifiable rewards
          Purpose: Reasoning capability, complex task solving

FUTURE ── Open-Ended RL ──────────────────────────────────────→
          Self-generated rewards, dynamic environments
          Purpose: Artificial SuperIntelligence (ASI)

The key insight of this era: task-solving capacity, not just alignment, is the new frontier. And RL turns out to be the right tool for it because math problems and code are easy to verify but hard to solve the ideal conditions for RL optimization.

2. LLMs as RL Agents

To understand how RL is applied to LLMs, we need to map classical RL concepts onto the language domain. The paper formalizes this elegantly.

The MDP Formulation for LLMs

In classical RL, we have a Markov Decision Process (MDP) defined by a tuple (S, A, P, R, γ). Here's how it maps to language models:

The Objective

The learning objective is:

max J(θ) = E_{x ~ D, y ~ π_θ(x)} [G]
      θ

Maximize expected cumulative reward over the data distribution D, where G is the return (typically R(x, y) for sequence-level rewards, or Σ γᵗ rₜ for token-level rewards).

Action Granularity: A Critical Design Choice

The paper identifies three levels of action granularity, each with different reward structures:

┌─────────────────────────────────────────────────────────────┐
│           Action Granularity in LLM RL                      │
├──────────────┬────────────────────┬────────────────────────┤
│  Trajectory  │ Entire sequence y  │ R(x, y) — sparse       │
│  Token       │ Each token aₜ ∈ V  │ rₜ = R(x, a₁:ₜ)          │
│  Step        │ Segment y^(k)      │ rₖ = R(x, y^(1:k))     │
│  Turn        │ Agent response/turn│ rᵤ per tool call, etc. │
└──────────────┴────────────────────┴────────────────────────┘

This granularity choice fundamentally affects the credit assignment problem ‹how does the model know which part of a long reasoning chain was responsible for getting the right answer?

3. The Timeline of Reasoning

The paper tracks the explosive growth of RL-trained reasoning models. Here’s a curated timeline:

Language Reasoning Models

Jan 2025  ─── DeepSeek-R1 (671B, GRPO) ────────── First open-source to match o1
Mar 2025  ─── QwQ-32B (Alibaba) ──────────────── Matched R1 performance
Apr 2025  ─── Qwen3 (0.6-235B, GRPO) ────────── Further SOTA improvements
May 2025  ─── Llama-Nemotron-Ultra (253B) ────── Balance accuracy & efficiency
May 2025  ─── Skywork OR-1 (7/32B) ──────────── Effective data mixtures
Jun 2025  ─── Magistral 24B ─────────────────── RL from scratch, no distillation
Jul 2025  ─── Kimi K2 (1T MoE, OPMD) ────────── Agentic tasks focus
Aug 2025  ─── gpt-oss (117B/21B, OpenAI) ──────── First OpenAI open-source reasoning

Key Model Families Compared

The “DeepSeek-R1 Moment”

DeepSeek-R1 was a watershed moment because it demonstrated that:

1. Pure RL (Zero RL) can induce sophisticated reasoning behaviors without supervised fine-tuning warmup
2. Open-source models can match closed-source frontier performance
3. GRPO (a critic-free algorithm) is scalable for complex reasoning

The famous “Aha moment” where models spontaneously develop self-reflection and backtracking behaviors was first observed in DeepSeek-R1’s training logs.

4. Foundational Component #1: Reward Design

Reward design is arguably the most critical and most difficult part of building an RL-trained reasoning model. The paper categorizes reward types into five major classes:

4.1 Verifiable Rewards

The gold standard. When the task has an objectively checkable answer, use rule-based verifiable rewards.

Two types in practice:

# Type 1: Accuracy Rewards (Math)
def math_reward(model_output: str, ground_truth: str) -> float:
    # Extract answer from \boxed{...} delimiter
    predicted = extract_boxed(model_output)
    return 1.0 if symbolic_equivalent(predicted, ground_truth) else 0.0

# Type 2: Format Rewards (Structure enforcement)
def format_reward(model_output: str) -> float:
    has_think = "<think>" in model_output and "</think>" in model_output
    has_answer = "<answer>" in model_output and "</answer>" in model_output
    return 0.1 if (has_think and has_answer) else 0.0

# Type 3: Code Execution Rewards
def code_reward(model_code: str, test_cases: List[Tuple]) -> float:
    results = []
    for inputs, expected_output in test_cases:
        try:
            actual = execute_code(model_code, inputs)
            results.append(1.0 if actual == expected_output else 0.0)
        except Exception:
            results.append(0.0)
    return sum(results) / len(results)  # Pass rate

# Combined reward (DeepSeek-R1 style)
def total_reward(model_output: str, problem: Dict) -> float:
    r_accuracy = math_reward(model_output, problem["answer"])
    r_format = format_reward(model_output)
    return r_accuracy + 0.1 * r_format

Verifier’s Law: The ease of training AI systems to perform a task is proportional to the degree to which the task is verifiable. This explains why math and code are so dominant in current RLVR research.

4.2 Generative Rewards (GenRMs)

For tasks where answers aren’t objectively checkable (open-ended writing, complex reasoning with multiple valid paths), we use Generative Reward Models that reason before judging.

Three paradigms:

1. Reasoning Reward Models (Learn to Think)
   ─────────────────────────────────────────
   Input: [question, response_A, response_B]
   Process: Generate detailed CoT critique
   Output: Scalar preference score or binary choice

   Example: RM-R1, Think-RM, DeepSeek-GRM
   Training: Self-RL with meta-rewards (is my verdict correct?)

2. Rubric-Based Rewards (Structure Subjectivity)
   ─────────────────────────────────────────────
   Input: [question, response, rubric_criteria]
   Process: Check each criterion in the rubric
   Output: Multi-dimensional score vector

   Example: RaR (Rubrics as Rewards), Rubicon
   Use case: Creative writing, scientific reviews

3. Co-Evolving Systems (Dynamic Policy-Reward)
   ─────────────────────────────────────────────
   Self-Rewarding: Model generates its own training signals
   Co-Optimization: Policy + reward model trained jointly

   Example: RL Tango (joint GenRM + policy training)
             Cooper (co-optimization for reward hacking robustness)

4.3 Dense Rewards

Sparse outcome rewards (right/wrong at the end) suffer from the credit assignment problem: which part of a 2000-token reasoning chain was responsible for the error? Dense rewards address this by providing feedback at intermediate steps.

Three density levels:

Token-level:   rₜ = R(x, a₁:ₜ)         [Each token gets a signal]
               ↑ PRIME, Implicit PRM, SRPO

Step-level:    rₖ = R(x, y^(1:k))       [Each reasoning step scored]
               ↑ Math-Shepherd, VinePPO, TreeRL, SPO
Turn-level:    rᵤ per agent-environment  [Each tool call / interaction]
               ↑ ToolRL, SWEET-RL, SPA-RL

Practical comparison of PRMs (Process Reward Models) vs ORMs (Outcome Reward Models):

ORM (Outcome Reward Model)
+ Simple to implement, fast to compute
+ Works well for short-horizon tasks
- Sparse signal → slow learning for long CoT
- Enables "answer first, hallucinate reasoning later"
- Susceptible to reward hacking at scale

PRM (Process Reward Model)
+ Dense feedback → faster credit assignment
+ Reduces unfaithful chain-of-thought
+ Strong empirical results (Lightman et al., 2024)
- Extremely expensive to annotate step-by-step
- Quality degrades across domains
- Heuristic/Monte Carlo synthesis introduces bias

Monte Carlo PRM estimation (sampling-based approach):

def estimate_step_value(
    partial_reasoning: str,
    problem: str,
    model: LLM,
    num_rollouts: int = 16
) -> float:
    """
    Estimate P(correct | partial reasoning) via Monte Carlo sampling.
    Used in VinePPO, SPO, TreeRL.
    """
    completions = model.generate(
        prompt=problem + partial_reasoning,
        n=num_rollouts,
        temperature=0.8
    )
    correct_count = sum(
        verify_answer(completion, problem["answer"])
        for completion in completions
    )
    return correct_count / num_rollouts

4.4 Unsupervised Rewards

When even automated verification is unavailable, unsupervised rewards eliminate the human annotation bottleneck entirely.

Model-Specific Rewards (Internal signals)
├── Output Consistency
│   └── TTRL: Majority voting among multiple generations
│       Correct answers → dense consistent cluster
│       EMPO: Clustering-based correctness estimation
│
├── Internal Confidence  
│   └── EM-RL: Negative entropy as reward proxy
│       RENT: Cross-attention scores as confidence
│       Assumption: Model is confident ↔ likely correct
│
└── Self-Generated Knowledge
    └── Self-Rewarding (Yuan et al.): Model evaluates own output
        Absolute Zero (AZR): Model proposes problems + solves them
        Reward: Difficulty-calibrated (not too easy, not too hard)

Model-Agnostic Rewards (External automated signals)
├── Heuristic Rewards
│   └── Length bonus (more thinking = reward, initially)
│       Format compliance (DeepSeek-R1 style)
│       Warning: Gameable → superficial improvements
│
└── Data-Centric Rewards
    └── RPT: Next-token prediction reframed as RL
        SEAL: Model generates own training data + hyperparameters

4.5 Reward Shaping

Structure-based reward shaping with GRPO (the most widely used approach):

GRPO Advantage Computation:
─────────────────────────────────────────────────────────
1. Sample G responses {y₁, y₂, ..., y_G} from same prompt x
2. Compute rewards {R(x, y₁), ..., R(x, y_G)}
3. Compute group-relative advantage:

   Â_i = [R(x, yᵢ) - mean(R(x, y₁...y_G))] / std(R(x, y₁...y_G))

4. This normalizes rewards within a group, reducing variance
   vs. individual REINFORCE baseline

─────────────────────────────────────────────────────────
Example:
  Group rewards: [0, 0, 1, 0, 1, 1, 0, 0]  (8 responses)
  mean = 0.375, std = 0.484
  Advantages: [-0.77, -0.77, +1.29, -0.77, +1.29, +1.29, -0.77, -0.77]

  Effect: Correct responses get positive advantage (+1.29)
          Incorrect responses get negative advantage (-0.77)
          Policy is updated to increase P(correct | x)

5. Foundational Component #2 : Policy Optimization

5.1 The Policy Gradient Foundation

The general PPO-style objective for LLM RL:

J(θ) = E_data [ (1/Z) Σᵢ Σₜ min(wᵢ,ₜ(θ) · Âᵢ,ₜ, clip(wᵢ,ₜ(θ), 1-ε_low, 1+ε_high) · Âᵢ,ₜ) ]

where:
  wᵢ,ₜ(θ) = π_θ(yᵢ,ₜ | x, yᵢ,<ₜ) / π_θ_old(yᵢ,ₜ | x, yᵢ,<ₜ)   [importance ratio]
  Âᵢ,ₜ = advantage estimate for token t in response i
  Z = normalization factor (total tokens, group size, etc.)
  ε_low, ε_high = clipping bounds (asymmetric in DAPO)

5.2 Critic-Based vs Critic-Free: The Central Architectural Choice

┌─────────────────────────────────────────────────────────────────┐
│                CRITIC-BASED ALGORITHMS                          │
│                                                                 │
│  Architecture: Policy model + separate Value/Critic model       │
│  Token-level advantage via GAE:                                 │
│    δₜ = rₜ + γV(yₜ₊₁) - V(yₜ)                                  │
│    Â_GAE,t = Σ_{l=t}^{T} (γλ)^l δₜ₊ₗ                          │
│                                                                 │
│  Algorithms: PPO (Schulman et al., 2017), VCPPO, VAPO, PRIME    │
│                                                                 │
│  ✓ Fine-grained token-level credit assignment                   │
│  ✓ Well-studied, stable for alignment tasks                     │
│  ✗ ~2x compute (running + updating critic alongside LLM)        │
│  ✗ Critic can overfit → reward hacking                          │
│  ✗ GAE's γ-decay scales poorly for long CoT chains             │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│               CRITIC-FREE ALGORITHMS                            │
│                                                                 │
│  Architecture: Policy model only                                │
│  Advantage: Sequence-level reward (all tokens share same Â)     │
│                                                                 │
│  Algorithms: REINFORCE, RLOO, GRPO, DAPO, REINFORCE++           │
│                                                                 │
│  ✓ No separate critic → 50% memory saving                       │
│  ✓ No reward hacking from critic                                │
│  ✓ Scales better for complex, long-horizon tasks                │
│  ✓ Works well with rule-based rewards (no learned critic needed)│
│  ✗ Higher variance (sequence-level → noisy gradients)           │
│  ✗ Coarser credit assignment                                    │
└─────────────────────────────────────────────────────────────────┘

5.3 Algorithm Deep-Dive: GRPO vs PPO vs DAPO

GRPO (Group Relative Policy Optimization) the dominant algorithm in 2025:

def grpo_loss(
    policy: LLM,
    ref_policy: LLM,
    prompt: str,
    group_size: int = 8,
    epsilon: float = 0.2,
    beta: float = 0.01  # KL regularization (often set to 0 in recent work)
) -> torch.Tensor:
    """
    GRPO: Critic-free PPO with group-relative advantage.
    Used in DeepSeek-R1, Qwen3, Phi-4 Reasoning, etc.
    """
    # Step 1: Sample G responses
    with torch.no_grad():
        responses = policy.generate(prompt, n=group_size)
        old_logprobs = [policy.log_prob(r) for r in responses]
        rewards = [verifiable_reward(r) for r in responses]

    # Step 2: Compute group-relative advantages
    reward_tensor = torch.tensor(rewards)
    advantages = (reward_tensor - reward_tensor.mean()) / (reward_tensor.std() + 1e-8)

    # Step 3: PPO-style clipped objective
    total_loss = 0.0
    for i, (response, advantage, old_logp) in enumerate(zip(responses, advantages, old_logprobs)):
        current_logp = policy.log_prob(response)

        # Importance sampling ratio (token-level)
        ratio = torch.exp(current_logp - old_logp)

        # Clipped surrogate loss
        clipped_ratio = torch.clamp(ratio, 1 - epsilon, 1 + epsilon)
        loss_i = -torch.min(ratio * advantage, clipped_ratio * advantage).mean()

        # Optional KL regularization
        if beta > 0:
            ref_logp = ref_policy.log_prob(response)
            kl = (current_logp - ref_logp).mean()
            loss_i += beta * kl

        total_loss += loss_i

    return total_loss / group_size

Key variants and innovations:

Algorithm     | Key Change from GRPO              | Effect
──────────────┼───────────────────────────────────┼──────────────────────
REINFORCE++   | + PPO-style clipping, global norm  | More stable gradients
DAPO          | Asymmetric clipping (ε_low≠ε_high) | Better exploration
              | Dynamic sampling (filter all-0/1)  | Informative mini-batches
Dr.GRPO       | Fixes token-level length bias      | Fairer per-token credit
GSPO          | Sequence-level clipping            | Better for MoE stability
CISPO         | Clipped IS-weight (not ratio)      | Used in MiniMax-M1
LitePPO       | Group-level mean, batch-level std  | Variance reduction

5.4 Off-Policy Optimization

A critical practical consideration: the model generating rollouts (inference) is often slightly different from the model being trained (training), due to:

• Precision mismatch: train in fp32, inference in fp8/int4
• Asynchronous replay: old trajectories reused from buffer
• SFT+RL mixing: combining offline demonstrations with online RL

Off-Policy Learning Objective:

L_policy(θ) = -E_{x~D, y~π_b} [ π_θ(y|x)/π_b(y|x) · r(x,y) ]
where π_b is the behavior policy (data collector)
and   π_θ is the target policy (being optimized)
Importance weight π_θ/π_b corrects for distribution shift.

Mix-Policy Methods (SFT + RL hybrid) increasingly standard:

Strategy 1: Loss-Level Mixing
  L_total = L_RL + α · L_SFT
  Methods: LUFFY, SRFT, UFT, ReLIFT

Strategy 2: Data-Level Mixing  
  Expert data as prefix anchors → branch rollouts from expert prefix
  Methods: BREAD (branched rollouts from expert anchors)
           Prefix-RFT (blend supervised + RL training)

5.5 Regularization Objectives

KL Regularization controversial but important:

L_KL = β · Σₜ KL(π_θ(·|yₜ) || π_ref/old(·|yₜ))

Debate in the field:
─────────────────────────────────────────────────────────────
PRO (keep KL):
  - Prevents catastrophic forgetting
  - Stabilizes training, especially in RLHF alignment
  - Maintains language quality / readability
AGAINST (remove KL):
  - Policy needs to explore freely for new CoT structures
  - KL penalty is an "unnecessary restriction" on discovery
  - Many SOTA papers (DAPO, PRIME, AceReason) remove it
  - Reduces memory cost and implementation complexity
Current consensus: Remove KL for pure reasoning RL (RLVR),
                   Keep KL for alignment tasks (RLHF)

Entropy Regularization preventing “entropy collapse”:

L_ent = -α · Σₜ H[π_θ(·|yₜ)]   (maximize entropy)

The problem: Entropy collapse
  ─────────────────────────────────────
  Without intervention, RL training causes the policy to
  become increasingly deterministic (entropy → 0), leading
  to mode collapse and failure to explore.

  Solutions:
  ● Clip-Higher (DAPO): ε_high > ε_low, so low-probability 
    tokens can increase probability more freely
  ● High-Entropy Training: Only train on top 20% entropy tokens
  ● Clip-Cov / KL-Cov: Regulate tokens with high covariance
    between output probability and advantage

Length Penalty controlling thinking budget:

def adaptive_length_reward(
    response: str,
    problem_difficulty: float,  # 0 to 1
    max_length: int = 8192
) -> float:
    """
    Adaptive length penalty: harder problems get more budget.
    Based on ALP (Adaptive Length Penalty) approach.
    """
    response_length = len(response.split())
    allowed_budget = max_length * (0.3 + 0.7 * problem_difficulty)

    if response_length <= allowed_budget:
        return 0.0  # No penalty within budget
    else:
        # Soft penalty for exceeding budget
        excess = response_length - allowed_budget
        return -0.001 * excess  # Gradually penalize over-thinking

6. Foundational Component #3 : Sampling Strategies

Sampling strategy is the first-class lever in modern RL fine-tuning it determines what the model gets to learn from.

6.1 Dynamic Sampling

The core problem: Uniform sampling wastes compute on problems that are either too easy (model always gets them right, zero-gradient) or too hard (model never gets them right, also zero-gradient).

Ideal training signal distribution:
─────────────────────────────────────────────────────────
                  ╔═══════════════════╗
                  ║   INFORMATIVE     ║
                  ║    GRADIENT       ║
 Loss → 0 region  ║   ZONE (~30-70%   ║  Loss → 0 region
 (all correct)    ║   pass rate)      ║  (all incorrect)
                  ╚═══════════════════╝

DAPO's Solution: Filter prompts with 0% or 100% pass rate
  → Keep only prompts with mixed outcomes (non-zero advantage)
  → Resample until each mini-batch has "live" gradients

K1.5's Solution: Prioritize failures
  → p(i) ∝ (1 - sᵢ)   where sᵢ = current success rate for problem i

Curriculum learning from easy to hard:

Stage 1: Short context (8K tokens)
  └── Force concise reasoning patterns
  └── Easy-to-medium difficulty problems
  └── Low temperature (0.7) for stable learning

Stage 2: Medium context (16K tokens)
  └── Introduce more complex multi-step problems
  └── Gradually increase difficulty

Stage 3: Long context (32K+ tokens)
  └── Competition-level problems
  └── Higher temperature (1.0-1.2) for exploration

Key finding (DeepScaleR, AceReason-Nemotron):
  Starting with short context is ESSENTIAL.
  It forces token-efficient reasoning before allowing verbosity.

6.2 Tree-Based Structured Sampling

Instead of linear chain-of-thought rollouts, tree-based methods generate branching reasoning trees:

Standard ChainRL:
  Problem → [token 1] → [token 2] → ... → [token T] → Answer
  Credit assignment: Same reward for all tokens (imprecise)

TreeRL / MCTS-based:
  Problem → Step 1
              ├── Branch A: Sub-step A1 → Sub-step A2 → Answer A (correct ✓)
              ├── Branch B: Sub-step B1 → Sub-step B2 → Answer B (wrong ✗)  
              └── Branch C: Sub-step C1 → Answer C (wrong ✗)

  Credit assignment: Node-level rewards via MC backpropagation
  P(correct | reach node) = proportion of correct completions

Benefits:
  ✓ Fine-grained process rewards without human annotation
  ✓ More sample-efficient (shared prefix computation)
  ✓ KV cache reuse across branches reduces GPU memory

Used in: TreeRL, TreeRPO, SPO, FR3E, TreePO

7. The Five Foundational Debates

The paper dedicates an entire section to five open, hotly-debated questions. These are genuinely unsettled scientific questions with real implications for practitioners.

Debate 1: Sharpening vs. Discovery What does RL actually do?

SHARPENING VIEW                    DISCOVERY VIEW
────────────────────               ─────────────────────
RL doesn't create new              RL can uncover genuinely
capabilities; it just              new reasoning patterns
reweights existing ones            not present in pretraining

Evidence for:                      Evidence for:
- Pass@K analysis shows            • ProRL: Extended RL can
  RL improves Pass@1 but             improve Pass@K too
  underperforms base model         • Yuan et al.: LLMs can
  at large K (Limit-of-RLVR)        learn new skills via
- "Spurious rewards" give            capability composition
  Qwen models gains                • Discovery happens with
- Entropy shaping explains           sufficient training time
  most improvements                  + regularization
- Even random rewards help!

Reconciliation: The debate may be asking different questions.
RL's mode-seeking (reverse KL) provides efficient convergence
to high-reward regions (Sharpening), while implicit reward
learning + sequential decisions enable composition of existing
capabilities into novel behaviors (Discovery) with sufficient
training.

Debate 2: RL vs SFT Generalization or Memorization?

Classic quote: "SFT memorizes, RL generalizes" (Chu et al., 2025)

Evidence:
┌─────────────────────────────────────────────────────────────┐
│ SFT on math → Negative transfer to non-math tasks           │
│ RL on math → Preserves or enhances non-math performance     │
│                                                             │
│ Mechanism (PCA + KL analysis):                              │
│   SFT: Representation drift (memorization)                  │
│   RL:  Preserves base-domain structure (generalization)     │
└─────────────────────────────────────────────────────────────┘
Nuance: RL is NOT a panacea
  • RL generalizes poorly under severe distributional shift
  • SFT can generalize when properly regularized
  • Best practice: Unified/alternating hybrid (SFT warmup → RL)
Pipeline that works:
  Cold-start SFT (stabilize format) → RLVR (reason deeply) → SFT (add new knowledge) → RL (amplify)

Debate 3: Model Prior Which base model to start from?

Base Model vs. Instruct Model:
  DeepSeek-R1 finding: Base models outperform instruct models as RL starting points
  Reason: Instruct models have "entrenched formatting and obedience priors"
          that interfere with reward shaping

R1-Zero: RL directly on base model → emergent reasoning (simpler, works!)
  R1:      Cold-start SFT → RL → more readable output
Model Family Differences (Critical finding!):
  Qwen family:  Highly "RL-friendly" - gains even under RANDOM rewards!
                ↳ Rich pretraining exposure to math/code CoT

  Llama/OLMo:  Often shows NO improvement under standard RLVR
               ↳ Solution: Mid-training with high-quality math/code data
                 (annealing phase with curated data + LR decay)
Practical Recipe for Llama → RL:
  1. Standard pretraining
  2. Mid-training annealing:
     - Reweight data: Increase math/code/CoT proportion
     - Linearly decay LR to zero
     - Inject high-quality reasoning traces
  3. Apply RLVR (now "RL-friendly")

Debate 4: Training Recipes, Tricks or Traps?

The field is riddled with conflicting findings about which training tricks actually work:

Widely used but contested techniques:
1. Clip-Higher (ε_low < ε_high):
   Claim: Better exploration for unlikely-but-useful tokens
   Used in: DAPO, AceReason, ProRL

2. KL regularization removal:
   Claim: More exploration freedom → better reasoning discovery
   Used in: Most 2025 GRPO papers
   Contested by: Works showing entropy collapse without KL

3. Reward normalization (GRPO-style):
   Xiong et al. finding: "Discarding incorrect samples" > complex normalization
   RAFT/Reinforce-Rej achieves comparable performance with simpler mechanism!

4. Staged context lengthening:
   Start: 8K context → 16K → 32K → 64K+
   Benefit: Forces concise reasoning early; prevents "overthinking" addiction
The problem: Inconsistent experimental settings across papers
→ Method A beats method B on dataset X but not dataset Y
→ Different base models, different data sizes, different hyperparams
→ Unified frameworks (ROLL) now trying to benchmark fairly
Researcher recommendation: Trust scalability curves, not single-point comparisons

Debate 5: Process vs. Outcome Rewards

OUTCOME REWARDS (ORM)          │  PROCESS REWARDS (PRM)
───────────────────────────────┼────────────────────────────────
Score only the final answer    │  Score each reasoning step
                               │
+ Simple, scalable             │  + Faithful reasoning
+ No annotation needed         │  + Prevents post-hoc rationalization
+ Works for math/code          │  + Lightman et al.: PRMs outperform ORMs
                               │
- Enables "answer first,       │  - Step annotations are expensive
  fake reasoning later"        │  - Quality degrades cross-domain
- Reward hacking at scale      │  - MC synthesis introduces bias
                               │
Examples: DeepSeek-R1 training │  Examples: Math-Shepherd, PRIME, PAV
          Pure RLVR pipelines  │           VinePPO, TreeRL

Best practice emerging:
  Combine both via implicit process modeling (PRIME)
  or generative verifiers (DeepSeek-GRM)›‹‹›

8. Data, Environments & Infrastructure

8.1 Key Datasets for RL Training

Math datasets:

Code datasets:

The key shift: From “scale-first” to “quality and verifiability-first”. 800 carefully curated examples (LIMO) can sometimes outperform 100K+ poorly filtered samples.

8.2 Dynamic Environments

Static datasets have a fundamental limitation: the model can memorize them. Dynamic environments provide an inexhaustible supply of novel problems:

Environment Types:
──────────────────────────────────────────────────────────────
STEM Simulators    │  Formal math provers (Lean, Coq)
                   │  Scientific simulation backends
                   │  Constraint satisfaction verifiers
                   │
Code Executors     │  Python/JS/C++ interpreters
                   │  Unit test runners (pass/fail)
                   │  Linters, type checkers
                   │
Game Environments  │  Puzzle games (logic, spatial)
                   │  Strategy games (PettingZoo)
                   │  TextArena (language-based games)
                   │
Agent Environments │  Web browsers (Playwright, WebArena)
                   │  GUI simulators (Android, desktop)
                   │  Tool-use APIs (calculators, search)

Key framework: InternBootcamp
  ├── 1000+ general reasoning tasks
  ├── 8 domains with difficulty-controllable generators
  ├── Rule-based verifiers for all tasks
  └── Demonstrated "Task Scaling": more tasks → better reasoning

8.3 RL Infrastructure

Running RL at scale for LLMs requires specialized infrastructure. Here’s a comparison of major open-source frameworks:

Framework   │ Algorithms              │ Async  │ Multi-Agent │ Scale
────────────┼─────────────────────────┼────────┼─────────────┼──────────
TRL         │ PPO, GRPO, DPO, SFT     │ ✓(vLLM)│ ✗           │ Multi-GPU
OpenRLHF    │ PPO, GRPO, RLOO, DPO    │ ✓       │ Partial     │ Multi-node
veRL        │ PPO, GRPO, GSPO, PRIME  │ ✓       │ ✓           │ 1000s GPU
AReaL       │ PPO (async-first)       │ ✓ (2.77x)│ Partial    │ 512 GPU
NeMo-RL     │ PPO, GRPO, SFT          │ Partial │ ✗           │ 100B scale
ROLL        │ GRPO, PPO, GSPO, TOPR   │ ✓       │ ✓           │ Megatron
slime       │ SGLang-native, GRPO     │ ✓       │ Partial     │ Dense+MoE

Key architectural principle all modern frameworks separate:

Generation Workers (Rollout)    Training Workers (Update)
──────────────────────────────  ─────────────────────────
vLLM / SGLang inference engine  DeepSpeed ZeRO / Megatron
GPU-optimized sampling          GPU-optimized gradient
Async (can run while training)  Receives batches from buffer
Policy model (inference mode)   Policy model (training mode)
                ↕ Ray communication layer ↕

9. Applications Across Domains

9.1 Coding

RL has revolutionized code generation. The progression:

Stage 1: Competitive Programming (2025 early)
  DeepCoder, AReaL, SkyWork OR-1, AceReason-Nemotron
  ↳ Unit test rewards, pass rate as signal
  ↳ Key finding: Math RL → positive transfer to code!

Stage 2: Software Engineering (2025 mid)
  SWE-RL: GRPO on GitHub patch generation loop
  Satori-SWE: Agent autonomously improves patch quality
  ↳ Real repository context, multi-file editing
  ↳ SWE-bench as evaluation (Claude Opus 4.1 SOTA)

Stage 3: Agentic Coding (2025 onwards)
  Tool-Integrated Reasoning (TIR):
    <code> → execute → result → refine → answer
    ↳ AutoTIR, CoRT, ToRL, ARPO

  Multi-turn code agent:
    Problem → Debug 1 → Execute → Error → Debug 2 → Pass ✓
    ↳ Closed-loop optimization via RL

9.2 Agentic Tasks

RL enables LLMs to use tools and navigate complex multi-step environments:

Search Agents:

Search-R1, R1-Searcher, ZeroSearch:
  Input: Complex question
  Action space: [query_search, read_document, answer, refine_query]
  Reward: Answer correctness + search efficiency

  Challenge: Online search = expensive API calls
  Solution (ZeroSearch): Simulate search engine during training
  → Sim2Real generalization at deployment

GUI/Computer-Use Agents:

UI-R1 → GUI-R1 → ZeroGUI progression:

  Action: Click(x, y) | Type(text) | Scroll | Hotkey

  Reward design evolution:
  Stage 1: Binary task completion (did it work?)
  Stage 2: Step-level rewards (action accuracy, argument correctness)
  Stage 3: Fully automated via environment feedback (ZeroGUI)

  Current best: UI-TARS 2 with end-to-end RL across GUI + Code + Tools

9.3 Multimodal Reasoning

Visual Understanding (RL extensions):
  Vision-R1, VLM-R1: "DeepSeek-R1 for vision" — thinking before seeing
  Deepeyes, CoF: Multimodal CoT (text + region-of-interest)
  Video-R1: T-GRPO for temporal reasoning

  Challenge 1: "Inconsistent reasoning": CoT doesn't match final answer
  Challenge 2: Long-chain exploration collapse → hallucinations
  Challenge 3: Data quality sensitivity (more severe than text-only RL)

Multimodal Generation:
  DanceGRPO, Flow-GRPO, T2I-R1: GRPO for diffusion models

  Technical challenge:
    GRPO assumes stochastic sampling for advantage estimation
    Diffusion ODE sampling is deterministic → no diversity!

    Solution: ODE-to-SDE conversion to re-introduce stochasticity
    Or: MixGRPO (mixed SDE + ODE sampling)

9.4 Medical Applications

Medical Understanding:
  Med-RLVR, MedVLM-R1, Open-Medical-R1
  Rewards: Clinical accuracy on verified QA, diagnostic correctness
  Challenge: No "ground truth" for clinical decision making

Medical Agents:
  MedResearcher-R1: Deep research for clinical questions
  MMedAgent-RL: Multi-modal medical agent with tool use

Key datasets:
  MedAgentGym: Executable coding environment for medical reasoning
               → Generate trajectories for LLM-based medical agents

Notable achievement: RL-trained models surpassing specialist
human physicians on specific diagnostic benchmarks

10. Future Directions

The paper identifies nine frontier research directions:

1. Continual RL for LLMs

Current approach: Mix data from all tasks, train jointly. But as AI systems must adapt to evolving tasks in dynamic environments, we need lifelong learning without catastrophic forgetting. Traditional CRL techniques (experience replay, policy reuse, reward shaping) need LLM-specific adaptations.

2. Memory-Based RL

Transform agent memory from task-specific buffers → experience repositories that are structured, reusable, and transferable across tasks. Key challenge: Teaching agents via RL how to manage and compose memories.

3. Model-Based RL

Build world models that predict environment dynamics and generate rewards, reducing the need for expensive real environment interactions. LLMs as world models is promising but deeply unexplored.

4. Efficient Reasoning (Overthinking Problem)

Current problem:
  Model trained to "think longer = think better"
  → Applies 2000-token reasoning chains to simple arithmetic
  → Massive inference waste

Goal: Resource-rational reasoning
  Easy question: [20 tokens] → Answer
  Hard question: [2000 tokens] → Answer

  Current approaches:
  ● Adaptive length-based reward shaping
  ● Length penalty in loss function
  ● Hard-coded reasoning budgets in prompts

  Open problem: Principled cost-performance trade-off

5. Latent Space Reasoning

Current CoT: Token-level discrete space
  "Let me think... 2+2=4... therefore the answer is..."

Latent Space Reasoning (LSR): Continuous latent space
  Hidden state tensor → Transforms → Hidden state tensor → Answer

  Advantage: No information bottleneck from discretization
             Smoother learning dynamics
             More naturally integrates with RL

  Challenge: How to verify "latent thoughts"?
             How to compute reward for continuous reasoning?

6. RL for Pre-Training

Traditional: Pretraining (next-token prediction) → Posttraining (RL)
             ↑ Treats pretraining and RL as completely separate

New direction: Reinforcement Pre-Training (RPT, Dong et al.)
  Reframe next-token prediction as RL with verifiable rewards
  from the corpus itself

  Claim: Consistent gains that scale with compute
         RL becomes a viable scaling strategy for pretraining!

  Challenge: How to verify quality at pretraining scale?

7. RL for Diffusion LLMs

Applying RL to masked diffusion language models (MDLM/DLLM) rather than autoregressive models a technically challenging frontier due to ELBO estimation complexity.

8. RL for Scientific Discovery

Moving beyond benchmark-driven improvements to genuine scientific contributions in biology, chemistry, and materials science requiring lab-in-the-loop verification and in silico environment simulation.

9. Architecture-Algorithm Co-Design

Current paradigm: Fixed architecture → RL tunes weights
Future paradigm:  Architecture IS the action space
Reinforced MoE:
  ● Learn routing policies via RL
  ● Reward function: Task accuracy + hardware efficiency
    (latency, memory, energy consumption)
  ● Result: Model dynamically adapts topology per prompt

  This goes beyond classical NAS (Neural Architecture Search):
  Instead of finding one fixed architecture, learn to
  ADAPT architecture at inference time per input difficulty

11. Practical Takeaways for Data Scientists & Researchers

If you want to run your own RLVR experiment today:

# Minimal GRPO setup using TRL or veRL
# This is the "DAPO-lite" recipe that works reliably

config = {
    # Model
    "base_model": "Qwen2.5-7B",  # Qwen is most RL-friendly
    "use_base_not_instruct": True,  # Base model > Instruct for RL

    # Algorithm
    "algorithm": "GRPO",
    "group_size": 8,          # G responses per prompt
    "epsilon_low": 0.2,       # Standard clipping lower bound
    "epsilon_high": 0.28,     # Clip-Higher: ε_high > ε_low
    "kl_beta": 0.0,           # Remove KL for pure reasoning

    # Sampling
    "temperature": 1.0,
    "dynamic_sampling": True,  # Filter all-0 and all-1 groups
    "max_length_initial": 8192,  # Start short, increase later

    # Reward
    "reward_type": "rule_based",  # Math accuracy + format
    "format_reward": 0.1,

    # Training
    "learning_rate": 1e-6,
    "batch_size": 32,
    "gradient_accumulation": 4,
}

Key decisions and their tradeoffs:

Red flags to watch out for:

🚨 All rewards = 0 in a batch
   → Problem too hard, no correct solutions
   → Fix: Easier curriculum, better data filtering

🚨 All rewards = 1 in a batch  
   → Problem too easy, zero gradient
   → Fix: Dynamic sampling (DAPO), harder problems

🚨 Entropy collapse (policy becomes deterministic)
   → Fix: Clip-Higher, entropy bonus, high temperature

🚨 Response length keeps growing to max token limit
   → "Overthinking" without quality gain
   → Fix: Length penalty, staged context length

🚨 Reward hacking (high reward, wrong behavior)
   → Model exploits verifier weaknesses
   → Fix: Multiple complementary verifiers, GenRM sanity check

The “Verifier’s Law” Checklist

Before investing in RLVR for a task, ask:

• Is there an objectively correct answer (not just “better/worse”)?
• Can you automatically verify answers at scale (no human in loop)?
• Can you generate many candidates for the same problem efficiently?
• Is the reward signal closely aligned with true correctness?
• Are there enough hard problems that the model can’t memorize?

If you check all five: RLVR will work well for your task. If you miss any: consider GenRMs, rubric-based rewards, or hybrid approaches.

Conclusion

This survey captures a pivotal moment in AI development. Reinforcement Learning for Large Reasoning Models is not just an academic curiosity it’s the core technology behind the most capable AI systems deployed today, and likely the path toward systems of even greater capability.

The key insights that practitioners should carry forward:

1. RLVR fundamentally changes what LLMs can do, not just how well they do existing things
2. Reward design is the hardest part verifiable rewards are your best friend; treat non-verifiable rewards with caution
3. GRPO is the dominant algorithm for a reason: it’s simpler, cheaper, and works better for long-horizon reasoning than PPO
4. The Sharpening vs. Discovery debate is not settled don’t assume RL only reveals latent capabilities; extended training may genuinely discover new ones
5. Model family matters enormously Qwen is currently the most RL-friendly family; Llama needs careful mid-training
6. Infrastructure is a real bottleneck veRL, AReaL, and OpenRLHF are the foundations to build on
7. The frontier is wide open continual RL, latent space reasoning, and RL for pretraining are all genuinely unsolved

We are at the beginning of a new scaling axis for AI: not just parameters and data, but reasoning time and interaction depth. Reinforcement Learning is the engine that makes this possible.

This article is a technical synthesis of the survey paper “A Survey of Reinforcement Learning for Large Reasoning Models” (arXiv:2509.08827v3) by Kaiyan Zhang, Yuxin Zuo, et al. from Tsinghua University and Shanghai AI Laboratory, 2025. All algorithms, equations, and findings are attributed to the original authors. The GitHub repository with related resources is available at TsinghuaC3I/Awesome-RL-for-LRMs.

Prerequisites: Familiarity with transformers, basic RL concepts helpful