DeepSeek Explained 6: All you need to know about Reinforcement Learning in LLM training

Image created by author using ChatGPT.

This is the sixth article in our DeepSeek series, where we will dive deeper into one of the key innovations in training strategies of DeepSeek models [1, 2]: Grouped Relative Policy Optimization (GRPO) [3].

To ensure this article is self-contained and also clarify the rationale behind GRPO, we will begin by covering fundamental concepts of Reinforcement Learning, highlighting the crucial role of RL and RLHF in LLM training. Following that, we will explore different paradigms including value-based, policy-based and Actor-Critic RL, review prior algorithms such as Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), and finally, explain the optimizations introduced by GRPO.

Table of contents for this article:

• Background: explain why LLM training needs RL, how Reinforcement Learning and Reinforcement Learning from Human Feedback (RLHF) works.
• Reinforcement Learning Paradigms: explain and compare value-based RL, policy-based RL and Actor-Critic RL paradigms.
• GRPO: revisit TRPO and PPO first, and then explain how GRPO further optimizes PPO.
• Summary.
• Reference.

In case you are interested in other articles in this DeepSeek series, here are the links:

• Part 1: Multi-head Latent Attention
• Part 2: DeepSeekMoE
• Part 3: Auxiliary-Loss-Free Load Balancing
• Part 4: Multi-Token Prediction
• Part 5: DeepSeek-V3-Base
• Part 7: Advancing LLM Reasoning with Reinforcement Learning
• Part 8: Post-Training of DeepSeek-V3

Background

Why RL is needed for LLM Training

Before RL is introduced to LLM training, NLP models are commonly trained with two stages, which are often referred to as the Pre-training then finetuning paradigm:

• Pre-Training: training on a large text corpus using unsupervised objectives such as predicting missing tokens. This stage helps the model to develop a broad understanding of language.
• Supervised Finetuning: training on human labeled datasets to specialize in specific tasks such as Question Answering. This helps the model to generate more useful and structured output for particular tasks.

However, even after training with both stages, LLM-generated responses are often misaligned with human preferences, for example hallucinating incorrect information, giving responses that are overly verbose or too concise, overlooking implicit context information, or misinterpreting sarcasm or humor, etc.

In other words, making an LLM truly helpful requires aligning it with human preferences, which is challenging to achieve with supervised finetuning alone.

So why is that?

The real challenge is translating alignment as a learnable target that can be properly labeled and used to construct a meaningful learning objective. Given that alignment is inherently vague and nuanced, it is impractical to exhaustively enumerate all possible misalignments and define specific labels for each case.

And that is exactly where Reinforcement Learning becomes useful.

How Reinforcement Learning Works

Machine learning algorithms can be broadly classified into three major categories:

• Supervised Learning: these algorithms learn from labeled data, where each input x is paired with a target y and the goal is to build a model that can predict y given x. The task is called a classification problem when y is discrete, and a regression problem otherwise.
• Unsupervised Learning: when labeled targets are not available, we can design algorithms to discover underlying patterns or structure within the input data. This category include dimension-reduction methods like Principal Component Analysis (PCA) and clustering methods such as K-Means.
• Reinforcement Learning (RL): in cases where defining explicit learning target is challenging, RL models can be learnt through interactions with a certain environment, collecting feedback or rewards for model update. This approach is commonly used for training agents, such as teaching a robot to balance and navigate a space.

The figure below shows the 5 elements in a Reinforcement Learning scenario.

Figure 1. Five elements in RL: Agent, Environment, Reward, State and Action. (Image from wiki.)

Now imagine you are teaching your puppy to sit, where you can find all the 5 elements:

• Agent: Your puppy learning this new command “sit”.
• Environment: Everything around your puppy.
• State: The situation your puppy is in (whether it is sitting or not).
• Reward: A treat that you give your puppy when it follows your command;
• Action: What your puppy could do, like sitting, jumping or barking.

So what’s the difference between a learning target in supervised learning vs. a reward in RL?

In supervised learning, each sample input should be paired with a label, and the goal is to train the model to minimize certain loss functions between its prediction with the target. However in RL, the agent interacts with the environment without explicit labels for each action, instead it receives a reward every now and then from the environment as the feedback to its actions.

Note that rewards is often delayed and can be very sparse, for example when playing chess you only get a win or loss at the end of the game. That means the agent may not know immediately if an action is good or bad. Instead it has to learn by trial and error over time, with the goal to maximize the cumulative reward over time.

This makes RL particularly useful in scenarios where clear labels are not available, but feedback can still serve as a cue for success. For example, when training a robot to walk, we cannot predefine the correct joint angles under each state. However, as the robot tries different actions, it may eventually succeed in taking a single step — an event that naturally serves as a reward, indicating that its previous actions were effective.

Back to our puppy training example, RL works like this:

• In the beginning your dog (agent) didn’t understand what “sit” means, but it will try different things like running, sitting or even barking (actions) in your house (environment).
• Every time it sits, it will get a treat (reward).
• Over time your puppy learns that sitting gets a treat and it appears like it finally understands “sit”.

In general, RL models are learnt following a trial-and-error approach, and the key is having a well-designed reward. This reward must be closely aligned with the goal; otherwise the model will not be able to learn the desired behaviors. Meanwhile, producing such a reward should be as easy and quick as possible, since if it is too slow or too complicated to calculate the reward, the RL process will also become extremely slow, making it less useful in practical tasks.

For example, in a game, every action the model takes will automatically get a score from the game environment, and this score is directly connected to the model’s performance in playing this game.

However, in many real-world applications, there is no ready-to-use reward like a score in a game. Instead researchers have to take great efforts in defining a proper reward function. Moreover, some desired behaviors are very difficult to translate into reward functions — for example, how could you define a reward function to guide the agent to answer questions more politely?

This leads to RLHF: Reinforcement Learning from Human Feedback.

Reinforcement Learning from Human Feedback (RLHF)

Again in the puppy training example, imagine your puppy finally learns to sit, but sometimes it also barks while sitting, or it will jump onto the couch first instead of sitting quietly on the floor.

What can you do in that case?

With RLHF, you don’t just give your puppy a treat every time it sits. Instead, you give treats by comparing its behaviors. For example, if the puppy sits quietly on the floor, it gets a bigger reward than if it sits while barking or after jumping onto the couch. This way, your puppy learns that sitting quietly on the floor is better, even though you didn’t explicitly explain what “quiet” means.

As we mentioned before, having an easy and fast reward is the key to RL, which makes it unrealistic to involve a human into the training loop to provide direct feedback. To overcome this issue, we can collect some human feedback first, and then use these feedback to learn a reward function to mimic human preferences when comparing two actions.

In summary, RLHF typically involves three stages:

• Collect human feedback: sampling model outputs, and ask human judges to compare which is better.
• Learn a reward model by mimicking human judge’s preferences.
• Train a better policy using the leant reward model in the RL process.

Here a policy refers to the agent’s strategy to choose actions based on the state of the environment. The figure below shows how RLHF is implemented in finetuning InstructGPT [6], which largely follows the three stages outlined above.

If you are interested in learning more about InstructGPT, here is the link to our previous article: Insights from Codex and InstructGPT.

Figure 2. An illustration of the RLHF steps in training InstructGPT/ChatGPT. (image from [6].)

As shown in the figure above, the RL process mainly occurs in step 3, where a policy is optimized using Proximal Policy Optimization (PPO).

So how does PPO actually work?

To understand that, in the next section we will first explain three key paradigms in RL: Value-based, Policy-based and Actor-Critic methods.

Reinforcement Learning Paradigms

Before we dive into all the mathematical details, let’s first explain value functions and policy in the context of RL.

In RL, value functions tell us the expected future return of taking action a in state s following a policy π:

where

• R_t is the reward at time step t.
• γ is called the discount factor indicating how much future rewards matter.

A few things to mention here:

• The reason we need to take future returns into account is that rewards in RL are often delayed, so we cannot decide if an action is good or bad based on its immediate return.
• Introducing a discount factor helps balance short-term rewards with long-term future return. It also stabilizes training and improves convergence, as the value function remains finite when 0≤γ<1.

The above definition for value function also involves policy π, which can be seen as the strategy the agent follows to decide what action to take under a given state, and is usually represented as a mapping from states to action probabilities.

The above definitions of value functions and policy suggest that RL agents can be trained by optimizing either the value functions or the policy. That leads to three different training paradigms: Value-based RL, Policy-based RL and Actor-Critic RL.

Value-based RL

Value-based RL approaches update the value function according to the Bellman Equation [8], which decomposes the value of a state into two parts: the immediate reward and the discounted value of the next state.

Taking Q-learning [7] as an example, where the value function can be updated by:

where

• alpha is a learning rate to combine the immediate and future value.
• Q(S_t, A_t) is the immediate value in current state.
• R_{t+1} is the reward observed if A_t is taken in state S_t.
• Q(S_{t+1}, a) is the value in the next state when taking action a, so that max over a gives the maximum reward can be obtained from S_{t+1}.

More specifically, the process looks like this:

• Initialization: we start with a random Q(S_t, A_t) value.
• Interaction with environment: at time step t, the agent selects an action A_t in state S_t, and then receives a reward R_{t+1} from the environment, transitioning to the next state S_{t+1}.
• Update value function using the above rule.
• Repeat this process until convergence.

However, the above updating process involves calculating an argmax operation, which is intractable in a continuous action space with an infinite number of possible actions. This is because computing argmax over all actions requires global optimization at every learning step, which is computationally expensive.

This is often complicated with the training instability issue when using Q-network such as in Deep Q-Networks (DQN), since Q-network is typically non-convex and sometimes a small update in Q(s, a) will lead to large changes in action selection.

For that reason, value-based RL is commonly used in scenarios with a discrete action space preferably with fewer possible actions, such as DQN in Atari or AlphaGo.

But what if we have to deal with a continuous or a large action space? That’s where policy-based RL can help.

Policy-based RL

As mentioned before, Policy refers the rule used by the agent to decide which actions to take and is represented as a mapping from states to action probabilities:

where policy π(a ∣ s) is often a differentiable function such as a neural network with parameters represented by θ.

Therefore, instead of searching over action space in value-based RL, policy-based RL searches over parameter space (θ) to maximize the expected reward.

More specifically, policy-based RL optimizes the policy network by conducting gradient ascend using a policy gradient:

The policy gradient is often estimated in the form below:

where R is the total return calculated as the sum of rewards.

With the introduction of policy gradients, policy-based RL eliminates the need to compute the argmax over the action space, making it more suitable for scenarios with large or continuous action spaces.

However, calculating policy gradients remains challenging. In many real-world RL tasks such as playing chess, the return depends on the cumulative rewards over an entire episode and can be highly noisy, leading to high variance in policy gradients and instability in training.

To address this issue, Actor-Critic RL is proposed to reduce variance and improve training stability by combining value-based and policy-based methods.

Actor-Critic RL

Actor-Critic RL is designed by combining merits from both value-based and policy-based RL, where:

• A policy network (called Actor) to select actions.
• And a value function (called Critic) to evaluate actions.

Since action selection is handled by the policy network, Actor-Critic RL approaches are also suitable for large or continuous action spaces. Additionally, by incorporating a Critic network, they can also help reduce variance in policy gradient estimates and improve training stability.

More specifically, the raw return R in the above policy gradient is replaced by the Advantage Function:

where

• Q(s, a) represents the expected return when taking action a in state s.
• V(s) serves as a baseline value function estimating the expected return of the state.

By introducing the baseline value function V(s), the advantage function can stabilize the learning process by normalizing rewards relative to state expectations, preventing large updates due to high-variance reward signals.

With that, the policy gradient can be rewritten as

Actor-Critic RL methods are widely used for scenarios involving large or continuous action spaces, such as RLHF training in LLM alignment and the development of advanced robotics and autonomous vehicles.

While its success in many applications, Actor-Critic RL still faces challenges related to training instability and hyper-parameter sensitivity, as it requires training both the actor and the critic network, maintaining a balanced update between them can be difficult.

In the next section, we will explain how DeepSeek’s GRPO addresses these challenges.

Grouped Relative Policy Optimization (GRPO)

To better understand GRPO, let’s briefly revisit two commonly used Actor-Critic methods that inspired it: Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO).

Trust Region Policy Optimization (TRPO)

As mentioned before, actor-critic RL approaches optimize the policy network by computing an estimator of the policy gradient with an advantage function, and the gradient estimator can be obtained by differentiating the objective function below:

One drawback of this method is that it often leads to excessively large policy updates, leading to instability in the RL process. To address this, TRPO [4] introduces a surrogate objective function, by adding a constraint to the size of the policy updates:

where π_θ and π_(θ_old) are the updated and reference policy networks before the update, respectively.

The above objective function explicitly differentiate between the old and the updated policy network, and use the KL divergence between these two policy network as a constraint, as KL divergence is commonly used for measuring difference between probability distributions.

Proximal Policy Optimization (PPO)

Building on the idea of stabilizing the policy gradients, PPO [5] further enhances TRPO by introducing a clipped surrogate objective.

More specifically, PPO first defines a probability ratio between the old and new policy network:

and then rewriting the objective function in TRPO as

where CPI stands for Conservative Policy Iteration.

On top of that, the clipped surrogate objective can be written as

where

• The first term in the min() operator is the same to TRPO, i.e., L^{CPI}.
• The second term in the min() operator clips the probability ratio with a hyper-parameter ϵ to the range [1 − ϵ, 1 +ϵ].

By applying the min() operator over the unclipped and clipped objectives, PPO can be considered as optimizing the lower bound of the original unclipped objective, leading to more conservative updates that helps stabilize the training process.

PPO is widely used in LLM training, such as InstructGPT [6]. However, it still inherits common challenges of actor-critic approaches as we mentioned before. Moreover, as LLMs continue to grow in size, maintaining a separate value network becomes increasingly costly, since the value network is typically of comparable size as the policy model.

In the next section, we will explain how GRPO eliminates the need for a separate value network.

Grouped Relative Policy Optimization (GRPO)

The core idea of GRPO is to remove the need for a separate value network while maintaining the stability in training, which also makes GRPO a purely policy-based RL method.

To make it easier to see the difference between PPO and GRPO, let’s rewrite the PPO objective function as below:

where q and o are questions and outputs sampled from the question dataset and the old policy network, respectively. This objective function organizes the average using queries, making it easier to extend to the grouped structure introduced in GRPO [3].

More specifically,

where the penalty term is formulated as the unbiased KL divergence:

and the advantage A_i is calculated as below:

where r_1 to r_G represent the rewards corresponding to the outputs within each group. Introducing such grouped structures is crucial in GRPO, as each query is different and calculating normalized advantage over the global rewards from different queries is less meaningful in guiding the model updates.

Here is the key difference between GRPO and PPO: by estimating advantage function using the relative rewards within each group, it completely removes the need for a separate value network, making it a purely policy-based RL approach rather than an Actor-Critic method.

As mentioned before, removing the value network significantly reduces training resources requirements, as the value network is typically of comparable size as the policy model. Additionally, it can enhance training stability by avoiding the complexity of optimizing two separate networks.

But why is this approach possible?

Recall that the advantage function measures how much better or worse an action is compared to the expected return from a given state. In PPO, it is estimated as the difference between the value function Q(s, a) and a baseline value function V(s).

That means that as long as we can define a metric that captures the same concept— how much better or worse the current action is relative to the expected return from that state — we can use it as an advantage function.

In GRPO, this metric is computed within each group (which can be interpreted as a query), as the group essentially represents the state the agent is in. Therefore, by comparing the reward for a specific action with the rewards of other sampled actions within the same group, GRPO naturally derives an estimate of the advantage function.

The figure below compares PPO and GRPO, note that there is no value model in the latter:

Figure 3. PPO vs. GRPO. Image from [3].

The following algorithm summarizes how GRPO is implemented:

Figure 4. GRPO algorithm. Image from [3].

Summary

This article provides a comprehensive introduction to reinforcement learning concepts essential for understanding the innovations behind DeepSeek’s Grouped Relative Policy Optimization (GRPO).

We begin with a high-level discussion on why RL is crucial for LLM training, highlighting its role in aligning model outputs with human preferences beyond supervised learning. We then explore how RL and RLHF work, explaining how reward models help align LLM behavior with human preferences.

To establish a solid foundation, we introduce the three major RL paradigms — value-based, policy-based, and actor-critic methods — outlining their respective strengths, limitations, and practical applications.

With that, we further examine two widely used actor-critic methods: Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). Finally, we introduce GRPO, the method proposed by DeepSeek, illustrating how it builds upon PPO by leveraging grouped structures and relative advantage estimation.

In the next article, we will take a deeper dive into DeepSeek’s training process, covering pretraining, fine-tuning, and RLHF alignment with GRPO, and how these stages work together to optimize model performance.

Reference

• [1] DeepSeek
• [2] DeepSeek-V3 Technical Report
• [3] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• [4] Trust Region Policy Optimization
• [5] Proximal Policy Optimization Algorithms
• [6] Training Language Models to Follow Instructions with Human Feedback
• [7] Q-learning
• [8] Bellman Equation