Post-training Large Language Models

Modern LLMs are built on the core task of next-token prediction. Given a sequence of text, the model computes the probability of the next word to complete the pattern. But how do we evolve from simple completion:

‘JYSK delivers a great Scandinavian offer for everyone…’

To a sophisticated, instruction-following assistant?

This article explores how language models are aligned with human preferences. We cover the key components of this process: beginning with instruction fine-tuning, which adapts large language models across a range of tasks, followed by reinforcement learning techniques that align model behavior with human values. We then examine DPO as a practical alternative, and GRPO as a novel approach in this space. Along the way, we discuss current limitations and draw concluding remarks.

Instruction Fine-tuning

While pretraining enhances NLP applications by serving as parameter initialization: it leverages large volumes of text to learn general linguistic patterns. Fine-tuning then adapts the pretrained model to specific tasks , such as sentiment classification, sequence tagging, rewriting, translation, question answering, summarization, and entity extraction, using human-validated labeled data.

A notable variant of this process is instruction fine-tuning, which collects (instruction, output) pairs spanning a diverse set of tasks, fine-tunes a language model on this curated data, and evaluates its generalization on unseen tasks.

Finetune language models on 1.8K tasks as instructions, and evaluates on unseen tasks [1].

An example dataset used in instruction fine-tunning is the Super-NaturalInstructions dataset contains over 1.6K tasks, 3M+ examples.

Super-NaturalInstructions dataset covers diverse range of task types [2].

Reinforcement learning from human preferences (RLHF)

Instruction fine-tuning, however, comes with notable limitations:

• Costly data collection: Gathering ground truth data for each task is expensive and time-consuming.
• No single right answer: Open-ended tasks like creative generation cannot be meaningfully evaluated against a fixed correct output.
• Uniform error penalization: Standard language modeling treats all token-level mistakes equally, even though some errors are far more consequential than others.

Even with instruction fine-tuning, a fundamental mismatch remains between the language model’s training objective and the broader goal of satisfying human preferences. To address this gap directly, we can turn to Reinforcement Learning, an approach that explicitly optimizes for human preference alignment.

Suppose we are training a language model on a given task, for example, summarization. For each sample s generated by the model, imagine we had a way to obtain a human reward score r(s) for that summary, where a higher score indicates better quality.

We now want to maximize the expected reward of samples drawn from our language model. For simplicity, consider the formulation for a single prompt, in practice, this is averaged over many prompts.

Let do a gradient ascent to change the LM parameters to maximize the expectation.

Two key challenges arise here: how to estimate the expectation, and how to optimize the objective when the reward function is not differentiable. Both are addressed by policy gradient methods in Reinforcement Learning (e.g., REINFORCE), which provide principled tools for estimating and optimizing this objective.

We aim to compute the gradient of the expectation. The first term of the above expression follows from the definition of expectation, while the second applies the linearity of the gradient operator.

Applying a useful identity known as the log-derivative trick, we take the gradient of the log-probability.

Plugging this expression back into the gradient of the expectation.

Now that the gradient is inside the expectation, we can approximate this objective using Monte Carlo sampling.

Substituting into the gradient ascent equation, we arrive at what is known as the reinforcement learning objective: good actions are reinforced by increasing the probability of their recurrence, while errors are penalized to reduce the likelihood of repeating them.

A key limitation of vanilla REINFORCE is its high variance in gradient estimates, which leads to unstable training and poor sample efficiency. This is addressed by Proximal Policy Optimization (PPO), which stabilizes learning by limiting how much the policy can change at each update through clipping the objective function.

rₜ(θ): it is the probability ratio between the pre-trained and post-trained policy.

A(sᵢ): the advantage function, measuring how much better an action performs relative to the expected baseline.

ϵ: The clipping range, controlling the maximum allowable deviation of the policy update (e.g., 0.2).

The resulting gradient ascent update is given by the following expression.

For any arbitrary, non-differentiable reward function R(s), we can train our language model to maximize the expected reward. However, two key challenges arise.

The first challenge is that human-in-the-loop feedback is costly. The solution is to model human preferences as a separate NLP problem (Knox and Stone, 2009), rather than directly querying humans at each step. Concretely, a reward model(RM) is trained to predict human preferences from an annotated dataset, and then optimize the reward model instead.

The second challenge is that human judgements tend to be noisy and miscalibrated. The solution is to solicit pairwise comparisons rather than direct ratings, as comparisons have been shown to be more reliable (Phelps et al., 2015; Clark et al., 2018). This naturally lends itself to the Bradley-Terry paired comparison model.

sʷ is the winning sample and sˡ is the losing sample. The wining should score higher than the losing sample.

The reward model must first be of sufficient quality, a large enough model trained on sufficient data can approach the performance of a single human annotator.

The reward model trained on all data begins approaching the accuracy of a single human [3].

Putting it all together, we now have all the necessary components:

• a pretrained model (possibly instruction-tuned),
• a reward model that produces scalar rewards for language model outputs, trained on a dataset of human comparisons,
• a method for optimizing language model parameters toward an arbitrary reward function.

We can now proceed with RLHF:

• Initialize a copy of the RL policy model with the parameters θ to be optimized.
• Optimize the following reward using reinforcement learning.

The fraction in the above equation acts as a penalty term that prevents the model from diverging too far from the pretrained model. In expectation, this quantity is known as the Kullback-Leibler (KL) divergence between the RL policy model and the pretrained model.

Illustrates the three steps of RLHF method: supervised fine-tuning (SFT), reward model (RM) training, and reinforcement learning via proximal policy optimization (PPO) on this reward model [4].

Experimental results show that RLHF yields consistent gains over pretraining and fine-tuning alone.

Fraction of the time humans prefer RLHF models’ summaries over the human-generated reference summaries on the TL;DR dataset [3]

Some limitations of RLHF are the following:

• Reward hacking: a common problem in RLHF, where the model exploits the reward function in unintended ways. Consider a cleaning robot that receives a reward for placing trash in a bin — reward hacking occurs if the robot learns to repeatedly pick up and re-deposit the same piece of trash to accumulate reward, regardless of whether the room is actually clean.
• Sycophancy: chatbots are incentivized to produce responses that appear authoritative and helpful, regardless of their truthfulness.
• Hallucinations: this can lead to models fabricating facts, as human preference models may reward confident-sounding responses even when they are factually incorrect, making the preference model itself an unreliable signal.
• Computational complexity: RL optimization can be computationally expensive and difficult to tune, particularly when it requires a value function and online sampling, both of which are slow. Furthermore, performance can be highly sensitive to hyperparameter choices. These challenges have motivated simpler alternatives to the full RLHF pipeline, such as Direct Preference Optimization (DPO), which will be covered in the next section.

Direct Preference Optimization (DPO)

Aligning language models with human preferences has become a central challenge, while Reinforcement Learning from Human Feedback (RLHF) achieves strong results, its complexity, requiring simultaneous management of multiple models and costly online sampling, motivates simpler alternatives (see below figure). Direct Preference Optimization (DPO) addresses this by reformulating alignment as a classification problem, eliminating the need for an explicit reward model.

Complex PPO workflow, depicting the sequential steps in the algorithm’s execution. The process begins with sampling from the environment, followed by the application of GAE for improved advantage approximation. The diagram then illustrates the computation of various loss functions employed in PPO, signifying the iterative nature of the learning process and the policy updates derived from these losses [5].

Direct Preference Optimization (DPO), is a streamlined method for aligning language models with human preferences without the need for separate reward models or complex reinforcement learning. While DPO targets the same goals as traditional RLHF maximizing rewards while staying true to the original model, it simplifies the process into a straightforward training task.

DPO optimizes for human preferences while avoiding reinforcement learning [6].

The core of DPO is a mathematical change of variables that allows the model to learn directly from a dataset of preferred and dispreferred answers using a simple binary cross entropy loss. Effectively, it increases the likelihood of good responses while decreasing the likelihood of bad ones, using a built-in weighting system to keep the model stable. By bypassing the multi-stage RLHF pipeline, DPO achieves optimal results through a much simpler and efficient training objective. Indeed, by expressing human preference probability in terms of the optimal policy rather than a separate reward model, it can derive a maximum likelihood objective for a parameterized policy. Let deep dive the DPO mathematical derivation.

Recall that we aim to maximize the following objective in RLHF.

This objective admits a closed-form solution.

Rearranging via a log transformation yields.

This holds for any arbitrary language model, giving rise to the following derived reward model.

The final DPO loss, derived via the Bradley-Terry model of human preferences, is given by:

Currently, most open-source RLHF implementations have moved away from traditional RL, with DPO emerging as the dominant approach. Open-source LLMs now adopted DPO, and it performs well in practice. However, DPO has notable limitations: it is restricted to offline training, and standard DPO struggles to capture the pluralistic nature of human preferences; instead of reflecting the diverse distribution of opinions within a group, it tends to gravitate toward dominant viewpoints and ignore conflicting perspectives. To overcome this bias, there is a novel framework called Group Distribution Preference Optimization (GDPO) which was designed to align language models with the full spectrum of collective preferences by integrating the underlying beliefs that drive individual choices.

Demonstration of PPO and our GRPO [7].

Conclusions

This article covers the full post-training pipeline, starting from instruction fine-tuning across a large number of tasks, followed by RLHF using Proximal Policy Optimization, analyzing its limitations and motivating the need for simpler alternatives. DPO is then introduced as a more practical approach, and finally, GRPO is presented as a novel variant of RLHF that simplifies PPO by eliminating the value model.

RLHF remains a rapidly evolving and largely underexplored area. While it advances beyond instruction fine-tuning, it is still data-intensive, and some fundamental limitations of large language models, such as scale and hallucination, may not be fully addressable through RLHF alone.

Disclaimer: The content of this article draws upon my professional experience and concepts from Stanford University’s CS224N: Natural Language Processing with Deep Learning course.

Bibliography

[1] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. (2022). Scaling Instruction-Finetuned Language Models.

[2] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi. (2022). Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks.

[3] Nisan Stiennon, Long Ouyang, Jeff Wu. (2022). Learning to summarize from human feedback.

[4] Long Ouyang, Jeff Wu, Xu Jiang. (2022). Training language models to follow instructions with human feedback.

[5] Rui Zheng, Shihan Dou, Songyang Gao. (2023). Secrets of RLHF in Large Language Models Part I: PPO.

[6] Rafael Rafailov, Archit Sharma, Eric Mitchell. (2024). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

[7] Zhihong Shao, Peiyi Wang, Qihao Zhu. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.