DPO and Modern AI Alignment: Fine-Tuning LLMs with Human Preferences

Large language models are not ready for real conversations right out of the box. Even if a base model knows a lot about language, code, history, science, and everyday reasoning, it might not always answer questions helpfully, refuse harmful requests, follow instructions well, or use the tone people expect from an assistant. Pretraining gives a model broad skills, but fine-tuning and alignment shape those skills into useful behavior. InstructGPT researchers showed that making a model bigger does not guarantee it will better follow users’ intentions. Large models can still give answers that are untruthful, toxic, unhelpful, or not what users expect.

This is why modern AI development puts so much focus on post-training. These are the steps after pre-training that teach a model how to act. Put simply, fine-tuning helps a model go from just generating text to giving useful, safe, and context-aware answers. Alignment takes this further by making sure the model not only completes tasks but also does so in a way that respects human preferences, safety, honesty, and the intended purpose.

Fine-Tuning: Teaching the Model the Job

The first major step is usually supervised fine-tuning, or SFT. In SFT, the model learns from examples of good behavior, such as strong answers to instructions, helpful replies in conversations, or expert-written responses to prompts. It is like giving a new employee a folder of great work samples and saying, “This is the kind of answer we want.”

Instruction tuning is one of the most important forms of SFT. Instead of leaving the model to imitate general internet text, instruction tuning trains it to respond to prompts as an assistant. In the InstructGPT work, researchers used labeler-written demonstrations and user-submitted prompts to fine-tune GPT-3, then used rankings of model outputs to further improve the model with human feedback. The resulting InstructGPT models were preferred by human evaluators over much larger base GPT-3 models in the researchers’ prompt distribution, showing that careful post-training can matter as much as raw model size.

SFT, however, has a limit. It teaches imitation. If the training data contains excellent answers, the model learns to imitate them. But many real-world situations are messy. There may be several acceptable answers, some more helpful than others. A reply might be factually correct but rude, safe but useless, polite but vague, or confident but wrong. Humans often know which answer they prefer, even when it is hard to write a perfect answer from scratch. That observation led to the development of reinforcement learning from human feedback.

RLHF: From “Correct Answer” to “Preferred Answer”

Reinforcement Learning from Human Feedback, or RLHF, became one of the defining alignment methods of the modern language-model era. The basic idea is elegant: instead of asking humans only to write ideal answers, ask them to compare model outputs. Given two or more responses to the same prompt, which one is better? The model then learns from those preferences.

One early example of this was in summarization. Researchers found that standard metrics, such as ROUGE, did not always align with what people wanted in a summary. So, they collected human comparisons of summaries, trained a reward model to predict which summaries people liked, and used reinforcement learning to fine-tune the summarization model. This approach led to summaries that people rated higher than those from models trained only with supervised learning.

The typical RLHF process has three main steps. First, supervised fine-tuning teaches the model to follow instructions. Next, a reward model is trained on preference data to score outputs based on human labelers’ preferences. Finally, the language model is improved using that reward model, often with a method called Proximal Policy Optimization, or PPO. PPO is a reinforcement learning technique that gradually updates the model, avoiding large changes. It became popular because it balances performance, simplicity, and efficiency better than older methods.

This process was very effective. InstructGPT used supervised examples, human rankings, and RLHF to create models that followed instructions better, gave more truthful answers, and produced less toxic output than the base model. Anthropic also used preference modeling and RLHF to train assistants that were both helpful and safer.

But RLHF is not simple. It requires training a separate reward model, sampling outputs, carefully tuning reinforcement learning, watching for reward hacking, and preventing the policy from drifting too far from the original model. It can be expensive, unstable, and sensitive to hyperparameters. This is the problem Direct Preference Optimization set out to solve.

DPO: The Alignment Method That Made Preference Training Simpler

Direct Preference Optimization, or DPO, offers a simpler way to train models using preferences. Instead of building a reward model and running a reinforcement learning loop, DPO fine-tunes the language model directly with pairs of responses: one chosen and one rejected for the same prompt.

The DPO paper describes RLHF as a complex and sometimes unstable process because it first fits a reward model and then fine-tunes the language model with reinforcement learning, while trying to avoid drifting from the original model. DPO changes the problem so the model can be optimized with a classification-style loss. This removes the need for a separate reward model, online sampling during training, or heavy reinforcement learning tools. The authors reported that DPO could match or improve response quality in summarization and dialogue, while being much simpler to train than PPO-based RLHF.

Here is a simple example. Imagine a dataset with this prompt:

“Explain photosynthesis to a 10-year-old.”

And two answers:

Answer A: clear, accurate, friendly, and age-appropriate.

Answer B: overly technical, dull, and confusing.

A human labeler chooses Answer A as the preferred answer. DPO trains the model to make Answer A more likely than Answer B. It does this by comparing the fine-tuned model to a reference model, typically the supervised fine-tuned model before preference optimization. This reference serves as a guide. The goal is to improve preferences while keeping the model close to its original helpful behavior, rather than letting it chase preference labels without limits.

This makes DPO attractive in practice. It is easier to set up than traditional RLHF, simpler to debug, and more accessible for teams without the resources for large-scale reinforcement learning. DPO changes alignment from a complex, multi-step process into something much closer to regular fine-tuning.

Still, DPO is not a perfect solution. It learns from the preferences it is given. If the data rewards long answers, the model might become wordy. If labelers prefer confident answers even when they are wrong, the model might value confidence over truth. If the dataset is too limited, the model might do well on familiar prompts but struggle with new ones. DPO makes optimization easier, but it does not answer the bigger question of what ‘better’ really means.

What DPO Gets Right — and What It Cannot Fix Alone

DPO’s main strength is that it makes aligning preferences practical. A team can collect pairs of chosen and rejected responses, fine-tune a model, and often see real improvements in tone, helpfulness, following instructions, and refusal behavior. For many use cases, such as customer support, writing assistants, tutoring, coding helpers, and internal tools, DPO works well because the desired behavior can often be demonstrated through comparisons.

Its second strength is stability. PPO-style RLHF can be powerful, but reinforcement learning on language models is delicate. A reward model can be exploited. A policy can learn strange shortcuts. Training can become sensitive to sampling settings, KL penalties, reward normalization, and many other details. DPO avoids much of that complexity by using a direct objective on offline preference data.

But DPO has important limits as well. It usually works offline, learning from a fixed dataset instead of exploring new outputs during training. It depends heavily on the quality of the preference data. DPO can pick up human labeler bias, company policy choices, and inconsistent annotations. It may also focus more on surface-level preferences, such as politeness, formatting, or confidence, rather than on deeper qualities like truthfulness, strong reasoning, or long-term safety.

A useful way to think about DPO is this: it is a very good steering wheel, but it still needs a good road map. The roadmap includes the preference dataset, the evaluation suite, the safety policy, the human review process, and the deployment feedback loop.

Modern Preference Optimization: Beyond DPO

DPO inspired many related methods. Many of these ask the same question: can we keep the benefits of RLHF while making training simpler, cheaper, more stable, or more data-efficient?

One family of methods extends preference learning beyond simple pairwise comparisons. Preference Ranking Optimization, or PRO, aims to learn from ranked lists of outputs rather than only one chosen and one rejected answer. This can be useful when human feedback provides richer ordering, such as “best, second-best, third-best,” rather than just “winner and loser.” The PRO paper argues that RLHF has drawbacks in terms of complexity, instability, and hyperparameter sensitivity, and proposes directly aligning the model’s output probability rankings with human preference rankings.

Another theoretical branch examines how methods such as RLHF and DPO relate to one another. Work on a general theoretical paradigm for learning from human preferences introduced a broader framework, ΨPO, and discussed Identity Preference Optimization as a special case that learns directly from preferences without a separate reward-modeling stage. This line of research matters because it helps explain when different algorithms are actually optimizing similar ideas and where their hidden assumptions differ.

Then there are methods designed to reduce dependence on a reference model or preference pairs. ORPO, or Odds Ratio Preference Optimization, combines supervised fine-tuning and preference alignment into a more unified objective. Instead of doing SFT first and a separate preference-optimization phase afterward, ORPO adds an odds-ratio term that contrasts favored and disfavored responses during training. Its authors present it as a reference-model-free, monolithic preference optimization method.

SimPO, or Simple Preference Optimization, is another reference-free method. It uses the average log probability of a sequence as an implicit reward and introduces a reward margin between winning and losing responses. The authors argue that this better matches how language models generate text and makes training more compute- and memory-efficient by eliminating the need for a reference model.

KTO, or Kahneman-Tversky Optimization, takes a different angle. Instead of requiring paired preference data, it can learn from a binary signal: whether an output is desirable or undesirable. Inspired by prospect theory, KTO models aspects of human utility, such as loss aversion, and proposes what the authors call a human-aware loss. In experiments, KTO matched or exceeded the performance of preference-based methods at model scales ranging from 1B to 30B while using this simpler binary feedback signal.al.

The main point is that no single method is the winner. Alignment is now a set of tools. DPO is a good default when you have high-quality preference pairs. ORPO and SimPO are useful when you need efficiency and simplicity. KTO works well when you have a few paired comparisons, but can collect simple labels. Ranking-based methods are useful when feedback is more detailed than a single pair.

AI Feedback, Constitutional AI, and Scalable Supervision

Human feedback is important, but it can be expensive, slow, and limited. People may disagree, miss subtle mistakes, or lack the expertise to judge complex scientific, legal, medical, or security outputs. Because of this, researchers have begun exploring AI-assisted feedback.

Constitutional AI is one influential approach. Instead of relying only on humans to label harmful outputs, the method uses a written set of principles — a “constitution” — to guide the model. In the supervised phase, the model generates critiques and revisions of its own responses based on those principles. In the reinforcement learning phase, AI-generated preferences are used to train a preference model, a process described as Reinforcement Learning from AI Feedback, or RLAIF.

This does not remove humans from the loop entirely. Humans still choose the principles, design the process, evaluate failures, and decide what behavior is acceptable. But it can reduce the amount of direct human labeling needed and make the alignment process more scalable. A related modern idea is deliberative alignment. Instead of teaching a model safety behavior only through many examples, deliberative alignment provides the model with explicit safety specifications and trains it to reason over them before answering. OpenAI describes this as a strategy for O-series models, in which reasoning models are trained on human-written safety specifications and how to apply them.em.

This is an important shift. Earlier alignment methods often used policies or constitutions to generate labels, but the model itself did not necessarily learn the policy text as something it could reason about. Deliberative alignment aims to embed the specification into the model’s decision process. The hope is that a model can handle borderline cases better when it has learned not only examples of good behavior but also the principles behind those examples.

Alignment as Defense: Constitutional Classifiers

Not all alignment happens inside the model’s weights. Some modern safety systems add external or semi-external guardrails around the model. Anthropic’s Constitutional Classifiers are an example: classifiers trained on synthetic data generated from a constitution, designed to monitor inputs and outputs for jailbreaks and harmful content. Anthropic reported that its first-generation system was tested through extensive human red-teaming, though later demo testing found a universal jailbreak.

In 2026, Anthropic described a next-generation version, Constitutional Classifiers++, using an ensemble-style, two-stage architecture that screens traffic cheaply first and escalates suspicious exchanges to a stronger classifier. The company reported lower costs, lower refusal rates, and greater robustness compared with the earlier approach.

This is a good reminder that alignment is not just about training. It also includes monitoring, evaluation, filtering, red teaming, policy design, and incident handling. A well-aligned system might use fine-tuning, preference optimization, retrieval, classifiers, tool restrictions, logging, and human review together.

Reasoning Alignment: GRPO, RLVR, and the Return of Reinforcement Learning

DPO and similar methods made offline preference optimization popular, but reinforcement learning is still important. In fact, for models that need strong reasoning, reinforcement learning has become even more valuable.

DeepSeekMath introduced Group Relative Policy Optimization (GRPO), a variant of PPO that improved mathematical reasoning while reducing PPO’s memory usage. The key idea is to use groups of generated outputs for the same prompt and learn from their relative performance, especially in domains like math, where correctness can be checked more objectively.

DeepSeek-R1 pushed this direction further. Its authors reported that reasoning abilities could be incentivized through reinforcement learning without human-labeled reasoning trajectories, leading to emergent behaviors such as self-reflection, verification, and dynamic strategy adaptation. The model was trained for verifiable tasks such as mathematics, coding competitions, and STEM problems, where the final answer or solution can often be automatically checked.

This connects to Reinforcement Learning with Verifiable Rewards, or RLVR. Unlike preference feedback, which asks humans or models to judge which answer is better, verifiable rewards use objective checks: Did the code pass the unit tests? Did the math answer match? Did the proof satisfy the verifier? Recent work on RLVR argues that it can improve reasoning by letting models explore and receive feedback from correctness signals, though researchers continue to debate how much it teaches genuinely new reasoning versus improving sampling and search.

This difference matters. Preference optimization works well for style, helpfulness, safety, and conversation quality. Verifiable-reward reinforcement learning is best when there is a clear external test. A strong coding model may need both: preference alignment to communicate well and avoid unsafe requests, and verifiable reward training to solve problems reliably.

How to Choose an Alignment Method

Choosing an alignment method is not about picking the newest acronym. It is about matching the method to the kind of feedback you have.

Supervised fine-tuning if you have expert demonstrations. It still serves as the basis for most assistant-style models. Given pairwise preference data, DPO is often a practical and strong next step. If you have ranked outputs, ranking-based optimization may be able to leverage that information more effectively. If you only have binary labels, for example, “acceptable” and “unacceptable”, then KTO might be worth looking into. ORPO or SimPO could be attractive if you need a lighter training setup without a reference model. If you have verifiable (e.g., math, coding, tool use, formal checking) tasks, RLVR-style training may give you a stronger signal than human preference alone.

Training alone is often not enough for safety-focused systems. Constitutional AI, deliberative alignment, classifiers, red-teaming, and policy-aware evaluation all matter. But the more powerful the model is, the more important it is to test not only normal use but also adversarial cases such as jailbreaks, hidden instructions, tricky requests, multi-step manipulation, and edge cases.

The best alignment process usually has multiple layers. A strong modern method is to pretrain a powerful base model, SFT, to teach it to follow instructions, use DPO or similar to improve preferences, use targeted RL or RLVR for areas needing strong reasoning, do safety training, evaluate with humans and automated tests, do aggressive red teaming, and deploy with monitoring and safeguards.

Evaluation: The Step No One Gets to Skip

Just the training loss is not enough to judge alignment. Even if a model has a perfect loss curve, it can still fail users in serious or embarrassing ways . Evaluation must ask real questions: Does the model obey instructions? Does it hallucinate less? Does it block malicious requests without blocking benign requests? Can it be generalized across languages, cultures, and topics? Is it still reliable if someone tries to fool it?

The InstructGPT paper noted that even improved models still made mistakes. That point remains true for modern systems. Alignment improves behavior, but it does not create perfection.

A good evaluation includes human preference tests, adversarial red-teaming, domain-specific benchmarks, factual checks, calibration tests, bias audits, and post-deployment live monitoring. Unit tests and verifiers are great for coding and maths. “Jailbreak tests and policies are required for safety. Businesses should test the model on their own workflows, documents, and potential failure cases, rather than relying solely on public benchmarks.

The Bigger Picture

DPO is valuable because it helps clarify alignment. It showed that a large part of the problem of preference alignment can be framed as a simple optimization problem: for a prompt, increase the likelihood of the preferred answer over the rejected answer while staying close to a reference model. Simplicity made DPO really influential.

But modern alignment is more than DPO. It also covers SFT, RLHF, RLAIF, ORPO, SimPO, KTO, ranking-based approaches, constitutional training, deliberative safety reasoning, external classifiers, GRPO, RLVR, and many hybrid methods. The field is moving away from one-size-fits-all training toward a more flexible approach in which different types of feedback drive distinct learning methods.

The most important lesson is simple: alignment is not a one-button process you turn on at the end of model development. It’s a process of iterating on how we shape behavior, testing our assumptions, finding failures, gathering better feedback, and improving the system. But DPO is one of the clearest and most useful tools in that process, and works best as part of a considered alignment stack.

A powerful model can still surprise us in unexpected ways. A well-aligned model can really help us. The future of AI is about not just making models smarter but also making them useful, honest, safe, and clear enough to earn our trust.