Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Reading guide

Problem and motivation

Large language models (LMs) acquire vast world knowledge during unsupervised pre-training, but steering their outputs toward helpful and safe behaviors remains challenging. Standard Reinforcement Learning from Human Feedback (RLHF) pipelines tackle this in three stages: Supervised Fine-Tuning (SFT), fitting a reward model on pairwise preferences via maximum likelihood under a Bradley-Terry model, and policy optimization via reinforcement learning (typically PPO) to maximize estimated reward while enforcing a Kullback-Leibler (KL) divergence penalty relative to a reference policy ${\pi }_{\text{ref}}$.

However, RLHF with actor-critic RL algorithms is notoriously complex, brittle, sensitive to hyperparameters, and computationally costly because it requires training multiple models (actor, critic, reference, reward model) and sampling generations in the training loop.

Key ideas

• Analytic mapping between policy and reward: The optimal solution to the KL-constrained reward maximization objective ${\max }_{{\pi }_{\theta }}\mathbb{E}[r(x,y)]-\beta {\mathbb{D}}_{\text{KL}}({\pi }_{\theta }\parallel {\pi }_{\text{ref}})$ has a known closed-form representation: ${\pi }_r(y\mid x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y\mid x)\exp \left(\frac{1}{\beta }r(x,y)\right)$.
• Reward reparameterization: Rearranging this identity allows the ground-truth reward to be expressed through the optimal policy: $r(x,y)=\beta \log \frac{\pi (y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}+\beta \log Z(x)$.
• Cancellation of the partition function: Under pairwise comparison models like Bradley-Terry, preference probabilities depend only on the difference in rewards $r(x,y_w)-r(x,y_l)$. The unknown prompt-dependent partition function $\beta \log Z(x)$ cancels out entirely.
• Implicit reward modeling: Instead of optimizing an explicit scalar reward network and running RL, DPO fits the policy directly via maximum likelihood on static preference pairs $\mathcal{D}=\{(x,y_w,y_l)\}$.

Method (with the key equations)

Under the Bradley-Terry preference model, the probability that completion $y_w$ is preferred over $y_l$ given prompt $x$ is: $$p^{\ast }(y_w\succ y_l\mid x)=\sigma (r^{\ast }(x,y_w)-r^{\ast }(x,y_l))$$

Substituting the policy-implied reward ${\hat{r}}_{\theta }(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$ yields the Direct Preference Optimization (DPO) objective: $${\mathcal{L}}_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\right)\right]$$

The gradient with respect to policy parameters $\theta$ is: $${\nabla }_{\theta }{\mathcal{L}}_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-\beta {\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\sigma ({\hat{r}}_{\theta }(x,y_l)-{\hat{r}}_{\theta }(x,y_w))\left[{\nabla }_{\theta }\log {\pi }_{\theta }(y_w\mid x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l\mid x)\right]\right]$$

Here, the gradient increases the likelihood of $y_w$ and decreases the likelihood of $y_l$. The dynamic scalar weight $\sigma ({\hat{r}}_{\theta }(x,y_l)-{\hat{r}}_{\theta }(x,y_w))$ scales the update based on how incorrectly the implicit reward currently scores the pair, preventing model degeneration that occurs with unweighted likelihood objectives.

Results (with the most important numbers)

• Controlled Sentiment Generation (IMDb + GPT-2-large): DPO strictly dominates PPO across the entire reward-KL frontier, achieving higher reward at every level of sequence-level KL divergence from ${\pi }_{\text{ref}}$, even outperforming PPO provided with ground-truth oracle rewards (PPO-GT).
• TL;DR Summarization (GPT-J 6B):
  • At temperature 0.0, DPO achieves an approximate 61% win rate against reference completions evaluated by GPT-4, compared to 57% for PPO (at its optimal temperature 0.0).
  • In a human study comparing head-to-head summaries on TL;DR, DPO completions at temperature 0.25 were preferred 58% of the time over PPO completions at temperature 0.0.
  • DPO maintains stability across temperatures, whereas PPO degrades sharply toward the base model at temperature 1.0.
• Out-of-Distribution Generalization (CNN/DailyMail): Evaluated zero-shot on CNN/DailyMail news articles without further fine-tuning, DPO achieves a 0.36 win rate against ground-truth summaries at temperature 0 (vs. 0.26 for PPO), and 0.31 at temperature 0.25 (vs. 0.23 for PPO).
• Single-turn Dialogue (Anthropic HH + Pythia-2.8B): DPO is the only computationally lightweight method tested that outperforms the preferred test-set completions, matching or slightly outperforming a computationally expensive Best-of-128 sampling baseline.

Limitations and open questions

• Out-of-distribution generalization: While initial tests on CNN/DailyMail show strong transfer, the paper notes more study is needed to understand how direct policy optimization compares to learning an explicit generalizable reward model over broader shifts.
• Reward over-optimization: The authors observe slight performance decreases late in training during dialogue experiments, leaving open whether and how classic "reward hacking" or over-optimization occurs within implicit reward models.
• Unlabeled data utilization: Unlike PPO, which can generate on arbitrary unlabeled prompts during RL rollouts, standard DPO is an offline algorithm trained on paired comparisons; extending it via self-labeling or iterative collection is an open area.
• Evaluation dependencies: Automated evaluation relied heavily on GPT-4 prompt phrasing (concise prompt vs. simple prompt), highlighting sensitivity in LLM-as-a-judge methodologies.