Training language models to follow instructions with human feedback

Reading guide

Problem and motivation

Standard large language models (LLMs) optimize next-token prediction on web corpora. This pretraining objective is misaligned with the user goal of having an assistant follow instructions helpfully, truthfully, and harmlessly. Consequently, baseline models like GPT-3 frequently hallucinate facts, invent fake context, reflect toxic text, or disregard explicit user constraints. Scaling up model parameters alone does not eliminate these issues.

Key ideas

• Alignment through Reinforcement Learning from Human Feedback (RLHF): Use human ratings to shape model outputs toward user intentions rather than raw continuation likelihood.
• Three-step Training Pipeline: (1) Supervised fine-tuning (SFT) on prompt-demonstration pairs, (2) Reward Model (RM) training on pairwise comparisons over ranked completions, and (3) Policy optimization using Proximal Policy Optimization (PPO).
• Alignment Tax Mitigation: Standard PPO fine-tuning causes performance degradations on standard NLP benchmarks (e.g., SQuADv2, DROP). Mixing pretraining distribution gradients into PPO updates (PPO-ptx) mitigates these capability losses without sacrificing human preference gains.

Method (with the key equations)

1. Supervised Fine-Tuning (SFT): Fine-tune base GPT-3 models on approximately 13k prompt demonstrations written by labelers and submitted via the OpenAI API Playground.
2. Reward Model Training: For each prompt $x$, labelers rank $K$ completions ($K\in [4,9]$), generating $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ comparisons. The 6B reward model $r_{\theta }(x,y)$ is trained with pairwise cross-entropy loss across all pairs in a batch element: $$\mathrm{loss}(\theta )=-\frac{1}{\left(\genfrac{}{}{0ex}{}{K}{2}\right)}{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log \sigma \left(r_{\theta }(x,y_w)-r_{\theta }(x,y_l)\right)\right]$$ where $y_w$ is the preferred completion over $y_l$.
3. RL Policy Optimization (PPO-ptx): Optimize the policy ${\pi }_{\varphi }^{\mathrm{RL}}$ against scalar rewards while penalizing deviation from the SFT model via token-level KL divergence, and optionally retaining pretraining log-likelihood: $$\mathrm{objective}(\varphi )={\mathbb{E}}_{(x,y)\sim D_{{\pi }_{\varphi }^{\mathrm{RL}}}}\left[r_{\theta }(x,y)-\beta \log \left(\frac{{\pi }_{\varphi }^{\mathrm{RL}}(y\mid x)}{{\pi }^{\mathrm{SFT}}(y\mid x)}\right)\right]+\gamma {\mathbb{E}}_{x\sim D_{\text{pretrain}}}\left[\log {\pi }_{\varphi }^{\mathrm{RL}}(x)\right]$$ Here $\beta$ is the KL penalty coefficient and $\gamma$ governs the pretraining gradient mix.

Results (with the most important numbers)

• Human Preferences: Labelers prefer outputs from the 1.3B InstructGPT model over those from the 175B base GPT-3 (a 100x parameter difference). Outputs from 175B InstructGPT are preferred to 175B GPT-3 $85\pm 3\%$ of the time, and $71\pm 4\%$ over few-shot prompted 175B GPT-3.
• Truthfulness and Hallucinations: On TruthfulQA, InstructGPT produces truthful and informative outputs roughly twice as often as GPT-3. On closed-domain API tasks, the hallucination rate decreases from 41% (GPT-3) to 21% (InstructGPT).
• Toxicity: InstructGPT yields approximately 25% fewer toxic outputs than GPT-3 on RealToxicityPrompts when prompted to be respectful.
• Generalization: InstructGPT reliably generalizes to held-out labelers ($77.3\%$ agreement) and out-of-distribution instructions such as code QA/summarization and non-English instructions, despite English constituting over 96% of training data.

Limitations and open questions

• Safety and Alignment Trade-offs: InstructGPT primarily prioritizes user instruction adherence during training; if prompted to be malicious or biased, it follows the instruction effectively.
• Simple Errors: The model still hallucinates, can hedge excessively due to epistemic humility conditioning, and struggles with false premises or complex multiple constraints.
• Representation: Fine-tuning relies on contractor feedback (around 40 hired labelers) and API Playground customer queries, which do not capture the broad spectrum of end users.