Llama 2: Open Foundation and Fine-Tuned Chat Models

Reading guide

Problem and motivation

While open-access pretrained foundational models like LLaMA-1, BLOOM, and Falcon matched the raw base performance of models like GPT-3, closed commercial chat assistants (e.g., ChatGPT, Claude) maintained a significant lead. This performance advantage stemmed from intensive post-pretraining alignment via instruction tuning and Reinforcement Learning with Human Feedback (RLHF), which requires massive compute, proprietary annotation pipelines, and red-teaming. Meta developed and openly released Llama 2 and Llama 2-Chat (7B, 13B, and 70B parameters, plus an unreleased 34B variant) to democratize aligned dialogue models and provide transparent methodologies for alignment and safety.

Key ideas

• Expanded Pretraining: Pretrained on 2.0 trillion tokens (a 40% increase over Llama 1) with context doubled to 4,096 tokens, and Grouped-Query Attention (GQA) introduced for the 34B and 70B models to accelerate autoregressive decoding.
• Quality-Centric SFT: Demonstrates that tens of thousands of high-quality vendor annotations outperform millions of noisy third-party examples, capping SFT at 27,540 curated demonstrations.
• Dual Reward Modeling: Trains two distinct reward models—one for Helpfulness and one for Safety—to avoid the tension where safety constraints degrade helpfulness.
• Two-Stage RLHF Alignment: Combines iterative Rejection Sampling fine-tuning (distilled from 70B) with subsequent Proximal Policy Optimization (PPO).
• Ghost Attention (GAtt): A context-distillation technique applied across multi-turn dialogues to preserve persistent system instructions across 20+ conversation turns without instruction fading.

Method

Pretraining Architecture

Llama 2 uses an autoregressive transformer architecture featuring RMSNorm pre-normalization, SwiGLU activations, and rotary positional embeddings (RoPE), with GQA added to larger scales (34B and 70B). Models were trained with AdamW (eta_1=0.9, eta_2=0.95), cosine learning rate decay down to 10%, weight decay of 0.1, and gradient clipping of 1.0.

Reward Modeling

Human annotators provide 4-point scale comparative preferences between two sampled responses. Reward models are initialized from pretrained chat checkpoints and optimized with a margin-augmented binary ranking loss: $${\mathcal{L}}_{\text{ranking}}=-\log \left(\sigma \left(r_{\theta }(x,y_c)-r_{\theta }(x,y_r)-m(r)\right)\right)$$ where $x$ is the prompt context, $y_c$ is the preferred response, $y_r$ is the rejected response, and $m(r)$ is a discrete margin dependent on annotator preference intensity.

RLHF Objectives

Optimization is performed via Rejection Sampling (sampling $K$ completions from the latest policy and picking the highest-reward completion) and PPO. The PPO objective maximizes expected reward while penalizing policy drift via KL divergence against the initial policy ${\pi }_0$: $$R(g\mid p)={\tilde{R}}_c(g\mid p)-\beta D_{\text{KL}}({\pi }_{\theta }(g\mid p)\parallel {\pi }_0(g\mid p))$$ where ${\tilde{R}}_c(g\mid p)$ is a whitened, logit-transformed piecewise combination of safety ($R_s$) and helpfulness ($R_h$) reward models: $$R_c(g\mid p)=\{\begin{array}{ll}{R}_s(g\mid p)& \text{if }\text{textsc}is\_safety(p)\text{ or }{R}_s(g\mid p)<0.15\\ R_h(g\mid p)& \text{otherwise}\end{array}$$

Ghost Attention (GAtt)

To force multi-turn consistency, instruction $inst$ is synthetically appended to each user prompt in an existing dialogue. Responses are sampled from the latest policy, but the loss is computed exclusively on the target turn's tokens (zeroing loss on earlier turns) while retaining $inst$ only in the first turn at inference time.

Results

• Base Model Benchmarks: Llama 2 70B outperforms all open-source base models, achieving 68.9 on MMLU (5-shot), 56.8 on GSM8K (8-shot), and 51.2 on Big-Bench Hard (3-shot), rivaling GPT-3.5 and PaLM-540B.
• Human Dialogue Evaluations: In human evaluations spanning ~4,000 prompts, Llama 2-Chat 70B scored a 36% win rate and 31.5% tie rate against ChatGPT (gpt-3.5-turbo-0301). Llama 2-Chat 34B won over 75% of head-to-head comparisons against Vicuna-33B and Falcon 40B.
• Safety & Toxicity: Safety-aligned Llama 2-Chat reduced toxic generation rates on ToxiGen from 24.60% (base 70B) down to 0.01%, showing lower safety violation percentages than ChatGPT and Falcon across human red-team benchmarks.

Limitations and open questions

• Coding & Complex Reasoning: Llama 2 70B lags substantially behind GPT-4 and PaLM-2-L on coding benchmarks (HumanEval pass@1 of 29.9% vs. 67.0% for GPT-4).
• Language Coverage: Approximately 89.7% of the pretraining data is English, making model performance and safety alignment brittle in other languages.
• False Refusals & Over-conservatism: Rigorous safety tuning introduces borderline prompt false refusals (e.g., refusing benign queries that mention words like 'bomb' or 'Christmas Crack').
• 34B Release Delay: The 34B model variant was withheld from public release due to insufficient red-teaming time prior to publication.