The Llama 3 Herd of Models

Reading guide

Problem and motivation

Modern artificial intelligence systems rely heavily on massive foundation models, yet the most capable models (such as GPT-4) have remained closed. Building an openly accessible, frontier-class model requires overcoming massive engineering bottlenecks across three pillars: data curation, compute scale, and managing complexity.

The authors develop Llama 3, a family of multilingual language models spanning 8B, 70B, and 405B parameters supporting up to 128K context windows, zero-shot and multi-step tool use, advanced reasoning, and code generation.

Key ideas

• Managing Complexity with Dense Architecture: Rather than switching to Mixture-of-Experts (MoE) architectures, the authors stick to a standard dense Transformer (126 layers, 16,384 dimension, 128 attention heads for 405B) with Grouped-Query Attention (GQA) and RoPE with base frequency $\theta =500,000$ to maximize pre-training stability.
• Data Scaling and Rigorous Filtering: Pre-training expands from Llama 2's 1.8T tokens to approximately 15.6T tokens across diverse domains (~50% general knowledge, 25% math/reasoning, 17% code, 8% multilingual).
• Predictable Scaling Laws: A two-stage scaling law framework links compute to downstream task performance (such as ARC Challenge) across four orders of magnitude before training commences.
• Iterative Post-Training: Six rounds of alignment combining Supervised Fine-Tuning (SFT), Rejection Sampling (RS), and Direct Preference Optimization (DPO) with targeted synthetic data generation.
• 4D Parallelism & Infrastructure: Pre-training 405B over up to 16K H100 GPUs using a combination of Tensor Parallelism (TP), Context Parallelism (CP), Pipeline Parallelism (PP), and Fully Sharded Data Parallelism (FSDP/DP).

Method (with the key equations)

1. Scaling Law Formulation

Compute-optimal token allocation is modeled as a power-law function of compute budget $C$:

$$N^{\star }(C)=A{C}^{\alpha }$$

Empirical fits over compute budgets between $6\times 10^{18}$ and $10^{22}$ FLOPs identified $(\alpha ,A)=(0.53,0.29)$. Extrapolating to the target budget of $3.8\times 10^{25}$ FLOPs suggested training a 402B parameter model on 16.55T tokens, guiding the final architecture of 405B parameters.

2. Context Window Extension and Attention Masking

Context scaling from 8K to 128K tokens occurs in six continuous pre-training stages across ~800B tokens. To prevent cross-contamination across documents packed within the same long context sequence, a custom document attention mask is applied.

Context Parallelism (CP) splits input sequences into $2\times \text{CP}$ chunks, running an all-gather communication pattern over key ($K$) and value ($V$) representations. Because GQA reduces $K$ and $V$ sizes relative to query ($Q$), all-gather communication overhead $O(S)$ is negligible compared to attention computation $O(S^2)$.

3. Direct Preference Optimization (DPO) Adaptations

In preference optimization, formatting tokens (headers and termination markers) are explicitly masked out from the DPO loss to prevent repetitive tails or premature stops. Furthermore, an auxiliary negative log-likelihood (NLL) regularization term is added to the chosen sequence:

$${\mathcal{L}}_{\text{reg}}=0.2\cdot {\mathcal{L}}_{\text{NLL}}(\text{chosen})$$

Results (with the most important numbers)

• Flagship 405B Benchmarks:
  • MMLU (5-shot): 87.3% (0-shot CoT: 88.6%)
  • GSM8K (8-shot, CoT): 96.8%
  • MATH (0-shot, CoT): 73.8%
  • HumanEval (0-shot): 89.0%
  • ARC Challenge (0-shot): 96.9%
  • IFEval: 88.6%
• Smaller Models (8B and 70B): Both models outperform comparable open-source peers (e.g., Gemma 2 9B, Mistral 7B, Mixtral 8x22B). For instance, Llama 3 8B attains 73.0% on 0-shot CoT MMLU and 84.5% on GSM8K.
• Long Context: Perfect 100% retrieval on single Needle-In-A-Haystack up to 128K tokens, and 98.1% on Multi-needle recall for the 405B model.
• Pre-training Efficiency: BF16 Model FLOPs Utilization (MFU) reached 38–43% across configurations on 16,384 H100 GPUs, maintaining over 90% effective training time over a 54-day snapshot despite 466 interruptions.

Limitations and open questions

• Hardware Reliability and Scale Fluctuation: At 16K GPU scale, hardware interruptions averaged several times per day (78% unexpected interruptions due to hardware/GPU issues). Simultaneous power fluctuations of tens of megawatts during checkpointing or collective steps challenge datacenter electrical grids.
• Adversarial Robustness Gap: While resilient on paraphrase benchmarks (PAWS), mathematical reasoning and QA adversarial benchmarks (GSM-Plus, Adv SQuAD) demonstrate significant degradation compared to non-adversarial benchmarks.
• Multimodal Availability: Vision and speech adapters demonstrated competitive capabilities in research settings, but were kept under active development and excluded from the initial public release.