LLaMA: Open and Efficient Foundation Language Models

Reading guide

Problem and motivation

Standard scaling laws (such as Chinchilla) compute the optimal trade-off between parameter size and dataset size for a fixed training compute budget. However, this framing neglects the inference budget, which dominates the lifetime operational cost of serving language models. For serving efficiency, smaller models trained on significantly more data than compute-optimal guidelines recommend are preferable because they are substantially cheaper to host and run.

Furthermore, premier state-of-the-art models (like GPT-3, Chinchilla, and PaLM) rely on private, non-reproducible datasets or prohibitively large architectures. LLaMA demonstrates that competitive foundation models can be trained exclusively on publicly available data, allowing open distribution to researchers.

Key ideas

• Inference-centric training: Rather than stopping training at compute-optimal dataset sizes (e.g., ~200B tokens for a 10B model), LLaMA trains 7B and 13B models on 1.0T tokens, and 33B and 65B models on 1.4T tokens.
• Open datasets only: All pre-training data is drawn from public sources: CommonCrawl (67%), C4 (15%), GitHub (4.5%), Wikipedia (4.5%), Books (4.5%), ArXiv (2.5%), and Stack Exchange (2%).
• Modernized Transformer architecture: Incorporates pre-normalization with RMSNorm for stability, SwiGLU activations, and Rotary Positional Embeddings (RoPE) instead of absolute positions.
• Engineering efficiency: Employs memory-efficient attention (via xformers), custom activation checkpointing, and sequence/model parallelism.

Method

LLaMA uses an autoregressive Transformer decoder with modifications:

1. Pre-normalization: Input normalization using RMSNorm before each sub-layer: $${\bar{a}}_i=\frac{a_i}{\text{RMS}(\mathbf{a})}{g}_i,\text{where }\text{RMS}(\mathbf{a})=\sqrt{\frac{1}{d}\sum _{i=1}^d{a}_i^2}$$
2. SwiGLU activation: Replaces standard ReLU/GELU activations with SwiGLU, sizing the intermediate dimension to $\frac{2}{3}4d$ instead of $4d$.
3. Rotary Embeddings (RoPE): Applied at every layer to encode relative positions without absolute position embeddings.
4. Optimization: AdamW with ${\beta }_1=0.9,{\beta }_2=0.95$, weight decay of $0.1$, gradient clipping of $1.0$, $2\text{,}000$ warmup steps, and a cosine learning rate schedule decaying to 10% of maximum learning rate. Batch size is 4M tokens across all model scales.

Energy and carbon footprint estimation across training runs follows: $$\text{Wh}=\text{GPU-h}\times (\text{GPU power consumption})\times \text{PUE}$$ $${\text{tCO}}_2\text{eq}=\text{MWh}\times 0.385$$

Results

• Common Sense Reasoning: LLaMA-13B beats GPT-3 (175B) across multiple benchmarks (e.g., 78.1% vs 60.5% on BoolQ; 73.0% vs 70.2% on WinoGrande). LLaMA-65B matches or surpasses PaLM-540B on PIQA (82.8% vs 82.3%) and HellaSwag (84.2% vs 83.4%).
• Closed-book QA: On TriviaQA (0-shot), LLaMA-65B reaches 68.2%, and LLaMA-13B reaches 56.6% (surpassing GPT-3's comparable performance while running on a single V100 GPU).
• Code Generation: On HumanEval (pass@1), LLaMA-65B achieves 23.7%, outperforming PaLM-62B (15.9%) and matching PaLM-cont 62B (23.7%).
• MMLU & Instruction Finetuning: Base LLaMA-65B reaches 63.4% 5-shot accuracy on MMLU (trailing PaLM-540B's 69.3%, likely due to smaller pre-training book corpora). However, instruction finetuning on basic prompts (LLaMA-I 65B) lifts performance to 68.9%.

Limitations and open questions

• Hallucination and Truthfulness: Scores on TruthfulQA remain modest (LLaMA-65B scores 0.57 on Truthful and 0.53 on Truthful*Informative), showing a continuing tendency to hallucinate.
• Toxicity scaling: On RealToxicityPrompts, toxicity increases with model size, especially on respectful prompts (0.141 for 65B vs 0.081 for 7B).
• Academic knowledge gap: Performance on MMLU lagged behind Chinchilla-70B and PaLM-540B, which the authors hypothesize is due to training on only ~177GB of books and academic papers compared to ~2TB in proprietary sets.