Training Compute-Optimal Large Language Models

Reading guide

Problem and motivation

Prior work (notably Kaplan et al., 2020) suggested that when compute increases, model parameters should scale aggressively ($N\propto C^{0.73}$) while token count increases much more slowly ($D\propto C^{0.27}$). Consequently, large models like GPT-3 (175B), Jurassic-1 (178B), Gopher (280B), and MT-NLG (530B) were consistently trained on around 300 billion tokens. However, this left large models significantly undertrained relative to their computational investments. This work revisits the trade-off: for a fixed compute budget $C$, how should parameter count $N$ and token count $D$ be allocated to minimize test loss?

Key ideas

• Equal scaling: For every doubling of model parameters, training tokens should also double ($N_{opt}\propto C^{0.5}$, $D_{opt}\propto C^{0.5}$).
• Cosine schedule alignment: Learning rate schedules must match the actual training horizon. Using intermediate points from a run decayed over a fixed long horizon (e.g., 130B tokens) overestimates loss for smaller token counts and skews empirical scaling estimates.
• Downstream efficiency: Training a smaller model on substantially more tokens yields identical or better accuracy at pretraining compute parity while dramatically cutting memory footprint, fine-tuning costs, and inference latency.

Method (with the key equations)

The authors evaluate over 400 models (70M to 16B parameters, trained across 5B to 500B tokens) using three complementary strategies:

1. Minimum over training curves: Vary the horizon over 4 durations for fixed model architectures. Extract the envelope of minimum loss per FLOP and fit power laws: $$N_{opt}\propto C^a,D_{opt}\propto C^b$$ yielding $a=0.50$, $b=0.50$.

2. IsoFLOP profiles: Fix total FLOP count across 9 discrete budgets ($6\times 10^{18}$ to $3\times 10^{21}$) with cosine decay aligned to the horizon. Fit parabolas over parameter counts to find the loss minima, yielding $a=0.49$, $b=0.51$.

3. Parametric loss fitting: Model the smoothed loss using a risk decomposition: $$\hat{L}(N,D)\triangleq E+\frac{A}{N^{\alpha }}+\frac{B}{D^{\beta }}$$ where $E$ represents natural text entropy, $A/N^{\alpha }$ represents representation suboptimality, and $B/D^{\beta }$ captures early stopping/finite data suboptimality. Minimizing Huber loss on log residuals gives $E=1.69$, $A=406.4$, $B=410.7$, $\alpha =0.34$, $\beta =0.28$. Under the constraint $\text{FLOPs}(N,D)\approx 6ND$: $$a=\frac{\beta }{\alpha +\beta }=0.46,b=\frac{\alpha }{\alpha +\beta }=0.54$$

Results (with the most important numbers)

• Chinchilla 70B: Trained with the same FLOP budget as Gopher ($5.76\times 10^{23}$ FLOPs, or $6.3\times 10^{23}$ under exact accounting) using 70B parameters and 1.4 trillion tokens.
• MMLU Benchmark: Chinchilla reaches 67.6% average 5-shot accuracy, outperforming Gopher (60.0%), GPT-3 (43.9%), and surpassing the June 2023 human expert forecast (63.4%). It achieves $>90\%$ accuracy on 4 distinct MMLU tasks.
• Downstream benchmarks: Outperforms Gopher on BIG-bench (65.1% vs 54.4%), LAMBADA (77.4% vs 74.5%), and closed-book Natural Questions (31.5% 5-shot vs 24.5%).
• Bias and Toxicity: Unprompted toxicity remains comparable between Gopher and Chinchilla (median 0.064 vs 0.066), while Winogender accuracy improves from 71.4% to 78.3% overall.

Limitations and open questions

• Only two large-scale frontier points at parity were directly verified (Gopher 280B vs Chinchilla 70B) due to training cost.
• The analysis assumes single-epoch training on web datasets; behavior when repeating data across multiple epochs was left unexplored.
• There is empirical evidence of slight negative curvature (concavity) in $\log (N_{opt})$ at high compute budgets, suggesting power laws may still slightly overestimate optimal model size.