Scaling Laws for Neural Language Models

Reading guide

Problem and motivation

Language modeling has progressed rapidly with Transformer architectures, but training large models requires substantial compute and data. Prior to this work, it was unclear how performance (measured by cross-entropy loss) depended on architectural shape (depth vs. width), parameter count, dataset size, and training compute. Without quantitative scaling relationships, researchers typically trained smaller models near convergence rather than understanding how to optimally allocate a fixed compute budget.

Key ideas

• Scale dominates architecture: Within broad bounds, Transformer performance depends heavily on the non-embedding parameter count $N$, dataset size $D$, and compute $C$, while architectural hyperparameters like depth, width, and number of attention heads have minimal impact when $N$ is fixed.
• Smooth power-law scaling: Cross-entropy test loss scales as a power law with $N$, $D$, and compute $C_{\text{min}}$ across more than six to eight orders of magnitude.
• Sublinear data growth to avoid overfitting: Avoiding overfitting requires scaling dataset size sublinearly with parameters: $D\propto N^{0.74}$.
• Compute efficiency means large models stopped early: To maximize performance under a fixed compute budget $C_{\text{min}}$, one should train very large models and stop significantly before convergence, allocating compute as $N\propto C_{\text{min}}^{0.73}$, batch size $B\propto C_{\text{min}}^{0.24}$, and steps $S_{\text{min}}\propto C_{\text{min}}^{0.03}$.

Method (with the key equations)

The authors evaluate decoder-only Transformer models on WebText2 (using BPE tokenization, context $n_{\text{ctx}}=1024$). Model size $N$ specifically excludes embedding parameters: $N\approx 12n_{\text{layer}}{d}_{\text{model}}^2$.

Non-embedding compute per forward-backward pass is approximated as $C\approx 6NBS$.

1. Individual Power Laws:

   • Parameter-limited: $$L(N)={\left(\frac{N_c}{N}\right)}^{{\alpha }_N},{\alpha }_N\approx 0.076,N_c\approx 8.8\times 10^{13}$$
   • Data-limited: $$L(D)={\left(\frac{D_c}{D}\right)}^{{\alpha }_D},{\alpha }_D\approx 0.095,D_c\approx 5.4\times 10^{13}$$
   • Compute-limited (at batch size $B\ll B_{\text{crit}}$): $$L(C_{\text{min}})={\left(\frac{C_c^{\text{min}}}{C_{\text{min}}}\right)}^{{\alpha }_C^{\text{min}}},{\alpha }_C^{\text{min}}\approx 0.050,C_c^{\text{min}}\approx 3.1\times 10^8\text{ PF-days}$$

2. Simultaneous Parameter and Data Scaling: $$L(N,D)={\left[{\left(\frac{N_c}{N}\right)}^{\frac{{\alpha }_N}{{\alpha }_D}}+\frac{D_c}{D}\right]}^{{\alpha }_D}$$

3. Training Dynamics and Critical Batch Size: $$B_{\text{crit}}(L)=\frac{B_{\ast }}{L^{1/{\alpha }_B}},B_{\ast }\approx 2\times 10^8\text{ tokens},{\alpha }_B\approx 0.21$$ The learning curve fits steps $S_{\text{min}}$ (steps when $B\gg B_{\text{crit}}$): $$L(N,S_{\text{min}})={\left(\frac{N_c}{N}\right)}^{{\alpha }_N}+{\left(\frac{S_c}{S_{\text{min}}}\right)}^{{\alpha }_S},{\alpha }_S\approx 0.76,S_c\approx 2.1\times 10^3$$

4. Optimal Allocations: $${\alpha }_C^{\text{min}}=\frac{1}{1/{\alpha }_S+1/{\alpha }_B+1/{\alpha }_N}\approx 0.052$$ Efficient training targets a loss approximately $10\%$ above converged loss ($\frac{{\alpha }_N}{{\alpha }_S}\approx 0.10$).

Results (with the most important numbers)

• Trends hold over 8 orders of magnitude in compute, 6 in parameters, and 2 in data.
• Doubling $N$ reduces loss by a factor of $2^{-{\alpha }_N}\approx 0.95$.
• Overfitting avoidance threshold: $D\gtrsim (5\times 10^3)N^{0.74}$. Increasing model size by $8\times$ requires only about a $5\times$ data increase.
• For a $10\times$ increase in compute budget $C_{\text{min}}$, compute-efficient training increases model size $N$ by $\approx 5\times$ ($N\propto C_{\text{min}}^{0.73}$), batch size by $\approx 1.7\times$ ($B\propto C_{\text{min}}^{0.24}$), and training steps by only $\approx 1.07\times$ ($S_{\text{min}}\propto C_{\text{min}}^{0.03}$).
• Compared to training to near-convergence ($2\%$ deviation), compute-optimal training uses $65\%$ less compute, $2.7\times$ more parameters, and $7.7\times$ fewer parameter update steps.
• Transfer loss to out-of-distribution corpora (Books Corpus, Common Crawl, English Wikipedia) improves smoothly with model size, incurring a nearly constant penalty offset relative to WebText2 validation loss.

Limitations and open questions

• The paper lacks a rigorous theoretical derivation (e.g., statistical physics foundation) explaining the precise exponents.
• The compute scaling $C\approx 6NBS$ omits attention-specific costs proportional to $n_{\text{ctx}}^2$ or context-dependent compute when $n_{\text{ctx}}\gtrsim 12d_{\text{model}}$.
• Hyperparameters such as regularization, dropout tuning, and learning rate schedules at extreme scales were held mostly fixed or not tuned per dataset size.
• Extrapolating compute-optimal trends leads to a contradiction where $L(C_{\text{min}})$ crosses $L(D)$ at $C^{\ast }\sim 10^4\text{ PF-days}$, $N^{\ast }\sim 10^{12}$ parameters, and $L^{\ast }\sim 1.7\text{ nats/token}$, signaling that the empirical trends must eventually break down or flatten near the natural language entropy floor.