GPT-4 Technical Report

Reading guide

Problem and motivation

Large language models (LLMs) showed promise across natural language processing, but prior iterations (such as GPT-3.5) exhibited notable weaknesses in complex reasoning, reliability, factual precision, and safety. Furthermore, training very large frontier models requires enormous computational budgets, where standard hyperparameter tuning on the final scale is infeasible. The goal of GPT-4 is to build a large-scale, multimodal model capable of processing arbitrary mixtures of image and text inputs while maintaining predictable scaling behavior and drastically improved safety and alignment.

Key ideas

• Predictable Scaling: Optimization pipelines and architectures can be stabilized such that final test loss and downstream coding benchmarks (like HumanEval pass rates) are reliably extrapolated from smaller models trained using $1,000\times$ to $10,000\times$ less compute.
• Multimodal Understanding: GPT-4 accepts arbitrarily interlaced images and text, performing complex reasoning across charts, diagrams, exam questions, and visual humor without degrading pure language benchmarks.
• Pre-training Dominance with Targeted Post-training: Base model pre-training provides core reasoning and exam knowledge (with base and RLHF multiple-choice averages essentially equal at 73.7% vs. 74.0%), while RLHF and rule-based checks steer safety, formatting, and factuality.
• Rule-Based Reward Models (RBRMs): Alignment relies on model-assisted safety classifiers providing targeted auxiliary rewards during RLHF to steer appropriate refusals of disallowed prompts and reduce over-hedging on safe prompts.

Method (with the key equations)

GPT-4 is a Transformer-based autoregressive model pre-trained to predict the next token on publicly available web data and licensed data, followed by Reinforcement Learning from Human Feedback (RLHF).

1. Scaling Law Extrapolation

To ensure infrastructure predictability without empirical tuning on the primary run, final loss on internal code tokens was predicted using an irreducible loss formulation: $$L(C)=a{C}^b+c$$ where $C$ denotes normalized compute, and $a,b,c$ are fitted parameters from runs using up to $10,000\times$ less compute.

For downstream capability scaling on HumanEval, the mean log pass rate across problem subsets $P$ was modeled via: $$-{\mathbb{E}}_P[\log (pass\_rate(C))]=\alpha \cdot C^{-k}$$ allowing pre-training registration of functional coding performance.

2. Model-Assisted Safety Pipeline

Alongside human feedback, the RLHF optimization incorporates Rule-Based Reward Models (RBRMs). An RBRM is a zero-shot GPT-4 classifier that accepts:

• The user prompt (optional),
• The policy model completion,
• A human-authored rubric (a multi-class rubric classifying responses into categories such as desired refusal, evasive/rambling refusal, safe answer, or disallowed content).

Optimal RBRM weights provide an automated reward signal to train the policy model to refuse harmful requests (e.g., CBRN threats, cyberattacks) while reliably serving benign requests in adjacent sensitive topics.

Results (with the most important numbers)

• Professional & Academic Exams: GPT-4 scored 298/400 (90th percentile) on the Uniform Bar Examination (compared to 213/400, ~10th percentile for GPT-3.5). On the LSAT, it achieved 163 (88th percentile), and it obtained a 5 on AP Biology, AP Calculus BC (4), AP Chemistry (4), and AP Art History (5).
• Academic NLP Benchmarks: GPT-4 set new state-of-the-art marks on 5-shot MMLU (86.4% vs. 70.0% for GPT-3.5), 10-shot HellaSwag (95.3%), 25-shot ARC Challenge (96.3%), and 0-shot HumanEval (67.0%). On 5-shot chain-of-thought GSM-8K, it achieved 92.0%.
• Multilingual Capability: Translated versions of MMLU across 26 languages showed GPT-4 outperforming the English-language baseline of GPT-3.5 and PaLM in 24 languages, including low-resource languages like Latvian (85.5%), Welsh (71.4%), and Swahili (82.6%).
• Safety & Factuality: GPT-4 scored 19 percentage points higher than the latest GPT-3.5 on internal factuality evaluations. Disallowed response rates fell by 82%, sensitive request adherence improved by 29%, and toxicity generation on RealToxicityPrompts dropped from 6.48% (GPT-3.5) to 0.73%.

Limitations and open questions

• Hallucinations & Calibration: Despite reductions in factual errors, GPT-4 still hallucinates facts and makes basic reasoning slip-ups. While the base pre-trained model exhibits near-perfect calibration, RLHF post-training significantly damages calibration, making the aligned model confidently incorrect.
• Temporal Knowledge Cutoff: The vast majority of pre-training data cut off in September 2021; the model does not continuously update its world state or learn from ongoing deployment interactions.
• Inverse Scaling & Jailbreaks: Certain counter-intuitive patterns reverse only at scale (e.g., Hindsight Neglect), but vulnerabilities to adversarial prompting, system message manipulation, and multi-step jailbreaks persist.
• Lack of Architectural Transparency: Due to competitive and safety concerns, architectural specifications (parameter count, MoE design, dataset sizes, hardware, and exact compute) are not disclosed.