Papers · arXiv 2005.14165
Language Models are Few-Shot Learners
Summary in three lines
- 🎯 Standard NLP models rely on task-specific fine-tuning on thousands of labeled examples, risking poor out-of-distribution generalization.
- 🔧 Train GPT-3 (up to 175B parameters) on 300B tokens and evaluate it using in-context prompt conditioning without any weight updates.
- 📈 In the few-shot setting, GPT-3 175B matches or nears fine-tuned models, achieving 71.2% on TriviaQA and 86.4% accuracy on LAMBADA.
Abstract
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
Reading guide
Problem and motivation
Standard pre-training plus fine-tuning pipelines require large task-specific labeled datasets (typically thousands to hundreds of thousands of examples). This requirement limits general applicability, increases susceptibility to spurious correlations, and diverges from human learning, which adapts to tasks using only instructions or minimal demonstrations. Previous attempts at in-context learning with smaller models trailed fine-tuning by wide margins.
Key ideas
- In-Context Meta-Learning: The model acquires pattern recognition skills during pre-training and performs task adaptation in the activation space during test-time forward passes without gradient updates.
- Spectrum of Conditioning: Tasks are tested across zero-shot (instruction only), one-shot (instruction + one demonstration), and few-shot ( demonstrations, typically 10 to 100 within a 2048-token window).
- Predictable Scaling: Power-law empirical scaling of validation loss translates into continuous downstream capability improvements across 8 model sizes (125M to 175B parameters).
Method
GPT-3 uses an autoregressive transformer architecture matching GPT-2, adding alternating dense and locally banded sparse attention patterns. Models are pre-trained on 300 billion tokens with an effective context size of .
Key evaluation formulations include:
- Per-Token Completion Likelihood: For multiple-choice datasets, candidate completions are evaluated by comparing normalized token probabilities.
- Unconditional Normalization: On benchmarks like ARC, OpenBookQA, and RACE, model likelihood is normalized against an unconditioned baseline:
where is a generic prompt like
"Answer: "or"A: ". - Generation Decoding: Free-form completion tasks use beam search with a beam width of 4 and length penalty .
Results
- Closed-Book QA: GPT-3 175B reaches 71.2% few-shot on TriviaQA (exceeding open-domain fine-tuned RAG at 68.0%), 41.5% on WebQuestions, and 29.9% on Natural Questions.
- Language Modeling & Cloze: Achieves a perplexity of 20.5 on PTB (zero-shot). On LAMBADA, framing the task as a cloze test yields 86.4% few-shot accuracy (18% above prior SOTA).
- Translation: Translating into English reaches 39.2 BLEU (Fr En), 40.6 BLEU (De En), and 39.5 BLEU (Ro En), outperforming prior unsupervised NMT baselines.
- Reasoning & Synthesis: Solves 2-digit addition at 100% and 3-digit addition at 80.4% in few-shot settings. Generates synthetic news articles that human evaluators distinguish from real articles with only 52% accuracy (near random chance).
Limitations and open questions
- Weak Tasks: Struggles with comparisons across sequences, notably NLI (e.g., ANLI, RTE) and Word-in-Context (WiC at 49.4%, near random chance).
- Structural Limitations: Purely autoregressive training prevents bidirectional conditioning, which hurts tasks requiring lookback or whole-sequence re-reading.
- Data Contamination: Heavy web scraping causes train-test overlap, necessitating extensive post-hoc -gram filtering audits (e.g., on Winograd and PIQA).
- Societal Impacts: Internet-scale training ingests significant gender, racial, and religious biases, and generated outputs can be deployed for malicious text generation.
Key terms
- in-context learning
- The inner-loop adaptation process where a pre-trained model infers and performs a task at test time purely from natural language instructions and prompt demonstrations without gradient updates.
- few-shot
- An evaluation setting where the model receives as many demonstration examples (typically 10 to 100) as fit within its context window alongside the task description.
- one-shot
- An evaluation condition allowing only a single task demonstration paired with a natural language task description.
- zero-shot
- An evaluation setup where the model receives only a natural language instruction describing the task, without any demonstration examples.
- closed-book question answering
- Answering factual questions relying solely on knowledge stored in model parameters without access to external documents or retrieval systems at test time.
- data contamination
- The accidental inclusion of downstream evaluation benchmarks, test sets, or dev sets within the broad internet-scale pre-training corpus.
- sparse attention
- An architectural modification alternating dense attention layers with locally banded sparse attention patterns to balance computational efficiency across long contexts.
- unconditional normalization
- Dividing conditional completion likelihood by the probability of the completion given only a generic prompt, used to eliminate length and frequency biases on multiple-choice tasks.
Questions
How many parameters does the full GPT-3 model have and how was it trained?
GPT-3 has 175 billion trainable parameters across 96 layers, 96 attention heads, and a hidden dimension of 12,288. It was trained for 300 billion tokens on filtered Common Crawl, WebText2, Books1, Books2, and Wikipedia using Adam on Microsoft V100 GPU clusters.
Does GPT-3 use fine-tuning for downstream benchmarks?
No, all primary results in the paper are evaluated without gradient updates or fine-tuning, relying solely on zero-shot, one-shot, or few-shot context prompts.
How easily can humans identify news articles generated by GPT-3 175B?
Human evaluators achieved roughly 52% accuracy in distinguishing short (200 word) and longer (500 word) GPT-3 generated news articles from human-written articles, which is barely above random chance (50%).
On which tasks does GPT-3 struggle the most?
GPT-3 performs poorly on natural language inference tasks (e.g., ANLI and RTE), bidirectional comparisons like Word-in-Context (WiC, scoring 49.4%), and certain reading comprehension tasks like QuAC and RACE.
The overview on this page was written by AI from the paper and may contain mistakes. Read the paper for the details.