Finetuned Language Models Are Zero-Shot Learners

Reading guide

Problem and motivation

Large language models (such as GPT-3 175B) excel at few-shot learning by leveraging in-context exemplars. However, their zero-shot capabilities remain substantially weaker on complex NLP benchmarks such as reading comprehension, natural language inference (NLI), and question answering. Untuned models rely on prompts formatted as continuations of pretraining data, which is an unnatural and brittle format for tasks typically articulated as explicit directions or queries.

The authors explore whether supervised finetuning on a broad collection of diverse NLP tasks, explicitly phrased as natural language instructions, enables language models to generalize zero-shot to completely unseen task types at inference time.

Key ideas

• Instruction Tuning: Convert existing NLP datasets into instruction-response formats using natural language prompt templates, combining multitask supervision with prompting.
• Task Cluster Holdout: Group datasets into thematic clusters (e.g., NLI, Closed-Book QA, Translation, Sentiment). When testing zero-shot generalization on a target cluster $\mathcal{D}$, all datasets belonging to $\mathcal{D}$ (and closely tied clusters) are excluded from training to guarantee true zero-shot cross-task evaluation.
• Classification with Options: For classification tasks, append an OPTIONS: suffix listing the valid candidate labels directly in the prompt, guiding the generative model's output distribution without restricting it purely to rank-classification logit comparisons.

Method

The base model is LaMDA-PT, a dense, left-to-right, decoder-only transformer with 137 billion parameters, pretrained on 2.49T BPE tokens across web documents, code, Wikipedia, and dialog data (about 10% non-English).

1. Data Aggregation: 62 datasets from TensorFlow Datasets are grouped into 12 clusters. For each dataset, 10 unique natural language instruction templates are composed (including up to 3 'inverted' templates, e.g., generating text given sentiment).
2. Training Setup: Training examples per dataset are capped at 30k. Batches are constructed using an examples-proportional mixing scheme with a mixing rate maximum of 3,000 to prevent large datasets from dominating. Training runs for 30k gradient steps with batch size 8,192 tokens using the Adafactor optimizer (learning rate $3\times 10^{-5}$), sequence length 1024 (input) and 256 (target), with sequence packing.
3. Evaluation Protocol: To evaluate across $c$ held-out clusters, $c$ distinct FLAN models are trained. Performance is reported both as the average across all instruction templates and with the best template chosen on a validation set.
4. Inference with Few-Shot Exemplars: When $k$ exemplars $(x_i,y_i)$ are available, inputs are formatted as: $$\text{instruct}(x_1)\oplus y_1\oplus \text{instruct}(x_2)\oplus y_2\oplus \cdots \oplus \text{instruct}(x_k)\oplus y_k\oplus \text{instruct}(x)$$ where $\oplus$ represents concatenation with a separator token, capped at 16 exemplars and 960 tokens.

Results

• Zero-Shot Comparison against GPT-3: Using the best dev template, zero-shot FLAN 137B outperforms zero-shot GPT-3 175B on 20 of 25 evaluated benchmarks and surpasses few-shot GPT-3 on 10 datasets, including ANLI R1–R3, RTE, BoolQ, OpenbookQA, and StoryCloze.
• Task Types: Large gains occur on tasks naturally formulated as instructions (e.g., NLI, reading comprehension, closed-book QA, translation). On RTE, FLAN achieves 84.1% (best dev template) vs. zero-shot GPT-3's 63.5% and few-shot GPT-3's 72.9%. On BoolQ, FLAN reaches 82.9% vs. GPT-3's 60.5% zero-shot and 77.5% few-shot.
• Scale Dependency: Ablations across 422M, 2B, 8B, 68B, and 137B parameters reveal that instruction tuning hurts zero-shot generalization on unseen tasks for models $\le 8\text{B}$ parameters. The positive transfer emerges only at scale ($\sim 68\text{B}$ and $137\text{B}$).
• Instruction Importance: Finetuning with no instruction templates (input-output only: 37.3% 4-cluster average) or dataset names only (46.6%) yields significantly worse zero-shot performance than natural instructions (55.2%).
• Prompt Tuning Compatibility: Parameter-efficient prompt tuning applied to FLAN yields higher SuperGLUE accuracy than when applied to the untuned LaMDA-PT base model, especially in 32-example low-resource regimes.

Limitations and open questions

• Sentence-Completion Tasks: Instruction tuning yields minimal benefit (and sometimes hurts) on language modeling-style tasks where instructions are redundant, such as Winogrande or sentence completions in commonsense reasoning.
• Instruction Complexity: The paper only investigates short, single-sentence instructions rather than long, elaborate prompt guidelines typically given to human raters.
• Context Window & Compute: FLAN's 1024-token context length prevented rigorous evaluation of summarization tasks. Furthermore, running and serving a 137B parameter dense model requires high compute resources.