Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Reading guide

Problem and motivation

Standard few-shot prompting gives language models input–output exemplars, enabling strong performance on many direct question-answering tasks. However, on multi-step reasoning tasks—such as grade school arithmetic, multi-hop commonsense questions, and symbolic manipulations—simply scaling model size under standard prompting often yields flat scaling curves and poor performance. Prior approaches that attempted to solve multi-step problems relied either on training/fine-tuning specialized rationale models (which require large amounts of expensive supervised rationale data) or formal neuro-symbolic representations. The paper investigates whether general-purpose language models can execute multi-step reasoning entirely in-context without any parameter updates.

Key ideas

• Chain-of-Thought Decomposition: Humans decompose complex problems into sequential reasoning steps before stating an answer. Language models can mimic this behavior when guided by in-context demonstrations structured as triples: $⟨\text{input},\text{chain of thought},\text{output}⟩$.
• Emergence at Scale: Chain-of-thought (CoT) prompting does not provide meaningful improvements—and often degrades performance—for small models ($<10\text{B}$ parameters). It emerges as an effective capability only in models with $\sim 100\text{B}$ parameters or more.
• Variable Intermediate Computation: Generating natural language tokens allows the model to dynamically allocate additional compute (via autoregressive token generation) to harder multi-step problems, which direct answer prediction cannot do.

Method (with the key equations)

The method modifies the few-shot prompting paradigm. Rather than prompting with input–output pairs $(x_i,y_i)$: $${\mathcal{P}}_{\text{standard}}=\{(x_1,y_1),(x_2,y_2),\dots ,(x_k,y_k)\}$$ where the model directly computes $P(y\mid x,{\mathcal{P}}_{\text{standard}})$, chain-of-thought prompting inserts a natural language reasoning chain $c_i=(c_{i,1},c_{i,2},\dots ,c_{i,m})$ before the final answer: $${\mathcal{P}}_{\text{CoT}}=\{(x_1,c_1,y_1),(x_2,c_2,y_2),\dots ,(x_k,c_k,y_k)\}$$ At inference time for a test input $x$, the model autoregressively generates both the reasoning chain and the answer: $$P(c,y\mid x,{\mathcal{P}}_{\text{CoT}})=\prod _{t=1}^{|c|}P(c_t\mid x,{\mathcal{P}}_{\text{CoT}},c_{<t})\prod _{j=1}^{|y|}P(y_j\mid x,{\mathcal{P}}_{\text{CoT}},c,y_{<j})$$ For arithmetic benchmarks, the authors also evaluate a post-hoc external calculator extension: mathematical equations generated within $c$ are evaluated using Python's eval, propagating results forward via string matching to prevent arithmetic calculation slips.

Results (with the most important numbers)

• Arithmetic Reasoning (GSM8K): On GSM8K, standard prompting on PaLM 540B achieves 17.9%, whereas 8-shot CoT prompting achieves 56.9% (and 58.6% with an external calculator), surpassing the prior supervised best of 55% from finetuned GPT-3 175B with a verifier. Codex achieves 63.1% (65.4% with calculator).
• Other Math Benchmarks: PaLM 540B reaches 79.0% on SVAMP (vs. 69.4% standard), 73.9% on ASDiv (vs. 72.1% standard), 35.8% on AQuA (vs. 25.2% standard), and 93.3% on MAWPS (vs. 79.2% standard).
• Commonsense Reasoning: PaLM 540B CoT establishes a new state of the art on StrategyQA at 75.6% (vs. 68.6% standard and 69.4% prior supervised best) and outperforms human sports enthusiasts on Sports Understanding (95.4% vs. 84.0%).
• Symbolic Manipulation & Length Generalization: On last-letter concatenation, PaLM 540B moves from 7.6% (standard) to 99.4% (CoT) in-domain (2 words). When tested out-of-domain on 4 words, standard prompting drops to 0.0%, while CoT retains 63.0% accuracy.
• Ablations: Ablations on LaMDA 137B confirm that "equation only" prompting (5.4% on GSM8K), "variable compute only" via dots (6.4%), and "reasoning after answer" (6.1%) fail to match standard CoT prompting (14.3%), demonstrating that intermediate natural language steps prior to the answer are essential.

Limitations and open questions

• Not Guaranteed to Reason Factually: Generated chains can be unfaithful, semantically flawed, or hallucinated, sometimes arriving at the right answer by coincidence (especially in binary or multiple-choice formats).
• Model Scale Constraint: The technique is expensive to serve in production environments because it requires models with $\sim 100\text{B}+$ parameters to work reliably.
• Open Question of True Reasoning: The paper notes that imitating human rationales does not prove that underlying neural representations execute formal reasoning.