ReAct: Synergizing Reasoning and Acting in Language Models

Reading guide

Problem and motivation

Large language models (LLMs) excel at reasoning via Chain-of-Thought (CoT) and at decision-making by outputting actionable commands. However, these two capabilities have typically been studied in isolation:

• CoT reasoning is ungrounded: the model relies exclusively on frozen internal representations, making it prone to factual hallucination, lack of up-to-date knowledge, and compounding error propagation.
• Action-only systems map contexts straight to environment actions: they lack an explicit working memory to formulate abstract goals, maintain intermediate subgoals, or handle unexpected state changes.

Inspired by human inner speech, this paper investigates whether interleaving reasoning steps with environmental interactions creates a mutually reinforcing loop: reasoning guides actions, while action feedback grounds reasoning.

Key ideas

• Action Space Augmentation: The agent's action space is extended to include language thoughts that produce no external observation but update internal context.
• Dense vs. Sparse Thought Regimes: Depending on the task, thoughts can be tightly coupled (alternating step-by-step for multi-hop QA) or asynchronous and sparse (used only when planning or handling surprises in long-horizon games).
• Hybrid Reasoning: Combining internal parametric knowledge (via CoT self-consistency) with retrieved external feedback (via ReAct) balances factual precision with reasoning agility.

Method (with the key equations)

Consider an interactive environment where at time $t$, an agent receives observation $o_t\in \mathcal{O}$, maintains trajectory context $c_t=(o_1,a_1,\dots ,o_{t-1},a_{t-1},o_t)$, and acts according to policy $\pi (a_t\mid c_t)$.

ReAct expands the action space to: $$\hat{\mathcal{A}}=\mathcal{A}\cup \mathcal{L}$$ where $\mathcal{L}$ is the space of free-form language expressions.

• If an agent generates an environmental action ${\hat{a}}_t\in \mathcal{A}$, the environment returns an observation $o_t\in \mathcal{O}$, forming the next context $c_{t+1}=(c_t,{\hat{a}}_t,o_t)$.
• If an agent generates a thought ${\hat{a}}_t\in \mathcal{L}$, no external observation is generated; the thought directly appends to the agent context: $c_{t+1}=(c_t,{\hat{a}}_t)$.

For knowledge tasks (HotpotQA, FEVER), the API action set is $\mathcal{A}=\{\text{Search}[\text{entity}],\text{Lookup}[\text{string}],\text{Finish}[\text{answer}]\}$. For hybrid setups:

• $\text{ReAct}\to \text{CoT-SC}$: If ReAct does not finish within fixed steps (e.g., 7 for HotpotQA, 5 for FEVER), fall back to majority voting from CoT self-consistency.
• $\text{CoT-SC}\to \text{ReAct}$: If majority agreement among $n$ samples drops below $n/2$, fall back to ReAct.

Results (with the most important numbers)

• Knowledge Reasoning (PaLM-540B):
  • On HotpotQA, combining both regimes ($\text{ReAct}\to \text{CoT-SC}$) achieves 35.1% Exact Match (EM) compared to Standard (28.7%), CoT (29.4%), and Act (25.7%).
  • On FEVER, $\text{CoT-SC}\to \text{ReAct}$ reaches 64.6% Accuracy vs. Standard (57.1%) and CoT (56.3%).
  • Error analysis shows ReAct hallucinates in 0% of its failure cases on HotpotQA, whereas hallucination accounts for 56% of CoT failures.
• Decision Making (ALFWorld & WebShop):
  • On ALFWorld, best-of-6 ReAct achieves an average success rate of 71% (and average across prompts of 57%), outperforming Act-only (best 45%), Inner Monologue-style prompting (best 53%), and an imitation learning agent BUTLER (best 37%) trained on $10^5$ trajectories.
  • On WebShop, 1-shot ReAct achieves a 40.0% success rate and 66.6 average score, beating imitation + reinforcement learning baselines (28.7% success, 62.4 score) trained on over 10,000 instructions.
• Finetuning: Bootstrapping PaLM-8B and PaLM-62B with 3,000 self-generated correct trajectories makes ReAct the top performer, with PaLM-8B finetuned ReAct outperforming all 62B prompting methods.

Limitations and open questions

• In-Context Prompting Limits: Due to context window constraints, LLMs can only observe a handful of demonstrations (1–6 exemplars), making it challenging to master massive action spaces purely from prompts.
• Search Derailment & Loops: Under greedy decoding, non-informative search returns cause ReAct to fail in 23% of error cases, occasionally trapping the model in repetitive loops.
• Safety & Tool Execution: Granting LLMs external actions in open environments introduces risks of unintended actions (e.g., executing unwanted web transactions or retrieving sensitive data).