Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Reading guide

Problem and motivation

Pre-trained language models store substantial factual knowledge within their parameters (parametric memory). However, they have significant drawbacks: they struggle to expand or update their knowledge without retraining, cannot easily provide provenance or interpretability for their outputs, and are prone to hallucinations.

While hybrid parametric/non-parametric models like REALM and ORQA combined differentiable retrieval with masked language modeling, they were restricted to extractive question answering. Standard sequence-to-sequence (seq2seq) architectures like BART and T5 remained largely closed-book. This work bridges that gap by proposing Retrieval-Augmented Generation (RAG), a general fine-tuning framework that couples a pre-trained dense neural retriever with an autoregressive seq2seq generator for both open-domain generation and classification.

Key ideas

• Hybrid Parametric and Non-Parametric Memory: Non-parametric memory consists of a dense vector index over 21M 100-word Wikipedia passages accessed via Maximum Inner Product Search (MIPS). Parametric memory is a pre-trained BART generator.
• End-to-End Fine-Tuning: The query encoder and generator are trained jointly by treating the retrieved passages as latent variables and minimizing the negative marginal log-likelihood of target tokens without explicit retrieval supervision.
• Two Conditioning Formulations:
  • RAG-Sequence: Conditions on the same retrieved document across the entire generated sequence.
  • RAG-Token: Allows each generated token to attend and marginalize over different retrieved passages.
• Hot-Swappable World Knowledge: The external index can be swapped at inference time (e.g., swapping a 2016 Wikipedia index for a 2018 index) to update world knowledge without modifying model weights.

Method

Given an input sequence $x$, the retriever $p_{\eta }(z|x)$ produces a distribution over documents $z$, and the generator $p_{\theta }(y_i|x,z,y_{1:i-1})$ generates target tokens conditioned on both $x$ and $z$.

The retriever uses Dense Passage Retriever (DPR): $$p_{\eta }(z|x)\propto \exp \left(\mathbf{d}(z{)}^{\top }\mathbf{q}(x)\right)$$ where $\mathbf{d}(z)={\text{BERT}}_d(z)$ and $\mathbf{q}(x)={\text{BERT}}_q(x)$. Top-$k$ candidates are retrieved using FAISS MIPS. The document index and document encoder ${\text{BERT}}_d$ are kept fixed, while ${\text{BERT}}_q$ and the BART generator parameters $\theta$ are fine-tuned.

RAG-Sequence

The model treats the retrieved document as a single latent variable across the entire sequence $y$: $$p_{\text{RAG-Sequence}}(y|x)\approx \sum _{z\in \text{top-}k(p(\cdot |x))}{p}_{\eta }(z|x)\prod _i^N{p}_{\theta }(y_i|x,z,y_{1:i-1})$$ Decoding can be evaluated using Thorough Decoding (running beam search per document and computing forward passes for candidate hypotheses across all documents) or Fast Decoding (approximating missing beam probabilities as 0).

RAG-Token

The model marginalizes over documents at each token step, enabling token-level content mixing across sources: $$p_{\text{RAG-Token}}(y|x)\approx \prod _i^N\sum _{z\in \text{top-}k(p(\cdot |x))}{p}_{\eta }(z|x)p_{\theta }(y_i|x,z,y_{1:i-1})$$ This yields a standard autoregressive transition probability that plugs directly into beam search.

Results

• Open-Domain QA: RAG sets new state-of-the-art results on open benchmarks using exact match (EM):
  • Natural Questions: 44.5 (RAG-Seq) vs. 41.5 (DPR) and 36.6 (T5-11B+SSM).
  • TriviaQA (standard open split): 56.8 (RAG-Seq); on the TQA-Wiki test split: 68.0 (RAG-Seq) vs. 60.5 (T5-11B+SSM).
  • WebQuestions: 45.5 (RAG-Token) vs. 41.1 (DPR).
  • CuratedTrec: 52.2 (RAG-Seq) vs. 50.6 (DPR).
  • Notably, on Natural Questions, RAG achieved 11.8% EM when the correct answer was completely missing from the retrieved context (where extractive models score 0%).
• Abstractive Generation: On Open MS-MARCO, RAG-Sequence achieves 40.8 Rouge-L and 44.2 Bleu-1, outperforming BART (38.2 Rouge-L, 41.6 Bleu-1).
• Jeopardy Question Generation: Human annotators rated RAG-Token as more factual than BART in 42.7% of cases versus BART being superior in only 7.1%. Evaluators also found RAG more specific (37.4% vs. 16.8%).
• Fact Verification: On FEVER 3-way, RAG reaches 72.5% label accuracy (within 4.3% of pipeline systems using gold evidence supervision); on 2-way, RAG achieves 89.5% accuracy.
• Knowledge Update via Hot-Swapping: Using an 82-leader temporal test between 2016 and 2018, RAG scored 70% accuracy for 2016 leaders using the 2016 index and 68% for 2018 leaders with the 2018 index, dropping drastically (12% and 4%) when mismatched.

Limitations and open questions

• Retrieval Collapse: For tasks that require weaker explicit factual conditioning (such as open-ended story generation), the retriever can collapse to retrieving static, invariant documents while the generator learns to ignore them.
• Fixed Document Indexing: The document encoder is kept static during task fine-tuning due to the computational cost of continually re-indexing 21M vectors, limiting retriever adaptation.
• Source Grounding and Bias: Non-parametric memory remains vulnerable to misinformation and biases embedded in the underlying knowledge corpus (Wikipedia).
• Joint Pre-training: The retriever and generator components were pre-trained separately prior to downstream task fine-tuning, leaving joint pre-training from scratch as an open direction.