LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu; Yelong Shen; Phillip Wallis; Zeyuan Allen-Zhu; Yuanzhi Li; Shean Wang; Lu Wang; Weizhu Chen

Papers · arXiv 2106.09685

LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

Published 2021-06-17cs.CLcs.AIcs.LG arXiv PDF

Read with translation and AI

Summary in three lines

🎯 Full fine-tuning of massive language models like GPT-3 175B requires retraining and storing billions of parameters per downstream task, creating prohibitive storage, VRAM, and deployment bottlenecks.
🔧 Freezes pre-trained weight matrices and injects trainable low-rank decomposition matrices ( $B A$ ) into Transformer dense layers, scaling the product by $\frac{α}{r}$ and enabling deployment merging without added latency.
📈 For GPT-3 175B, LoRA reduces trainable parameters by 10,000x and VRAM by 3x (from 1.2TB to 350GB) while matching or outperforming full fine-tuning on GLUE, WikiSQL, and SAMSum.

Abstract

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

Reading guide

Problem and motivation

Adapting large-scale pre-trained language models to diverse downstream applications traditionally relies on full fine-tuning (updating all parameters $Φ_{0}$ to $Φ_{0} + ΔΦ$ ). For massive architectures like GPT-3 175B, storing and serving independent copies of 175B-parameter models is prohibitively expensive in terms of storage, GPU memory, and I/O overhead. Prior parameter-efficient tuning techniques present distinct drawbacks:

Series Adapter Layers: Insert bottleneck modules sequentially into Transformer blocks. Because these sequential operations cannot be parallelized, they introduce significant inference latency—slowing forward passes by up to $30.3%$ in single-GPU, small-batch, online serving regimes.
Prefix / Prompt Tuning: Optimizes continuous prompt vectors prepended to layer activations. These methods are challenging to optimize, scale non-monotonically with prompt length, and consume sequence length budget otherwise available for context.

LoRA addresses this efficiency-quality trade-off based on the hypothesis that weight adaptation updates $Δ W$ have a low intrinsic rank during downstream transfer.

Key ideas

Low-Rank Reparametrization: For a frozen dense matrix $W_{0} \in R^{d \times k}$ , parameterize the adaptation update as $Δ W = B A$ , where $B \in R^{d \times r}$ , $A \in R^{r \times k}$ , and the rank $r ≪ min (d, k)$ .
Zero Added Inference Latency: Because the reparametrization is linear, the trained low-rank matrices can be directly folded into pre-trained weights ( $W = W_{0} + B A$ ) at deployment time, yielding identical inference architecture and zero latency penalty compared to fine-tuning.
Shared Base Model with Hot-Swapping: The large frozen model remains static in GPU memory; switching tasks requires only swapping low-rank matrices $A$ and $B$ , drastically cutting storage from hundreds of gigabytes to tens of megabytes.
Selective Attention Adaptation: Applying rank decomposition exclusively to self-attention projections (notably query $W_{q}$ and value $W_{v}$ ) achieves competitive performance while leaving feed-forward MLP layers frozen.

Method

For a dense projection layer producing $h = W_{0} x$ , LoRA modifies the computation to: $h = W_{0} x + Δ W x = W_{0} x + \frac{α}{r} B A x$

Key implementation details:

Initialization: $A$ is initialized from a random Gaussian distribution $N (0, σ^{2})$ and $B$ is initialized to zero, ensuring $Δ W = 0$ at the start of training.
Scaling Factor: The update $Δ W x$ is scaled by $\frac{α}{r}$ , where $α$ is a constant hyperparameter. When optimizing with Adam, setting $α$ equal to the initial rank $r$ avoids retuning learning rates when experimenting with different values of $r$ .
Deployment Merging: In production, the adapted weight matrix $W \in R^{d \times k}$ is explicitly materialized as $W = W_{0} + B A$ . When changing tasks, $W_{0}$ is restored via subtraction ( $W_{0} = W - B A$ ) and merged with new adapter weights ( $W^{'} = W_{0} + B^{'} A^{'}$ ).
Target Layers: While applicable to any dense layer, experiments focus primarily on Transformer attention projections ( $W_{q}, W_{k}, W_{v}, W_{o}$ ). Applying LoRA with $r = 4$ to $W_{q}$ and $W_{v}$ yields optimal downstream efficiency.

Results

GPT-3 175B: LoRA reduces trainable parameters from 175,255.8M to 4.7M ( $r = 1$ or $r_{v} = 2$ ) or 37.7M ( $r = 8$ ), reducing checkpoint size from 350GB to 35MB (a 10,000x reduction) and training VRAM from 1.2TB to 350GB. Training throughput increases by $25%$ (from 32.5 to 43.1 tokens/s per V100 GPU).
Downstream Accuracy (GPT-3):
- WikiSQL logical form validation accuracy: Full fine-tuning achieves $73.8%$ ; LoRA (37.7M) achieves $74.0%$ .
- MultiNLI matched validation accuracy: Full fine-tuning achieves $89.5%$ ; LoRA (4.7M) achieves $91.7%$ .
- SAMSum dialogue summarization: Full fine-tuning scores 52.0 / 28.0 / 44.5 ROUGE-1/2/L; LoRA (4.7M) achieves 53.8 / 29.8 / 45.9.
RoBERTa and DeBERTa (GLUE Benchmark): On RoBERTa Large, LoRA matches fine-tuning ( $88.9$ vs $89.0$ average score) using 0.8M parameters vs. 355.0M. On DeBERTa XXL (1.5B), LoRA scores $91.3$ average on GLUE with 4.7M parameters compared to $91.1$ for full fine-tuning (1,500.0M parameters).
Intrinsic Rank Analysis: Subspace similarity analysis reveals that top singular directions of $Δ W$ for $r = 8$ and $r = 64$ overlap heavily ( $> 0.5$ normalized similarity for dimension 1). In addition, $Δ W$ amplifies directions that are present but not heavily emphasized in $W$ , exhibiting an amplification factor $\frac{∥Δ W ∥ _{F}}{∥ U ^{⊤} W V ^{⊤} ∥ _{F}} \approx 21.5$ for $r = 4$ .

Limitations and open questions

Multi-Task Batching: When low-rank weights $B A$ are merged into $W_{0}$ to eliminate inference latency, batching inputs for multiple tasks simultaneously in a single forward pass is not straightforward.
Heuristic Layer Selection: The current practice relies on empirical heuristics (adapting $W_{q}$ and $W_{v}$ ) rather than a principled selection framework for determining which matrices to adapt across attention and MLP layers.
Extreme Domain Shifts: For downstream tasks that differ drastically from the pre-training distribution (such as adapting an English LLM to a distinct target language), low-rank adaptation may prove insufficient compared to higher-rank or full adaptation.

Key terms

Low-Rank Adaptation (LoRA): A parameter-efficient fine-tuning method that freezes base model weights and trains low-rank decomposition matrices in parallel to dense layers.
Intrinsic Rank: The low effective dimensionality or number of singular vectors required to capture meaningful parameter adaptation during task-specific transfer.
Adapter Layers: External bottleneck modules inserted sequentially into Transformer blocks, which introduce non-parallelizable sequential compute and inference latency.
Prefix Tuning: An adaptation strategy that prepends continuous trainable vectors to input tokens or layer hidden states, reducing available sequence context.
Weight Folding / Weight Merging: The arithmetic addition of low-rank matrices $B A$ directly into the pre-trained weight matrix $W_{0}$ to eliminate runtime overhead during inference.
Grassmann Distance Subspace Similarity: A metric $ϕ (A, B, i, j)$ measuring the overlap of column spans between orthonormal singular vector matrices of adaptation updates.
Feature Amplification Factor: The ratio between the Frobenius norm of $Δ W$ and the projection of $W$ onto $Δ W$ 's singular subspace, measuring how strongly task-specific directions are emphasized.

Questions

Why does LoRA introduce zero additional inference latency compared to adapters?

Because LoRA is linear ( $h = W_{0} x + B A x$ ), the low-rank product $B A$ can be explicitly computed and added into $W_{0}$ prior to serving ( $W = W_{0} + B A$ ). Inference then executes standard matrix multiplication without the sequential bottleneck layers that cause latency in adapter architectures.

Which Transformer weight matrices should LoRA be applied to?

Experiments on GPT-3 show that adapting both query ( $W_{q}$ ) and value ( $W_{v}$ ) projection weights yields the best performance given a fixed parameter budget. Adapting multiple attention matrices with a smaller rank (e.g., $r = 4$ ) outperforms allocating all parameters to a single matrix type with a higher rank.

What rank $r$ is sufficient for adapting large language models?

A very small rank such as $r = 1$ , $2$ , or $4$ is often sufficient for downstream tasks. Subspace similarity analysis reveals that the top singular vectors learned with small ranks overlap significantly with those learned with higher ranks (such as $r = 64$ ), while higher ranks mostly capture random optimization noise.

How does LoRA initialize its low-rank matrices?

Matrix $A$ is initialized using a random Gaussian distribution, while matrix $B$ is initialized to zero. This ensures that $Δ W = B A = 0$ at the start of training, leaving the model's pre-trained forward pass unchanged.

The overview on this page was written by AI from the paper and may contain mistakes. Read the paper for the details.

Read with translation and AI