RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su; Yu Lu; Shengfeng Pan; Ahmed Murtadha; Bo Wen; Yunfeng Liu

Papers · arXiv 2104.09864

RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu

Published 2021-04-20cs.CLcs.AIcs.LG arXiv PDF

Read with translation and AI

Summary in three lines

🎯 Standard position embeddings are either additive absolute vectors or relative modifications that break compatibility with linear self-attention.
🔧 RoPE multiplies query and key vectors by orthogonal block-diagonal rotation matrices so that their inner product depends solely on relative token distance.
📈 Achieves 27.5 BLEU on WMT 2014 En-De, converges faster during pre-training, and outperforms WoBERT on CAIL2019-SCM by 1.5% accuracy at sequence length 1024.

Abstract

Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}.

Reading guide

Problem and motivation

Self-attention in Transformers is inherently permutation-invariant, requiring explicit position information. Previous approaches broadly fall into two camps: absolute position embeddings added directly to word representations (learned or fixed sinusoidal vectors) and relative position embeddings integrated into the attention matrix (e.g., Shaw et al., Transformer-XL, T5, DeBERTa). However, existing relative methods largely rely on additive decompositions that modify intermediate attention logits, rendering them incompatible with kernel-based linear self-attention mechanisms that require factorizable representations. Furthermore, models often struggle to scale seamlessly across variable sequence lengths.

Key ideas

Relative Distance via Vector Rotation: Instead of adding position vectors, RoPE applies a position-dependent rotation matrix to the affine-transformed queries and keys. The inner product between a rotated query at index $m$ and a rotated key at index $n$ naturally depends only on the relative distance $m - n$ .
Orthogonality and Norm Preservation: Because the rotation operator is orthogonal, it preserves the Euclidean norm of queries and keys, stabilizing hidden states.
Long-term Decay: Following sinusoidal frequency scaling, token interactions naturally decay as relative distance increases, matching the linguistic intuition that distant tokens have weaker dependencies.
Compatibility with Linear Attention: Because position information is applied directly to token vectors before taking dot products, RoPE can be combined with kernel-based linear attention (such as Performer).

Method (with the key equations)

The objective is to find functions $f_{q} (x_{m}, m)$ and $f_{k} (x_{n}, n)$ whose inner product depends only on embeddings and relative distance: $⟨ f_{q} (x_{m}, m), f_{k} (x_{n}, n)⟩ = g (x_{m}, x_{n}, m - n)$

In a 2D subspace, treating real vectors as complex numbers yields the solution: $f_{q} (x_{m}, m) = (W_{q} x_{m}) e^{im θ}, f_{k} (x_{n}, n) = (W_{k} x_{n}) e^{in θ}$

For an even embedding dimension $d$ , the space is partitioned into $d /2$ 2D subspaces with frequencies $Θ = {θ_{i} = 1000 0^{- 2 (i - 1) / d}, i \in [1, \dots, d /2]}$ . The general transformation applies the block-diagonal rotation matrix $R_{Θ, m}^{d}$ : $f_{{q, k}} (x_{m}, m) = R_{Θ, m}^{d} W_{{q, k}} x_{m}$

The resulting attention score between positions $m$ and $n$ satisfies: $q_{m}^{⊺} k_{n} = (R_{Θ, m}^{d} W_{q} x_{m})^{⊺} (R_{Θ, n}^{d} W_{k} x_{n}) = x_{m}^{⊺} W_{q} R_{Θ, n - m}^{d} W_{k} x_{n}$

To compute this efficiently without sparse matrix multiplication, RoPE is implemented via element-wise operations: $R_{Θ, m}^{d} x = x_{1} x_{2} ⋮ x_{d - 1} x_{d} \otimes cos m θ_{1} cos m θ_{1} ⋮ cos m θ_{d /2} cos m θ_{d /2} + - x_{2} x_{1} ⋮ - x_{d} x_{d - 1} \otimes sin m θ_{1} sin m θ_{1} ⋮ sin m θ_{d /2} sin m θ_{d /2}$

Results (with the most important numbers)

Machine Translation: On WMT 2014 English-German, RoFormer reaches 27.5 BLEU, outperforming the Transformer-base baseline (27.3 BLEU).
Pre-training Convergence: When training BERT-base with RoPE on BookCorpus and Wikipedia, RoFormer converges faster and reaches lower masked language modeling (MLM) loss than vanilla BERT.
GLUE Fine-Tuning: Pre-trained RoFormer significantly improves downstream scores on MRPC (89.5 vs. 88.9 F1), STS-B (87.0 vs. 85.8 Spearman), and QQP (86.4 vs. 71.2 F1), though it performs lower than BERT on SST-2 (90.7 vs. 93.5), QNLI (88.0 vs. 90.5), and MNLI-m/mm (80.2/79.8 vs. 84.6/83.4).
Linear Attention (Performer): On the Enwik8 dataset, adding RoPE to a 12-layer character-based Performer significantly accelerates convergence and decreases training loss.
Long Text Semantic Matching: On CAIL2019-SCM, RoFormer-512 achieves 68.29% test accuracy (vs. 67.77% for BERT-512 and 68.10% for WoBERT-512). Scaling to RoFormer-1024 increases test accuracy to 69.79% (a 1.5% absolute improvement over WoBERT-512).

Limitations and open questions

The paper notes a lack of theoretical explanation for why the 2D rotary formulation converges faster during pre-training compared to additive alternatives.
While the long-term decay property is mathematically demonstrated, why RoFormer generalizes noticeably better than peer models on long sequences remains an open question.
Evaluating RoFormer across large-scale pre-training requires substantial GPU compute, leaving broader scaling properties for future work.

Key terms

Rotary Position Embedding (RoPE): A position encoding mechanism that injects position information by multiplying query and key representations with orthogonal block-diagonal rotation matrices.
RoFormer: A Transformer architecture equipped with Rotary Position Embedding rather than standard absolute or additive relative position encodings.
Long-term Decay: The mathematical property where the inner product of queries and keys diminishes as the relative positional distance $∣ m - n ∣$ grows.
Linear Attention: Attention mechanisms that compute kernel feature maps to alter matrix multiplication order and achieve linear $O (N)$ computational complexity.
PerFormer: A linear attention Transformer architecture tested in the paper to demonstrate that RoPE seamlessly equips linear attention with relative position information.
CAIL2019-SCM: The Chinese AI and Law 2019 Similar Case Matching benchmark, consisting of legal case triplets used to evaluate long-text modeling capabilities beyond 512 tokens.
WoBERT: A word-based Chinese BERT model used as a comparative baseline and pre-training foundation for RoFormer on Chinese corpora.

Questions

How does RoPE differ from standard sinusoidal position embeddings?

Sinusoidal embeddings are added to input token vectors before linear projections, whereas RoPE is applied multiplicatively as a rotation directly onto query and key projections inside the attention layer.

Why is RoPE compatible with linear self-attention while other relative encodings are not?

Existing relative encodings modify pairwise attention logits inside softmax operations, breaking the factorization needed for linear attention. RoPE rotates query and key vectors individually without changing their norms, allowing linear attention kernels to proceed normally.

Does applying RoPE require dense matrix multiplication by $R^d_{\Theta, m}$?

No. Because $R_{Θ, m}^{d}$ is a sparse block-diagonal matrix composed of $2 \times 2$ blocks, it is implemented computationally as element-wise multiplication and pairwise recombination using cosine and sine vectors.

How does RoFormer perform when scaling to longer contexts?

On the CAIL2019-SCM benchmark, expanding context length from 512 to 1024 increases RoFormer test accuracy from 68.29% to 69.79%, showing strong length generalizability.

The overview on this page was written by AI from the paper and may contain mistakes. Read the paper for the details.

Read with translation and AI