BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Reading guide

Problem and motivation

Prior language representation models relied either on feature-based approaches (such as ELMo) or fine-tuning approaches (such as OpenAI GPT). While effective, both paradigms were restricted by unidirectional language modeling objectives: standard auto-regressive models process text strictly left-to-right (or right-to-left). In unidirectional models, tokens can only attend to previous tokens in self-attention layers to prevent words from trivially "seeing themselves" across multi-layer contexts. ELMo attempted bidirectionality by shallowly concatenating separately trained left-to-right and right-to-left LSTMs, but this fails to capture deep, joint conditioning on bidirectional context at every layer. This unidirectionality limits sentence-level representations and is especially detrimental for token-level downstream tasks such as question answering, where understanding both left and right context simultaneously is essential.

Key ideas

• Deep Bidirectional Transformer Architecture: BERT uses a multi-layer bidirectional Transformer encoder, allowing self-attention layers to condition jointly on both left and right context across all layers.
• Masked Language Model (MLM): Inspired by the Cloze task, BERT randomly masks 15% of the input tokens and trains the model to predict the masked tokens using their context, resolving the "seeing itself" triviality issue.
• Pre-train/Fine-tune Mismatch Mitigation: Because [MASK] tokens do not appear during fine-tuning, the selected 15% tokens are replaced with [MASK] 80% of the time, a random token 10% of the time, and kept unchanged 10% of the time.
• Next Sentence Prediction (NSP): To capture sentence-pair relationships critical for Natural Language Inference (NLI) and Question Answering (QA), BERT pre-trains on a binarized task predicting whether sentence $B$ naturally follows sentence $A$ ($IsNext$) or is a random sentence from the corpus ($NotNext$).
• Unified Architecture for Downstream Tasks: The exact same pre-trained architecture is reused for downstream tasks with minimal added parameters (e.g., a simple linear classification head or start/end span pointers).

Method

Model Configuration

BERT is evaluated primarily on two sizes:

• ${\text{BERT}}_{\text{BASE}}$: $L=12$ layers, $H=768$ hidden size, $A=12$ attention heads, total parameters = $110\text{M}$.
• ${\text{BERT}}_{\text{LARGE}}$: $L=24$ layers, $H=1024$ hidden size, $A=16$ attention heads, total parameters = $340\text{M}$. Feed-forward filter sizes are $4H$.

Input and Output Representations

Inputs can represent a single text sequence or a pair of sentences packed together. Embeddings are constructed by summing:

1. WordPiece token embeddings (30,000 token vocabulary).
2. Segment embeddings (learned embedding identifying whether a token is from sentence $A$ or sentence $B$).
3. Learned positional embeddings.

Every sequence begins with [CLS]. Its final hidden vector $C\in {\mathbb{R}}^H$ represents the sequence aggregate. Sentence pairs are separated by [SEP]. Final token hidden representations are denoted $T_i\in {\mathbb{R}}^H$.

Pre-training Objectives

1. Masked LM: Given masked token $i$, the representation $T_i$ is fed into a softmax over the vocabulary with cross-entropy loss: $${\mathcal{L}}_{\text{MLM}}=-\log P(w_i\mid T_i)$$
2. Next Sentence Prediction: The aggregate vector $C$ is fed into binary classification: $${\mathcal{L}}_{\text{NSP}}=-\log P(\text{IsNext}\mid C)$$ The overall pre-training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction likelihood.

Fine-tuning Adaptations

• Classification (GLUE): Adding a projection matrix $W\in {\mathbb{R}}^{K\times H}$, loss is computed as $\log (\text{softmax}(C{W}^T))$.
• Span Extraction (SQuAD v1.1): Learning start vector $S\in {\mathbb{R}}^H$ and end vector $E\in {\mathbb{R}}^H$. The probability that token $i$ is the start of the answer span is: $$P_i=\frac{e^{S\cdot T_i}}{{\sum }_j{e}^{S\cdot T_j}}$$ The score for span $(i,j)$ is $S\cdot T_i+E\cdot T_j$, maximized over $j\ge i$.
• Unanswerable Questions (SQuAD v2.0): Extended by allowing the [CLS] token to serve as the null answer span, scoring $s_{\text{null}}=S\cdot C+E\cdot C$, and predicting a non-null answer only when ${\hat{s}}_{i,j}>s_{\text{null}}+\tau$.

Results

BERT was evaluated across 11 NLP tasks:

• GLUE Leaderboard: ${\text{BERT}}_{\text{LARGE}}$ achieved an average GLUE score of 80.5% (a 7.7% absolute improvement over the previous state of the art) and 86.7% accuracy on MNLI (a 4.6% absolute gain).
• SQuAD v1.1: ${\text{BERT}}_{\text{LARGE}}$ achieved 84.1 EM and 90.9 F1 on Dev as a single model; an ensemble with TriviaQA pre-training reached 87.4 EM and 93.2 F1 on the Test set, outperforming human performance (82.3 EM / 91.2 F1).
• SQuAD v2.0: ${\text{BERT}}_{\text{LARGE}}$ reached 80.0 EM and 83.1 F1 on the Test set, outperforming the previous top system by +5.1 F1.
• SWAG: ${\text{BERT}}_{\text{LARGE}}$ scored 86.3% accuracy on Test, outperforming OpenAI GPT (78.0%) and human expert baseline (85.0%).
• Ablation Findings: Removing NSP degraded performance across MNLI (-0.5%), QNLI (-3.5%), and SQuAD F1 (-0.6%). Replacing MLM with a standard Left-to-Right (LTR) LM caused large drops on SQuAD (-10.7 F1) and MRPC (-9.2% accuracy).

Limitations and open questions

• Pre-train / Fine-tune Discrepancy: The artificial [MASK] token never appears during fine-tuning, requiring heuristic replacement rules (80/10/10) to mitigate mismatch.
• Pre-training Efficiency: Because BERT only predicts 15% of tokens per batch rather than 100% of tokens as in standard autoregressive models, it requires more pre-training steps (1,000,000 steps with 128,000 tokens/batch) to converge.
• Quadratic Attention Bottleneck: Standard Transformer attention scales quadratically ($O(N^2)$) with sequence length, making long sequences expensive and requiring a two-phase pre-training strategy (90% steps at sequence length 128, 10% steps at 512).
• Text Generation Inapplicability: As a bidirectional encoder, BERT is directly structured for representation and discriminative downstream tasks, and cannot directly perform autoregressive left-to-right generation without architectural modifications.