Neural Machine Translation by Jointly Learning to Align and Translate

Reading guide

Problem and motivation

Standard neural machine translation (NMT) architectures, such as the basic RNN Encoder–Decoder, compress the entirety of a variable-length source sentence into a single fixed-length vector $c$. From this fixed context vector, the decoder generates the target translation. This creates an informational bottleneck: as sentence length increases, compressing all source semantics into a fixed-size representation becomes increasingly difficult, causing translation quality (BLEU score) to deteriorate significantly for longer sentences.

Key ideas

• Sequence of Annotations Instead of a Single Vector: Instead of forcing the encoder to compress the entire sentence into one vector, the model preserves a sequence of hidden state vectors (annotations) across the source sentence using a Bidirectional Recurrent Neural Network (BiRNN).
• Soft Alignment Mechanism (Attention): Rather than treating alignment as a discrete or latent variable requiring heuristic extraction, the decoder calculates a continuous "soft alignment" weight over the source annotations at each output step.
• Differentiable End-to-End Training: Because the alignment is parameterized as a small feedforward neural network, the entire system—encoder, attention mechanism, and decoder—is trained jointly via backpropagation.

Method

1. Encoder (Bidirectional RNN)

The source sequence $\mathbf{x}=(x_1,\dots ,x_{T_x})$ is processed by a forward RNN $\overrightarrow{f}$ and a backward RNN $\overleftarrow{f}$ using gated hidden units: $${\overrightarrow{h}}_j=\overrightarrow{f}(x_j,{\overrightarrow{h}}_{j-1}),{\overleftarrow{h}}_j=\overleftarrow{f}(x_j,{\overleftarrow{h}}_{j+1})$$ The annotation for source word $x_j$ concatenates both directions: $$h_j={\left[{\overrightarrow{h}}_j^{\top };{\overleftarrow{h}}_j^{\top }\right]}^{\top }$$

2. Decoder with Soft Attention

To generate target word $y_i$, the decoder calculates a distinct context vector $c_i$ as a weighted sum of encoder annotations: $$c_i=\sum _{j=1}^{T_x}{\alpha }_{ij}{h}_j$$ where the alignment weight ${\alpha }_{ij}$ is computed using a softmax over energy scores $e_{ij}$: $${\alpha }_{ij}=\frac{\exp (e_{ij})}{{\sum }_{k=1}^{T_x}\exp (e_{ik})}$$ The energy score $e_{ij}$ represents how well decoder state $s_{i-1}$ aligns with source annotation $h_j$ and is parameterized as a single-layer feedforward network: $$e_{ij}=a(s_{i-1},h_j)=v_a^{\top }\tanh (W_a{s}_{i-1}+U_a{h}_j)$$ Here $s_{i-1}$ is the decoder hidden state, updated via: $$s_i=f(s_{i-1},y_{i-1},c_i)$$ The next-word probability distribution is conditioned on the current state, previous token, and the dynamic context: $$p(y_i\mid y_1,\dots ,y_{i-1},\mathbf{x})=g(y_{i-1},s_i,c_i)$$

Results

On the WMT '14 English-to-French benchmark (news-test-2014, 3,003 sentences):

• Robustness to Length: While the basic encoder-decoder (RNNencdec-50) suffers a severe BLEU drop as sentence length exceeds 30 words, RNNsearch-50 maintains consistent BLEU scores even beyond 50 words.
• Translation Quality:
  • On all test sentences, RNNsearch-50 scores 26.75 BLEU (and 28.45 BLEU when trained longer as RNNsearch-50*), outperforming RNNencdec-50 (17.82 BLEU).
  • On sentences containing no unknown words (No UNK), RNNsearch-50* reaches 36.15 BLEU, outperforming the conventional phrase-based translation baseline Moses (35.63 BLEU), despite Moses using additional monolingual data (418M words).
• Alignment Visualizations: Heatmaps of ${\alpha }_{ij}$ reveal intuitive linguistic alignments, handling phrase reorderings (e.g., English noun-adjective order into French adjective-noun order) and multi-word phrases naturally.

Limitations and open questions

• Computational Complexity: Computing $e_{ij}$ requires $O(T_x\times T_y)$ evaluations per sentence pair, which is computationally expensive for very long sequences.
• Handling Rare/Unknown Words: The model relies on a fixed vocabulary (30,000 most frequent words), mapping out-of-vocabulary words to [UNK], which harms performance on open test sets.
• Monolingual Data Utilization: Unlike phrase-based statistical systems that easily leverage massive target-side monolingual corpora via separate language models, purely end-to-end NMT was limited to parallel data.