Deep Residual Learning for Image Recognition

Kaiming He; Xiangyu Zhang; Shaoqing Ren; Jian Sun

Papers · arXiv 1512.03385

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Published 2015-12-10cs.CV arXiv PDF

Read with translation and AI

Summary in three lines

🎯 Deeper plain neural networks suffer from optimization degradation, where increasing depth yields higher training error despite non-vanishing gradients.
🔧 Reformulate stacked layers to fit a residual mapping $F (x) = H (x) - x$ using parameter-free identity shortcut connections $y = F (x) + x$ .
📈 Enabled training of networks up to 152 layers on ImageNet (achieving 3.57% top-5 ensemble error) and won 1st place in ILSVRC and COCO 2015 competitions.

Abstract

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Reading guide

Problem and motivation

Network depth is critical for visual recognition representations, but stacking more layers exposes a fundamental degradation problem: as depth increases, accuracy saturates and then degrades rapidly. Crucially, this degradation is not caused by overfitting—the deeper networks produce higher training error than their shallower counterparts, despite techniques like Batch Normalization and normalized initialization ensuring that forward activations and backpropagated gradients do not vanish.

In theory, a deeper model has an identity-mapping solution by construction: copy the learned shallower model and set extra layers to identity transformations. However, conventional solvers fail to find this solution when learning unreferenced nonlinear mappings directly.

Key ideas

Residual Learning Framework: Instead of optimizing stacked layers to approximate a desired mapping $H (x)$ , explicitly let the layers approximate the residual mapping $F (x) := H (x) - x$ . The original mapping is recovered as $F (x) + x$ .
Identity Shortcut Connections: Realize $F (x) + x$ via feedforward shortcut connections that skip one or more layers. Identity shortcuts require zero extra parameters and add virtually no computational overhead.
Easier Optimization & Preconditioning: If identity mappings are optimal, optimizers can simply drive residual weights toward zero. In practice, residual function responses have small magnitudes, indicating that identity connections act as effective preconditioning.
Bottleneck Architecture: For very deep variants (ResNet-50, ResNet-101, ResNet-152), each 2-layer residual unit is replaced with a 3-layer bottleneck block ( $1 \times 1$ , $3 \times 3$ , $1 \times 1$ convolutions), reducing computation while preserving representational capacity.

Method

For a residual building block where input and output dimensions match: $y = F (x, {W_{i}}) + x$ where $x$ and $y$ are input and output vectors, and $F = W_{2} σ (W_{1} x)$ for a 2-layer unit (with $σ$ denoting ReLU, and biases omitted for simplicity). The element-wise addition is performed before applying the second ReLU activation $σ (y)$ .

When feature map dimensions change (e.g., during spatial downsampling or channel doubling): $y = F (x, {W_{i}}) + W_{s} x$ Here, $W_{s}$ denotes either:

Option A: Zero-padding extra dimensions along with identity shortcuts (stride 2 for subsampling), which introduces 0 extra parameters.
Option B: A $1 \times 1$ linear convolution projection $W_{s}$ used only when increasing dimensions, while using pure identity connections elsewhere.
Option C: Linear projection shortcuts $W_{s}$ for all residual blocks.

The paper adopts Option B for ImageNet models (ResNet-50/101/152) because Option C adds parameter bloat without meaningful accuracy gains, and pure identity shortcuts are critical for keeping bottleneck structures efficient.

Results

ImageNet Classification:
- ResNet-34 outperforms plain-34 by 3.5% top-1 error (25.03% vs. 28.54% 10-crop val error) and resolves the training degradation.
- ResNet-152 achieves a single-model top-5 validation error of 4.49% (11.3 billion FLOPs, lower complexity than VGG-16 at 15.3 billion FLOPs).
- An ensemble of ResNets achieved 3.57% top-5 error on the test set, winning 1st place in the ILSVRC 2015 classification challenge.
CIFAR-10: ResNet depths scale smoothly through 20, 32, 44, 56, and 110 layers, with ResNet-110 reaching 6.43% test error. An experimental 1202-layer network was trained with zero optimization failure ( $< 0.1%$ training error).
Transfer & Downstream Tasks: Swapping VGG-16 for ResNet-101 in Faster R-CNN improved MS COCO mAP@[.5, .95] from 21.2% to 27.2% (a 28% relative increase) and PASCAL VOC 2007 test mAP from 73.2% to 76.4%. Deep residual networks also won 1st place in ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Limitations and open questions

Overfitting on Ultra-Deep Architectures: On CIFAR-10, the 1202-layer model showed higher test error (7.93%) than ResNet-110 (6.43%) despite reaching $< 0.1%$ training error, due to lack of regularization on the small dataset.
Theoretical Cause of Degradation: The paper experimentally shows that gradient vanishing is not the issue, conjecturing that plain networks suffer from exponentially low convergence rates, but leaves theoretical proofs of the optimization dynamics to future work.
Single-Layer Residuals: The residual formulation provides no noticeable benefit when $F$ consists of only a single layer (reducing to $y = W_{1} x + x$ ).

Key terms

degradation problem: The counterintuitive phenomenon where adding more layers to a deep neural network leads to higher training error, despite no presence of vanishing/exploding gradients.
residual learning: Reformulating stacked neural network layers to explicitly learn a residual function $F (x) = H (x) - x$ relative to their inputs rather than fitting an unreferenced mapping $H (x)$ directly.
identity shortcut connection: A parameter-free skip connection that passes the input vector or feature map unchanged and adds it element-wise to the output of stacked convolutional layers.
projection shortcut: A shortcut connection using a linear transformation matrix or $1 \times 1$ convolution ( $W_{s}$ ) to match dimensions when input and output channel depths or spatial sizes differ.
bottleneck design: A 3-layer residual building block comprising $1 \times 1$ , $3 \times 3$ , and $1 \times 1$ convolutions, where $1 \times 1$ layers compress and restore feature dimensions to reduce computational complexity.
plain network: A baseline convolutional network architecture that simply stacks standard convolutional and pooling layers without shortcut or residual connections.

Questions

Why is the degradation problem not caused by vanishing gradients?

The networks were trained using Batch Normalization and normalized weight initialization, which were verified to maintain non-zero signal variances in forward propagation and healthy gradient norms in backpropagation. Furthermore, deeper plain networks like plain-34 still converged to non-trivial solutions, but converged with higher training error than shallower ones.

What is the difference between ResNet identity shortcuts and Highway Networks?

Highway networks use data-dependent, parameterized gating functions that can close and turn layers non-residual, whereas ResNet shortcuts are parameter-free identity mappings that are never closed, guaranteeing residual learning throughout.

When is projection used instead of an identity shortcut?

Projection (using $1 \times 1$ convolutions) is primarily used when input and output feature dimensions do not match (e.g., when doubling channels and halving spatial resolution with stride 2). For matching dimensions, identity mapping is sufficient, more economical, and avoids parameter inflation in bottleneck units.

Why did ResNet-1202 perform worse on CIFAR-10 than ResNet-110?

ResNet-1202 suffered from overfitting rather than optimization failure. It successfully reached $< 0.1%$ training error, but its 19.4M parameters were unnecessarily large for the small CIFAR-10 dataset without additional regularization techniques like dropout.

The overview on this page was written by AI from the paper and may contain mistakes. Read the paper for the details.

Read with translation and AI