Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Reading guide

Problem and motivation

During deep network optimization with SGD, changes in earlier layer parameters alter the distribution of inputs to subsequent layers. The authors term this phenomenon Internal Covariate Shift.

Internal covariate shift creates two primary challenges:

1. Sub-networks must continuously adapt to shifting activation distributions.
2. Activations readily drift into the saturating regimes of nonlinearities (such as sigmoid functions), causing vanishing gradients and requiring small learning rates and meticulous parameter initialization.

While whitening (decorrelating and standardizing) layer inputs across the entire dataset could stabilize distributions, computing full covariance matrices and their inverse square roots at every parameter update is computationally prohibitive and can cause gradients to blow up if normalization is not integrated into backpropagation.

Key ideas

• Differentiable Normalization Layer: Instead of standardizing activations outside optimization, normalization is integrated directly into the model architecture, ensuring backpropagation accounts for the dependencies between layer activations and mini-batch statistics.
• Per-Dimension Normalization: Rather than full joint whitening, each scalar feature dimension is normalized independently to zero mean and unit variance.
• Restoring Representational Capacity: To ensure normalization does not destroy the expressive capability of a layer (such as forcing a sigmoid into purely linear behavior), learnable parameters $\gamma$ (scale) and $\beta$ (shift) are introduced to allow the layer to recover the identity transformation if needed.
• Deterministic Inference: During training, statistics are calculated over each mini-batch $\mathcal{B}$. At test time, fixed population statistics (computed via running averages or across training batches) replace mini-batch statistics, ensuring deterministic single-sample inference.
• Parameter Scale Invariance: Scale transformations of layer weights satisfy $\text{BN}((aW)\mathrm{u})=\text{BN}(W\mathrm{u})$, meaning larger weights lead to smaller effective gradients, preventing gradient explosion.

Method (with the key equations)

For a mini-batch $\mathcal{B}=\{x_{1\dots m}\}$ corresponding to a single activation dimension:

1. Compute the mini-batch mean: $${\mu }_{\mathcal{B}}\leftarrow \frac{1}{m}\sum _{i=1}^m{x}_i$$
2. Compute the mini-batch variance: $${\sigma }_{\mathcal{B}}^2\leftarrow \frac{1}{m}\sum _{i=1}^m(x_i-{\mu }_{\mathcal{B}}{)}^2$$
3. Normalize activations (with stability constant $ϵ$): $${\hat{x}}_i\leftarrow \frac{x_i-{\mu }_{\mathcal{B}}}{\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}}$$
4. Scale and shift using learnable parameters $\gamma$ and $\beta$: $$y_i\leftarrow \gamma {\hat{x}}_i+\beta \equiv {\text{BN}}_{\gamma ,\beta }(x_i)$$

Application to Layers

• Placement: Applied immediately before nonlinearities: $z=g(\text{BN}(Wu))$. The original layer bias $b$ is omitted because mean subtraction cancels it (its role is subsumed by $\beta$).
• Convolutional Layers: To preserve the translation equivariance property, normalization is applied per feature map rather than per spatial activation. For batch size $m$ and spatial resolution $p\times q$, the effective mini-batch size becomes $m^{\prime }=m\cdot p\cdot q$, with a single pair of $(\gamma ,\beta )$ shared across each feature map.

Inference Transformation

At test time, the transformation becomes a static linear operation using population statistics $\mathrm{E}[x]$ and $\mathrm{Var}[x]$: $$y=\frac{\gamma }{\sqrt{\mathrm{Var}[x]+ϵ}}\cdot x+\left(\beta -\frac{\gamma \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x]+ϵ}}\right)$$

Results (with the most important numbers)

• MNIST with Sigmoids: Networks equipped with Batch Normalization retained stable activation distributions across training, accelerating convergence and reaching higher test accuracy compared to unnormalized networks.
• ImageNet Training Speed (Single Network):
  • Unnormalized Inception reached 72.2% validation accuracy after $31.0\times 10^6$ steps.
  • Baseline Batch-Normalized Inception reached 72.2% in $13.3\times 10^6$ steps.
  • With a 5x higher learning rate (BN-x5), it reached 72.2% in $2.1\times 10^6$ steps—a 14-fold reduction in required training steps, ultimately peaking at 73.0%.
  • With a 30x higher learning rate (BN-x30), the model reached 74.8% maximum accuracy after $6.0\times 10^6$ steps.
• Saturating Nonlinearities: While unnormalized Inception with sigmoid activations failed to train (staying near 0.1% chance accuracy), BN-x5-Sigmoid achieved 69.8% top-1 accuracy.
• ImageNet SOTA Ensemble: An ensemble of 6 batch-normalized networks achieved a top-5 validation error of 4.9% and a test error of 4.82% on the ILSVRC test server, outperforming human rater baselines.

Limitations and open questions

• Batch Size Dependency: Because normalization relies on mini-batch sample statistics, very small mini-batch sizes may degrade the quality of variance and mean estimations.
• Recurrent Networks: The paper tests feedforward and convolutional models, identifying recurrent neural networks (RNNs) as an untested open question where internal covariate shift may also be critical.
• Theoretical Underpinnings: The authors conjecture that Batch Normalization causes layer Jacobians to have singular values close to 1, but noted that precise theoretical verification of gradient behavior remains open for future analysis.