Denoising Diffusion Probabilistic Models

Reading guide

Problem and motivation

Although diffusion probabilistic models were originally introduced by Sohl-Dickstein et al. (2015) based on nonequilibrium thermodynamics, early iterations lagged significantly behind GANs and autoregressive architectures in visual fidelity. They were generally perceived as slow, difficult to tune, and unproven on complex, high-resolution natural image datasets. The authors set out to re-examine diffusion models by establishing a formal connection between their variational objectives and score matching with Langevin dynamics, demonstrating that with appropriate parameterization and objectives, diffusion models can achieve state-of-the-art perceptual sample quality.

Key ideas

• Noise-Prediction Parameterization: Rather than predicting the posterior mean ${\tilde{\mathbf{\mu }}}_t$ directly, the reverse process neural network ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$ is trained to predict the unscaled standard Gaussian noise vector $\mathbf{ϵ}$ added during the forward step.
• Connection to Denoising Score Matching: Denoising diffusion matches denoising score matching across multiple noise scales, while the learned reverse chain acts as an annealed Langevin dynamics sampler derived via variational inference.
• Simplified Loss Function ($L_{\mathrm{simple}}$): Discarding the strict variance-dependent weighting from the variational lower bound produces an unweighted MSE loss that down-weights low noise levels ($t\approx 1$), forcing the model to focus on perceptually critical coarse structures.
• Progressive Lossy Compression View: Analysis reveals that more than half of the model's lossless codelength represents imperceptible high-frequency pixel noise, meaning diffusion models implicitly optimize rate-distortion by decoding global content first before filling in fine texture.

Method

The forward process incrementally corrupts data ${\mathbf{x}}_0$ over $T=1000$ steps with a fixed variance schedule ${\beta }_1,\dots ,{\beta }_T$: $$q({\mathbf{x}}_t|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t;\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I})$$ Using ${\alpha }_t=1-{\beta }_t$ and ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$, intermediate latents can be directly sampled in closed form: $$q({\mathbf{x}}_t|{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_t;\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0,(1-{\bar{\alpha }}_t)\mathbf{I})$$

The reverse chain $p_{\theta }({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)=\mathcal{N}({\mathbf{x}}_{t-1};{\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t),{\sigma }_t^2\mathbf{I})$ parameterizes the mean as: $${\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t)=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)$$ where ${\sigma }_t^2$ is fixed to either ${\beta }_t$ or ${\tilde{\beta }}_t=\frac{1-{\bar{\alpha }}_{t-1}}{1-{\bar{\alpha }}_t}{\beta }_t$.

The network is optimized using the simplified surrogate loss: $$L_{\mathrm{simple}}(\theta )={\mathbb{E}}_{t,{\mathbf{x}}_0,\mathbf{ϵ}}\left[{\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }(\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},t)\Vert }^2\right]$$ where $t\sim \text{Uniform}(\{1,\dots ,T\})$ and $\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$.

Sampling proceeds backward from ${\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})$ using: $${\mathbf{x}}_{t-1}=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{1-{\alpha }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)+{\sigma }_t\mathbf{z}$$ with $\mathbf{z}\sim \mathcal{N}(0,\mathbf{I})$ for $t>1$, and $\mathbf{z}=0$ when $t=1$.

Results

• CIFAR-10: Achieved an unconditional Inception Score (IS) of $9.46\pm 0.11$ and an FID of $3.17$ using $L_{\mathrm{simple}}$, outperforming contemporaries like SNGAN (FID 21.7) and NCSN (FID 25.32).
• High-Resolution Synthesis: Successfully generated realistic $256\times 256$ images on LSUN (Bedroom FID 4.90, Church FID 7.89, Cat FID 19.75) and CelebA-HQ.
• Likelihood vs. Quality Trade-off: While training with the exact variational bound $L$ gave better negative log-likelihoods ($\le 3.70$ bits/dim test) than $L_{\mathrm{simple}}$ ($\le 3.75$ bits/dim test), $L_{\mathrm{simple}}$ achieved vastly superior FID scores (3.17 vs. 13.51).

Limitations and open questions

• Sampling Speed: Sampling requires evaluating the neural network $T=1000$ sequential times per image batch, making inference orders of magnitude slower than GANs or single-pass flow models.
• Competitive Log-Likelihoods: The models do not match the raw lossless test likelihoods of pure autoregressive models like Sparse Transformers (which achieved $2.80$ bits/dim on CIFAR-10).
• Fixed Variance Scheduling: The forward variance schedule ${\beta }_t$ was fixed linearly rather than learned or adaptively scaled, leaving open whether non-linear schedules or learned continuous dynamics could yield further improvements.