High-Resolution Image Synthesis with Latent Diffusion Models

Reading guide

Problem and motivation

Diffusion probabilistic models generate high-fidelity, mode-covering imagery by inverting a forward noise process with a sequence of denoising autoencoders. However, training and evaluating them directly in high-dimensional RGB pixel space is computationally demanding. Optimization often takes hundreds of GPU days (e.g., 150 to 1000 V100 days), and multi-step sequential sampling can take days for standard evaluation runs. Because likelihood-based generative modeling spends excessive capacity on high-frequency imperceptible image details before learning semantic structures, direct pixel-space evaluations lead to wasteful gradient and function evaluations.

Key ideas

• Two-stage decomposition: Separate perceptual compression (eliminating imperceptible details) from semantic generative modeling (learning image structure and composition).
• Latent Diffusion Models (LDMs): Train denoising diffusion models inside the latent space of an autoencoder trained with perceptual and adversarial losses.
• Moderate spatial compression: Unlike autoregressive models that need heavy spatial reduction (e.g., $f=16$ or $f=32$), LDMs maintain spatial inductive bias with a 2D convolutional UNet, allowing moderate downsampling factors like $f=4$ or $f=8$ that preserve fine details.
• Universal cross-attention conditioning: Generalize UNets to multimodal inputs (text descriptions, bounding boxes, semantic maps) using domain-specific encoders mapped into intermediate network layers via cross-attention.

Method

1. Perceptual Image Compression: An encoder $\mathcal{E}$ maps an image $x\in {\mathbb{R}}^{H\times W\times 3}$ to a lower-dimensional latent representation $z=\mathcal{E}(x)\in {\mathbb{R}}^{h\times w\times c}$, where downsampling factor $f=H/h=W/w=2^m$. A decoder reconstructs $\tilde{x}=\mathcal{D}(z)$. The model is regularized using either a weak KL-divergence penalty towards $\mathcal{N}(0,\mathbb{I})$ (KL-reg.) or vector quantization (VQ-reg.) absorbed into the decoder.

2. Generative Latent Modeling: A time-conditional UNet ${ϵ}_{\theta }(z_t,t)$ is trained on the latents using the reweighted denoising objective: $$L_{LDM}:={\mathbb{E}}_{\mathcal{E}(x),ϵ\sim \mathcal{N}(0,1),t}\left[\Vert ϵ-{ϵ}_{\theta }(z_t,t){\Vert }_2^2\right]$$

3. Cross-Attention Conditioning: For conditioning input $y$ (e.g., text, classes, layouts), a domain encoder ${\tau }_{\theta }(y)\in {\mathbb{R}}^{M\times d_{\tau }}$ is introduced. Intermediate UNet features ${\phi }_i(z_t)\in {\mathbb{R}}^{N\times d_{ϵ}^i}$ interact with conditioning via cross-attention: $$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d}}\right)\cdot V$$ where $Q=W_Q^{(i)}{\phi }_i(z_t)$, $K=W_K^{(i)}{\tau }_{\theta }(y)$, and $V=W_V^{(i)}{\tau }_{\theta }(y)$. The conditional objective is: $$L_{LDM}:={\mathbb{E}}_{\mathcal{E}(x),y,ϵ\sim \mathcal{N}(0,1),t}\left[\Vert ϵ-{ϵ}_{\theta }(z_t,t,{\tau }_{\theta }(y)){\Vert }_2^2\right]$$

Results

• Perceptual compression trade-off: Downsampling factors $f=4$ and $f=8$ strike the best balance between reconstruction accuracy, synthesis quality, and throughput. On ImageNet, an FID gap of 38 points separates pixel-based LDM-1 from LDM-8 after 2M training steps under identical computational budgets.
• Unconditional generation: Achieved state-of-the-art FID of $5.11$ on CelebA-HQ ($256\times 256$), $4.98$ on FFHQ, $4.02$ on LSUN-Churches, and $2.95$ on LSUN-Bedrooms.
• Class-conditional ImageNet: LDM-4 with classifier-free guidance reached an FID of $3.60$ and Inception Score of $247.67$, outperforming ADM ($4.59$) while requiring fewer parameters (400M vs. 608M) and substantially less training compute (271 vs. 962 V100 days).
• Text-to-image: A 1.45B parameter model (LDM-KL-8-G) achieved an FID of $12.63$ on MS-COCO with 250 DDIM steps.
• Super-resolution and inpainting: LDM-4 set a new state-of-the-art inpainting FID of $1.50$ on Places $512\times 512$ (and $9.39$ on hard 40–50% masked images), while operating at over $2.7\times$ higher throughput than pixel-space diffusion.

Limitations and open questions

• Sampling latency: While significantly faster than pixel-based diffusion models, sequential multi-step sampling is still slower than single-pass GAN generators.
• Reconstruction bottleneck: Even with low downsampling ($f=4$), compression can lose fine-grained high-frequency pixel details, making it slightly suboptimal for ultra-precise pixel-critical tasks.
• Societal impact & safety: Lower compute barriers democratize creative tools but also ease creation of deepfakes and misuse, while potentially inheriting biases from uncurated web-scale training datasets.