Latent diffusion runs the denoising process in a compressed latent space instead of pixel space, cutting compute cost

Pixel-space diffusion operates on full-resolution images and demands enormous compute. Latent Diffusion Models (LDM; Rombach et al. 2022, the basis of Stable Diffusion) run the entire iterative denoising on a compressed representation instead. A pretrained autoencoder’s encoder maps a high-resolution image into a small 2D latent; diffusion trains and runs entirely in that compact space; only at the very end does the decoder reconstruct a pixel image in a single pass. Because the latent has far fewer dimensions than the pixel grid, each denoising step is dramatically cheaper — which is what lets high-resolution generation run on consumer hardware where pixel-space diffusion could not. The architecture splits labor: the autoencoder performs perceptual compression (removing pixel-level redundancy), while the diffusion model handles semantic generation; the autoencoder must be trained to reconstruct images faithfully so quality survives the compression. Flexible conditioning (class labels, text, semantic maps) is injected into the latent-space U-Net via cross-attention, each modality paired with its own encoder.

Examples

A 512×512×3 pixel image = 786,432 values, compressed to a 4×64×64 = 16,384-value latent. Many denoising steps run on the small latent; the decoder runs once to paint the final 512×512 image. Text conditioning enters the U-Net through cross-attention.

Assessment

Explain why running diffusion in latent space is more efficient than in pixel space. Describe the two-stage LDM architecture and identify which stage handles perceptual vs. semantic work, the role of the encoder and decoder, and what the autoencoder must be trained to do for quality to survive compression.