The DDPM ELBO decomposes into per-timestep KL divergences between Gaussians, each computable as an L2 loss

DDPM’s loss function is derived from the ELBO (variational lower bound) of the log-likelihood, treating the model as a hierarchical VAE. The ELBO decomposes into a sum of terms L_0 + L_1 + … + L_T. Each intermediate term L_t is the KL divergence between two Gaussians (the true posterior q(x_{t-1}|x_t,x_0) and the model’s reverse p_theta(x_{t-1}|x_t)). Since KL between Gaussians has a closed form, each L_t reduces to an L2 (mean squared error) between the means. After reparametrizing to predict noise, this further simplifies to MSE between true and predicted noise. This derivation justifies using a simple MSE/Huber loss for training.

Examples

The final simplified objective: ||epsilon - epsilon_theta(sqrt(alpha_bar_t)*x_0 + sqrt(1-alpha_bar_t)*epsilon, t)||^2. This is what p_losses() computes.

Assessment

Explain the chain: ELBO → per-timestep KL → L2 on means → MSE on noise. Why can each term be written as an L2 loss?