Score-based models learn the gradient of the log data density and generate samples via Langevin dynamics

The score function of a distribution q(x) is the gradient of its log-density: grad_x log q(x). A score network s_theta(x) is trained to approximate this gradient using score matching or denoising score matching. To generate samples, Langevin dynamics iteratively nudges x toward higher-density regions by following the score plus a little noise. Song and Ermon (2019) added noise-conditioned score networks (NCSN): because most data lies on a low-dimensional manifold, scores are unreliable in low-density regions, so perturbing the data with noise at multiple levels lets the score network be trained across the whole space. The connection to DDPM: the DDPM noise-prediction network is proportional to a score estimator at each noise level, s_theta(x_t,t) approx -epsilon_theta(x_t,t)/sqrt(1-alpha_bar_t), unifying the two frameworks.

Examples

NCSN trains s_theta(x,t) where t indexes the noise level. At inference, Langevin sampling runs at decreasing noise levels, analogous to DDPM’s reverse chain.

Assessment

Define the score function and explain how Langevin dynamics uses it to generate samples. Then explain why noise conditioning is necessary for stable score estimation.