live-coding-academy_ ⌕ search
home/ modules/ diffusion-theory-and-objectives

Diffusion Theory: Objectives, Schedules, and Backbones

L3Craft 6h K · AI & real-time generative AV 21 atoms

Learning objectives

  • learner can derive the noise-prediction objective and the ELBO/VLB decomposition into per-timestep KL terms
  • learner can explain the nice property, fixed vs learned variance, and the U-Net time-conditioning assembly
  • learner can compare diffusion to GANs/VAEs on the tractability-vs-flexibility axis

Capstone — one whole task that evidences the objectives

Write a rigorous explainer that derives the DDPM training objective: show the forward Markov chain and Gaussian noise, the nice property / closed-form noising, the noise-prediction reparameterization, the ELBO decomposing into analytically tractable per-timestep KL terms, the fixed-variance choice and time-conditioned U-Net assembly, and why diffusion trades sampling speed for tractability and mode coverage vs GANs/VAEs.

Prerequisite modules

Diffusion models now sit behind most neural audio and visual generation a live coder is likely to lean on — sample-pack synthesis, texture and visual generation for the rig, and the text-to-audio tools reshaping electronic music production. Using them well (choosing checkpoints, debugging training runs, reading papers about samplers) requires actually owning the DDPM math, not just running pipelines. This module builds toward one whole task: writing a rigorous explainer that derives the DDPM training objective end to end.

The arc starts from the picture-level definition — a fixed forward Markov chain that corrupts data with scheduled Gaussian noise, and a learned reverse chain that approximates the intractable denoising posterior. With that frame in place, the learner works a supported first exercise: deriving the closed-form noising formula, using “the ‘nice property’ lets you sample any diffusion timestep directly” and “the reparameterization trick lets you sample a noisy x_t at any timestep” as JIT how-to pointers. From there the derivation deepens: the ELBO/VLB decomposition into per-timestep Gaussian KL terms, the noise-prediction reparameterization, the fixed-variance simplification, and the empirically simplified uniform-MSE loss. A parallel thread assembles the backbone — sinusoidal time embeddings, FiLM-conditioned ResNet blocks, and the full U-Net — so “time-conditioned network” is concrete, not hand-waved. The capstone then demands the whole derivation unsupported, closing with the tractability-vs-flexibility argument against GANs, VAEs, and flows.

Every required atom is a link in that derivation chain: skip one and the explainer has a hole the capstone rubric will find. The supporting atoms — cosine schedules, learned-variance interpolation, the follow-up landscape, group normalization — enrich the story with “what came next” and implementation nuance without gating it. The two closed-form-sampling drills deserve repetition until automatic, since they recur in every diffusion derivation the learner will ever read.

Atoms in this module

Required — these gate the capstone

Diffusion models learn to reverse a Markov chain that progressively corrupts data with Gaussian noise
Concept L3 Craft K
The forward diffusion process adds scheduled Gaussian noise over T steps until the signal becomes isotropic noise
Concept L3 Craft K
The reverse diffusion process is a neural network that approximates the intractable denoising posterior at each timestep
Concept L3 Craft K
DDPM's training objective is to predict the noise added to an image rather than the original image directly
Principle L3 Craft K
The 'nice property' lets you sample any diffusion timestep directly from the original image without iterating the forward process
Concept L3 Craft K
The reparameterization trick lets you sample a noisy x_t at any timestep t directly without simulating the full chain
Fact L4 Performance K
DDPM trains a network to predict the added noise epsilon rather than the clean sample or the reverse mean
Concept L4 Performance K
The DDPM ELBO decomposes into per-timestep KL divergences between Gaussians, each computable as an L2 loss
Principle L3 Craft K
The diffusion training objective decomposes into per-timestep KL divergence terms plus a reconstruction term
Concept L4 Performance K
Every KL term in the diffusion VLB compares two Gaussians and is therefore analytically tractable
Fact L4 Performance K
DDPM keeps the reverse-process variance fixed and only learns the mean, simplifying the training target
Concept L3 Craft K
Dropping per-timestep loss weights and training with uniform MSE on noise prediction improves DDPM in practice
Fact L4 Performance K
The denoising U-Net uses sinusoidal position embeddings to communicate the current noise level (timestep) to every layer
Concept L3 Craft K
DDPM ResNet blocks inject the timestep embedding via scale-and-shift (FiLM-style) conditioning
Procedure L3 Craft K
The DDPM U-Net assembles encoder, bottleneck, and decoder stages using ModuleList, with skip connections via concatenation
Procedure L3 Craft K
Diffusion models are attractive because they are simultaneously analytically tractable and flexible
Principle L3 Craft K
Diffusion models trade sampling speed for stable training and broad mode coverage relative to GANs and VAEs
Concept L3 Craft K

Supporting — enrichment, not gating

A cosine noise schedule gives gentler early-stage transitions than the linear schedule, improving likelihood
Fact L4 Performance K
Learning the reverse-process variance as an interpolation between two fixed endpoints improves likelihood
Fact L4 Performance K
Key DDPM follow-up works include improved variance learning, cascaded generation, classifier guidance, and classifier-free guidance
Fact L3 Craft K
Group normalization divides channels into groups and normalizes within each group, working well with small batch sizes unlike batch normalization
Concept L3 Craft K

Part of curricula

Unlocks — modules that require this one