live-coding-academy_ ⌕ search
home/ modules/ steering-and-accelerating-diffusion-sampling

Steering and Accelerating Diffusion Sampling

L4Performance 6h K · AI & real-time generative AV 17 atoms

Learning objectives

  • learner can steer generation with classifier, classifier-free, and residual guidance
  • learner can cut sampling steps with DDIM, progressive distillation, consistency, LCM-LoRA, and score-based views
  • learner can situate cascaded diffusion, latent diffusion, DiT backbones, and cosine schedules in the design space

Capstone — one whole task that evidences the objectives

Produce a decision guide for fast, controllable diffusion sampling: implement or spec classifier-free (and residual) guidance, compare DDIM/consistency/progressive-distillation/LCM-LoRA for few-step inference, position score-based modeling and cascaded/latent diffusion and the DiT backbone, and recommend a config for real-time-capable generation with justified quality/speed trade-offs.

Prerequisite modules

For a live visualist, diffusion is only useful if it keeps up with the music: an AV set feeding audio-reactive prompts into a StreamDiffusion rig needs frames in tens of milliseconds, not the thousand-step crawl of vanilla DDPM. This module builds toward exactly that judgment call — a decision guide that picks a guidance mechanism and a few-step sampler for a real-time-capable rig, and defends the quality/speed trade-offs.

The arc runs from steering to speed to positioning. You start supported: implement classifier-free guidance on a toy conditional model, using the classifier-guided predecessor as a foil for why CFG won (no second network, no adversarial classifier exploits). Then make it cheap — “Residual CFG approximates classifier-free guidance at near-zero extra cost” is the JIT pointer when the doubled UNet pass blows your frame budget. Next, attack step count: “DDIM makes diffusion sampling deterministic” is the gateway (determinism is what progressive distillation and consistency models both build on), and “Fusing LCM-LoRA into an SD model enables 2–4 step inference” is the hands-on route on a stock SD 1.5 checkpoint. Finally you zoom out, placing score-based modeling, cascaded vs. latent pipelines, DiT backbones, and cosine schedules on the map before writing the unsupported capstone guide.

Every required atom gates the capstone: each guidance variant, each few-step method, and each design-space landmark is a row or axis in the guide — omit one and the recommendation loses its justification. The supporting atoms deepen rather than gate: the DDPM follow-up landscape and variance learning give historical texture, while the U-Net backbone, linear attention, and xformers atoms explain where the compute actually goes when your recommended config hits real hardware.

Atoms in this module

Required — these gate the capstone

Classifier-guided diffusion steers generation by adding classifier-score gradients to the noise prediction
Concept L3 Craft K
Classifier-free guidance combines conditional and unconditional model outputs at inference to steer generation without a separate classifier
Concept L3 Craft K
Residual CFG approximates classifier-free guidance at near-zero extra cost by recycling a stored noise residual
Concept L3 Craft K
DDIM makes diffusion sampling deterministic by setting the stochasticity parameter eta to zero, enabling far fewer steps
Concept L3 Craft K
Progressive distillation halves the required sampling steps by training a student to match two teacher steps in one
Concept L4 Performance K
Consistency models map any point on a diffusion trajectory directly to the trajectory origin, enabling single-step generation
Concept L4 Performance K
Fusing LCM-LoRA into an SD model enables 2–4 step diffusion inference without retraining the base model
Concept L3 Craft K
Score-based models learn the gradient of the log data density and generate samples via Langevin dynamics
Concept L4 Performance K
Cascaded diffusion chains models at increasing resolutions, using noise-conditioning augmentation between stages
Concept L4 Performance K
Diffusion Transformers replace the U-Net backbone with a Vision Transformer operating on patchified latents
Concept L4 Performance K
A cosine noise schedule gives gentler early-stage transitions than the linear schedule, improving likelihood
Fact L4 Performance K
Latent diffusion runs the denoising process in a compressed latent space instead of pixel space, cutting compute cost
Concept L2 First instrument K

Supporting — enrichment, not gating

Key DDPM follow-up works include improved variance learning, cascaded generation, classifier guidance, and classifier-free guidance
Fact L3 Craft K
Learning the reverse-process variance as an interpolation between two fixed endpoints improves likelihood
Fact L4 Performance K
U-Net's symmetric downsampling-upsampling structure with skip connections makes it a standard diffusion backbone
Concept L3 Craft K
Linear attention scales O(n) in sequence length — making it practical for image feature maps without the O(n^2) cost of full attention
Concept L3 Craft K
xformers memory-efficient attention reduces VRAM usage in diffusion UNets at modest speed cost
Concept L3 Craft K

Part of curricula