Imagen conditions image generation on a frozen large language model, and scaling that encoder matters more than scaling the U-Net

Imagen (Saharia et al. 2022) shows that a generic frozen large language model makes a strong text encoder for text-to-image diffusion: instead of a CLIP text encoder it uses a frozen T5-XXL model to embed the prompt. A key empirical finding is that increasing the size of this text encoder improves image quality and image-text alignment more than increasing the size of the diffusion U-Net — the language model’s understanding of the prompt is the bottleneck, not the generator’s capacity. Imagen also introduces dynamic thresholding (per-step percentile clipping of the pixel prediction) to keep high guidance weights from pushing pixels out of the valid range, and an efficient U-Net with rebalanced resolution blocks. The lesson generalises: for prompt-driven generative AV, invest in the conditioning encoder, not only the renderer.

Examples

Imagen uses a frozen T5-XXL encoder; on human evaluation (DrawBench) T5-XXL is preferred over a CLIP text encoder. Dynamic thresholding rescales predictions whose high-percentile absolute value exceeds 1.

Assessment

What text encoder does Imagen use, and why is it frozen? State Imagen’s finding about where scaling helps most, and explain what dynamic thresholding fixes at high guidance weights.