unCLIP generates images by mapping text to a CLIP image embedding, then decoding that embedding to pixels with diffusion

DALL-E 2 (unCLIP, Ramesh et al. 2022) exploits the aligned CLIP embedding space, where semantically similar text and images map to nearby vectors. unCLIP trains two models: a prior P(c_i|y) that maps a text input to a CLIP image embedding, and a decoder (a diffusion model) that generates a pixel image from that image embedding. At inference: text -> CLIP text embedding -> prior -> CLIP image embedding -> diffusion decoder -> image. Because the intermediate is a CLIP image embedding, the decoder can also produce variations of a given image (preserving style and semantics) and enables zero-shot, text-driven image manipulation.

Examples

Zero-shot image manipulation: encode a photo to its CLIP embedding, nudge it with a text-derived CLIP direction, then decode to get a modified image that keeps the original’s style.

Assessment

Describe the two-stage generation process in unCLIP. What role does the CLIP image embedding play, and how does it enable zero-shot image manipulation?