CLIP jointly trains text and image encoders so matching pairs get high cosine similarity

Stable Diffusion’s text encoder is the text half of a CLIP model (ClipText, a GPT-based transformer). CLIP is trained on ~400 million image/caption pairs (web images plus their ‘alt’ tags). An image encoder and a text encoder embed each pair; training pushes matching pairs to high cosine similarity and, crucially, includes negative (non-matching) image/caption pairs that must get low similarity. Repeated over the dataset with large batches, this makes the text encoder embed language into the same space as visual concepts — ‘a picture of a dog’ lands near images of dogs. Because the text encoder carries this visual grounding, upgrading it (e.g. OpenCLIP up to 354M params vs ClipText’s 63M) improves generation quality more than enlarging the image-generation network does.

Examples

ClipText output: 77 token embeddings × 768 dims. Training: embed image + caption, raise cosine similarity for true pairs, lower it for negative (mismatched) pairs. SD v2 switched to OpenCLIP.

Assessment

Explain how CLIP’s contrastive training (positive and negative pairs, cosine similarity) links text to visual concepts, and why upgrading the text encoder can improve image quality more than upgrading the image generator.