Diffusion Transformers replace the U-Net backbone with a Vision Transformer operating on patchified latents

DiT (Peebles and Xie 2023) replaces the convolutional U-Net with a Transformer that operates in the same latent space as an LDM. The input latent is patchified into a sequence of tokens that pass through standard Transformer blocks. Conditioning on timestep t and class label c is injected with adaptive layer norm (adaLN-Zero): the scale and shift parameters are regressed from the sum of the t and c embeddings, and this approach outperforms in-context conditioning and cross-attention. Because Transformers scale predictably with compute, larger DiT models and more compute consistently improve sample quality, making DiT more straightforwardly scalable than U-Net architectures. DiT-style backbones underpin later systems such as SD3 and video generators like Sora.

Examples

DiT patchifies an IxIxC latent into (I/p)^2 tokens. Scaling from small to XL DiT models steadily lowers FID, illustrating Transformer scaling behaviour in generative modelling.

Assessment

Explain the patchification step in DiT. What is adaLN-Zero and why does it outperform cross-attention for conditioning the Transformer on timestep and class information?