Group normalization divides channels into groups and normalizes within each group, working well with small batch sizes unlike batch normalization

Group normalization (GN) divides the channel dimension into G groups and computes mean/variance within each group per sample. Unlike batch normalization (BN), it does not depend on the batch size — statistics are computed per sample, not across the batch. This makes it effective for generative models trained on small batches or single samples, where BN statistics would be noisy. DDPM’s U-Net uses group normalization at every layer, combined with weight-standardized convolutions (a combination shown to work synergistically). GN is applied before attention layers via the PreNorm wrapper.

Examples

PyTorch: nn.GroupNorm(num_groups, num_channels). In DDPM U-Net: Block uses nn.GroupNorm(groups=8, dim_out). PreNorm applies nn.GroupNorm(1, dim) before attention.

Assessment

Explain why group normalization is preferred over batch normalization in diffusion model training. What happens to batch normalization when batch size is 1?