ControlNet adds spatial conditioning to a frozen diffusion U-Net via a trainable copy connected with zero-convolution layers

ControlNet (Zhang et al. 2023) enables fine-grained spatial control of image generation (edge maps, depth, human pose, segmentation, scribbles, etc.) without retraining the base model. It freezes the original U-Net, clones its encoder blocks into a trainable copy, and connects the two with zero-convolution layers (1x1 convolutions whose weights and biases are initialised to zero). The conditioning input is injected into the trainable copy. Early in training the zero-convs output zero, so the frozen backbone is unaffected; as training progresses the zero-convs learn to pass spatial conditioning. Zero initialisation protects the pretrained backbone from random-noise gradients in the initial steps, preventing catastrophic forgetting. At inference, the frozen and trainable branches are merged additively.

Examples

A ControlNet trained on skeleton poses: given a text prompt plus a pose image, it generates a person in that exact pose. The frozen U-Net supplies appearance; the trainable copy supplies layout.

Assessment

Why are ControlNet’s zero-convolution layers initialised to zero? What would happen if they were initialised randomly, and how does zero init protect the backbone?