The denoising U-Net uses sinusoidal position embeddings to communicate the current noise level (timestep) to every layer

Because the network’s weights are shared across all timesteps, the network needs to know what noise level it is currently operating at. DDPM encodes the scalar timestep t using sinusoidal position embeddings (borrowed from Transformers) to produce a vector, then projects it through an MLP. This embedding is added to the feature maps inside each ResNet block via a scale-and-shift operation. Using sinusoidal embeddings (rather than a learned lookup table) provides smooth interpolation between timesteps and works even for timesteps not seen during training. The embedding maps a single timestep per batch element to a high-dimensional continuous vector.

Examples

Code: SinusoidalPositionEmbeddings(dim) takes (batch_size, 1) → (batch_size, dim). The time MLP: nn.Sequential(SinusoidalPositionEmbeddings(dim), nn.Linear(dim, time_dim), nn.GELU(), nn.Linear(time_dim, time_dim)).

Assessment

Explain why sinusoidal embeddings are preferred over a simple integer timestep input for time conditioning. What is the shape of the embedding output for a batch of 8 images?