xformers memory-efficient attention reduces VRAM usage in diffusion UNets at modest speed cost

xformers implements a fused, memory-efficient attention kernel (FlashAttention-style) that reduces the peak VRAM footprint of the UNet’s self- and cross-attention layers. StreamDiffusion uses this via pipe.enable_xformers_memory_efficient_attention(). It is the default acceleration mode in several examples (acceleration='xformers') because it requires no compilation step unlike TensorRT. The speed improvement is moderate—lower than TensorRT—but it makes 512×512 diffusion feasible on GPUs with less VRAM (e.g., RTX 3060 12 GB). xformers is also combined with TensorRT in the sfast (stable-fast) path, which adds Triton kernel fusion and CUDA graphs on top.

Examples

pipe.enable_xformers_memory_efficient_attention()
# or via acceleration parameter:
StreamDiffusionWrapper(..., acceleration='xformers')

Assessment

Rank the three acceleration modes (none, xformers, tensorrt) by (a) throughput and (b) setup complexity. Under what circumstance would you choose xformers over tensorrt in a live performance context?