Replacing the standard VAE with TinyVAE (TAESD) reduces encode/decode latency at small quality cost

The VAE encode and decode steps bookend every diffusion inference pass. For real-time use the standard Stable Diffusion VAE is a bottleneck: it is large and slow. Tiny AutoEncoder for Stable Diffusion (TAESD, madebyollin/taesd) is a distilled VAE approximately 4× smaller. StreamDiffusion swaps the pipeline’s VAE with TAESD by default (use_tiny_vae=True). The switch is a single assignment: stream.vae = AutoencoderTiny.from_pretrained('madebyollin/taesd'). Because latent-space dimensionality is preserved, the UNet is unaffected. At real-time frame rates the perceptual quality reduction is typically invisible at normal viewing distances but can be noticeable in static high-quality renders.

Examples

from diffusers import AutoencoderTiny
# Enable (default in StreamDiffusionWrapper):
stream.vae = AutoencoderTiny.from_pretrained('madebyollin/taesd').to(device=pipe.device, dtype=pipe.dtype)
# Disable for higher fidelity (slower):
StreamDiffusionWrapper(..., use_tiny_vae=False)

Assessment

A performer notices blocking artifacts in their live diffusion output. Should they switch use_tiny_vae on or off, and what is the expected speed trade-off?