Performing Live AI Visuals in a Real-Time Rig

This module is where real-time diffusion stops being a benchmark and becomes an instrument. The whole task is a short live AV set — the kind you’d play at an algorave or a club visuals slot — where the AI layer is not a pre-rendered video but a living process: whatever your screen or webcam shows gets re-diffused frame by frame, steered by prompts you change mid-set, all composited inside TouchDesigner alongside the rest of your rig. In that setting, a dropped pipeline or a two-second latency spike is not a footnote; it is the performance failing in front of an audience.

The arc starts supported: run the screen-capture-to-img2img loop standalone, changing prompts live to feel how the feedback loop responds (the screen-region-as-diffusion-input pattern is your first JIT how-to). Then pre-render a stylised clip with the vid2vid procedure — same wrapper, file in, file out — which doubles as an offline sandbox for prompt and model choices. Next, move the pipeline inside TouchDesigner using the StreamDiffusionTD operator so diffusion output routes like any other texture in the node graph. Only then do you rehearse the unsupported capstone: mode switches, prompt steering, and mixing the live layer over your pre-rendered backdrop.

The required atoms gate the capstone directly: you cannot switch modes on the fly without knowing the img2img/txt2img latent and CFG constraints, cannot host the pipeline without the TD integration atoms, and cannot defend your latency choices without understanding why StreamDiffusion is fast at all — Stream Batch, RCFG, and the realistic fps envelopes are the vocabulary of that latency documentation. Supporting atoms deepen the craft: the wrapper-vs-raw API split, the similarity filter’s frame-skipping economics, a transferable segment-size-vs-VRAM budgeting principle (drawn from audio stem separation, useful when your rig shares one GPU across processes), and the live-coding culture debate about what AI assistance means on stage.