Generative & AI AV Artist — real-time machine-driven performance

This path is for the coder/artist who already moves fluently inside a live-coding or creative-coding rig — who can sync visuals to audio, route OSC, and build reactive sketches — and wants to go much further: to build and perform with neural models they have trained and engineered themselves. The north star is a full live, machine-collaborative AV set on your own AI rig, where RAVE/DDSP neural audio instruments and steered real-time diffusion visuals run together under your hands.

The arc moves through five segments, each closing a whole creative loop before raising fidelity toward that north star.

The path opens by orienting you fast and getting a model into the creative loop immediately. “Orienting to Generative AI for Live AV” frames the philosophy and the representations; “Generating Drum Patterns with an ML Groove Model” wires a GrooveTransformer into a live beat the same week. The segment milestone — an ML-generated groove driving an audio-reactive GLSL visual over your own live-coded audio — proves the end-to-end pipeline works before any GPU training run begins. GLSL shader fundamentals run in parallel to build the visual-language footing that this first milestone already exercises; Hydra audio-reactive skills are assumed as background from the av-performer path.

Segment two is the neural audio engine. You work through DDSP from first principles — assembling harmonic synthesizers, noise components, and trainable reverb in the Processor API — then run real timbre transfer with “Engineering DDSP Timbre Transfer Internals” before training, architecturally dissecting, and eventually owning a RAVE model. The milestone is a released track whose sound design is carried entirely by two playable neural voices you trained.

Segment three deploys those models into your live rig via nn~ in Max/MSP or Pure Data. “Scripting Custom nn~ Models for Live Rigs” extends the rig to expose real-time source separation as a controllable external. The parallel WebGPU/WGSL thread builds the GPU pipeline foundation — render pipeline, WGSL shaders, ping-pong compute — that the next segment’s diffusion acceleration will need.

Segment four builds the diffusion visual instrument from scratch, in sequence: derive the DDPM, understand schedules and backbones, map the full text-conditioned stack with ControlNet and unCLIP variants, then steer and accelerate sampling for real-time use. “Optimizing Real-Time Diffusion with StreamDiffusion” is the engineering capstone of this segment, benchmarking the pipeline against published performance targets with TensorRT, TinyVAE, and residual CFG.

The final segment converges all five threads: “Performing Live AI Visuals in a Real-Time Rig” is the north-star performance itself.

This path deliberately skips the entire audio-visual sync and integration spine — OSC/MIDI wiring, audio- feature extraction, audio-reactive Hydra, Ableton Link, beat and onset detection, and cue-driven AV mapping — all treated as assumed footing taught by the av-performer path. Live-coded-music language fluency (Tidal, SuperCollider, Sonic Pi) and p5/Hydra creative-coding fundamentals are also assumed via general background from the live-coder and visual-live-coder siblings. Deep GPU raymarching and SDF artistry hand off to a dedicated shader-focused sibling. Music theory, mixing/mastering, modular/dawless synthesis, VJing/projection/LED/DMX hardware, and DJ stagecraft are all out of scope.

If you have not yet built a synchronized AV rig, done audio-feature extraction in Max, or used Ableton Link in a live set, start with av-performer first — every skill listed in the assumed prerequisites is taught there.