Audio-Visual Performer — integrated, synced live AV

This path is for a performer who has already found their footing in one medium — music or visuals — and is ready to fuse both into a single, tightly-coupled instrument. The north star is a fully integrated live AV set where sound and image are generated and synced in real time on a shared clock, across multiple machines, with a planned transition executed cleanly and a live recovery that never drops the show. If you want to drive a room where the bass kick physically deforms the image and the visual architecture shifts with the musical structure, this is your path.

The arc moves through five fidelity levels, each ending with a whole performable task. Segment one, “Pair sound and image,” is deliberately unsynced: you build the minimum viable AV rig — a Hydra visual engine running beside your audio, connected by a single OSC or MIDI wire — so you understand the physical and conceptual plumbing before adding reactivity. “Orienting in the audiovisual-performance field” gives you the lineage, and “Hydra: live-coding video-synth visuals” gives you the visual engine you will use throughout. Segment two, “Make the image listen,” is where the visual surface wakes up: “Extracting Loudness and Spectrum From Live Audio” and “Building Audio-Reactive Visuals in Hydra” teach you to pull RMS, FFT, and beat triggers out of a live signal and route them into Hydra’s modulation inputs, so every kick and spectral shift lands in the image. Segment three, “Lock the clock,” is the technical spine of professional AV work: “Syncing Tempo and Phase Across Apps with Ableton Link” and “Controlling Ableton Live Over OSC with AbletonOSC” let you lock every machine to a shared quantum and steer the whole rig from a single control surface, so loops launch quantized and a tempo nudge propagates everywhere. Segment four, “Compose the whole,” layers generative logic and AI on top of that locked infrastructure: “Composing Audio-Visual Sets From Musical State and Cues” teaches you to drive visuals from Live’s internal state — clip envelopes, per-track meter energy, scene changes — and “Performing Live AI Visuals in a Real-Time Rig” (required) adds a real-time diffusion layer that reacts to your feed in under a second, scaffolded by “Optimizing Real-Time Diffusion with StreamDiffusion” (required) so the AI layer is guaranteed to be performable, not merely theorised. “The Text-Conditioned Image Generation Stack” (recommended) provides the conceptual underpinning for the diffusion tools you will run live. Segment five is the north star capstone: a multi-machine Link jam where everything you have built runs together. The capstone module, “Performing a Multi-Artist Link-Synced AV Jam,” explicitly includes planning and rehearsing transitions between sections and practising at least one deliberate recovery scenario (a machine drop, a clock desync, a visual crash) so that the north-star milestone’s requirements — a planned transition executed cleanly and one live recovery handled on stage — are scaffolded, not assumed.

The path deliberately skips single-medium fundamentals. Deep music theory, synthesis, and groove work are assumed from the live-coder path. The full creative-coding and shader progressions, VJ clip-mixing, Resolume and TouchDesigner rig-building, and projection mapping belong to the live-visualist and vj paths — this path treats the visual engine as already in hand. The heavy AI research track (training RAVE/DDSP models, diffusion internals from scratch, stem-separation training) hands off to the generative-ai path; here those tools appear only as finished, performable instruments.

You should come in having completed the live-coder path’s musical footing — specifically the Strudel and Tidal groove foundations, mini-notation, and the live-coding ethos — or their equivalent in another tool. Without that, the audio half of your AV rig will be under-formed. The visual fundamentals (p5.js foundations, shader basics, visual design principles) from the live-visualist path are recommended but the path includes a short Hydra bridge so a motivated newcomer to visuals can still follow the spine.