Generative and Multi-Tool Audio-Visual Mapping

This module is where audio-reactive visuals stop being a novelty and start being a set. In a real club or festival slot — you as the visualist beside a live electronic act, or driving both sound and image solo — a single amplitude-to-brightness map reads as mechanical within a minute. The whole task here is a scene that listens like a musician: kick, snare, lows and mids each own a different visual parameter, and at least one visual behaviour is chosen probabilistically, biased by the music rather than chained to it.

The arc starts supported: in your VJ tool of choice, follow the band-splitting procedure (drag named analysis bands onto parameters, one band per visual job) until routing a kick to a pulse and a low swell to a slow bloom is a reflex — this binding move is the part-task drill. Next, replace one direct map with a rule inspired by the probabilistic feature-to-visual principle: let a feature weight a random choice among visual actions, so the audio biases the distribution and chance supplies the surprise. Finally, zoom out to the performance frame: the sync-granularity spectrum tells you whether your role wants per-parameter firehose or coarse cue-passing, and the performer-anchored concept shows how audio plus positional tracking pins visuals to a moving body — both feed the written half of the capstone.

The four required atoms gate the capstone directly: no band-splitting, no scene; no probabilistic rule, no generative claim; no granularity or anchoring understanding, no credible documentation. The supporting atoms enrich the “across tools” view — Gibber’s assignment-plus-scaling idiom and TouchDesigner’s any-channel-drives-any-parameter insight show the same mapping discipline transposed, useful when your rig changes but not prerequisites for this build.

Walkthrough

Direct mapping (one band → one knob) is where you started; a generative audio-reactive scene layers band-splitting, a probabilistic rule, and an autonomous bed so the visuals feel composed rather than merely triggered. Open the Hydra editor, route audio in, and run each with Ctrl-Shift-Enter. Each fence is complete. (time is Hydra’s autonomous clock; a.fft[0..3] are the bands.)

1 — band-split into a scene. Split the spectrum and give each band its own visual system: bass drives a core’s size, mids a colour field, high-mid the spin. Three independent reactive systems on one canvas is the skeleton of a generative scene ([[td-band-split-audio-reactive-mapping]]).

a.setBins(4)
shape(4, 0.3).scale(() => 1 + a.fft[0])                  // bass → core size
  .add(osc(10, 0.2).color(() => a.fft[1], 0.5, 1), 0.5)  // mids → colour field
  .rotate(() => a.fft[2] * 2)                            // high-mid → spin
  .out(o0)

2 — a probabilistic rule (not a direct map). Instead of a band setting a value, let it set a probability: a loud highs band raises the chance of a symmetry burst each frame. The result is related to the audio but not mechanically locked — the generative move ([[audio-feature-probability-visual-generation]]).

a.setBins(4)
osc(15, 0.1, 0.5)
  .kaleid(() => (a.fft[3] > 0.4 && Math.random() < 0.3) ? 8 : 2)   // stochastic burst
  .color(0.8, 0.5, 1)
  .out(o0)

3 — the sync-granularity spectrum. Motion can be locked tightly (per-frame reactivity) or loosely (a slow, bar-scale autonomous drift). Running both at once gives depth: a slow rotation you read across the whole phrase, plus a fast bass warp. Where each motion sits on this spectrum is a composition choice ([[av-sync-granularity-spectrum]]).

a.setBins(4)
noise(2, 0.2)
  .rotate(() => time * 0.05)                    // loose: autonomous, bar-scale
  .modulateScale(osc(4), () => a.fft[0] * 0.4)  // tight: per-frame bass reactivity
  .color(0.6, 0.8, 1)
  .out(o0)

4 — anchor to a performer, not just the audio. Audio can drop, and a performer moves off-beat — so anchor the scene to a slow autonomous motion that never depends on the signal. The reactive elements ride on this stable bed, keeping the visual alive through breakdowns ([[performer-anchored-av-from-audio-and-tracking]]).

a.setBins(4)
gradient(0.1).rotate(() => time * 0.03)         // the anchor — always moving, never reactive
  .add(shape(6, 0.2).scale(() => 1 + a.fft[0] * 0.5), 0.5)   // reactive element rides on top
  .out(o0)

5 — layer all three behaviours. An autonomous bed, continuous mid reactivity, and a probabilistic highs burst in one patch — the three temporal registers (always-on, continuous-reactive, stochastic-event) that make a scene read as living rather than looping.

a.setBins(4)
osc(8, 0.1, 0.6).rotate(() => time * 0.04)      // autonomous bed
  .modulate(noise(3), () => a.fft[1] * 0.3)      // continuous mid reactivity
  .kaleid(() => (a.fft[3] > 0.4 && Math.random() < 0.25) ? 6 : 2)  // probabilistic burst
  .color(0.9, 0.6, 1)
  .out(o0)

6 — the generative scene (the capstone). Everything, tuned: a slow autonomous gradient anchor, mids driving continuous colour, bass a fast continuous warp, and a probabilistic highs symmetry burst — a scene that spans the granularity spectrum and would survive a performer wandering off the grid ([[audio-feature-probability-visual-generation]]):

a.setBins(4)
a.setSmooth(0.85)
a.setScale(6)
gradient(0.05).rotate(() => time * 0.03)                       // performer anchor: autonomous drift
  .add(osc(12, 0.1).color(() => 0.4 + a.fft[1], 0.6, 1), 0.5)  // mids → continuous colour
  .modulateScale(noise(2), () => a.fft[0] * 0.4)               // bass → fast continuous warp
  .kaleid(() => (a.fft[3] > 0.45 && Math.random() < 0.3) ? 8 : 2)  // highs → probabilistic burst
  .out(o0)

What good looks like. A generative scene should feel authored by the music yet never quite repeating — the anchor gives continuity, the continuous bands give responsiveness, and the probabilistic bursts give surprise you couldn’t have keyframed. Document where each motion sits: what’s per-frame, what’s per-bar, what’s stochastic. The failure mode is either total lock (mechanical, every hit identical) or total randomness (noise with no relationship to the track). The burst probability is the dial between them. (Skill map: live-visualist Domain C2 — musical reactivity, plus keeping a non-reactive bed so reactive parts have contrast.)

Now make it yours. Raise/lower the burst probability and feel the scene move from tight to loose. Add a second probabilistic rule on a different band. Slow the anchor to a near-still drift for an ambient section. Swap gradient for a voronoi bed. Document your scene’s sync-granularity as a one-line note: which element anchors, which reacts, which surprises.