Mapping Audio Features to Visual Parameters

This module builds toward the bread-and-butter task of the audiovisual live-coder: making a visual rig breathe with the music coming out of your DAW. In a club or gallery set, nobody hands you a clean analysis feed — your techno kick, pad swells, and hat patterns leave the DAW as an output stream, and the visuals only react if you close the loop yourself and then choose mappings that read as musical rather than merely synchronized.

The arc starts fully supported. First, get signal flowing: the loopback-routing procedure (“route system audio back in as an input via a platform-specific loopback device”) is your JIT how-to for the single most common setup failure, whether you’re on PulseAudio monitors, BlackHole, or a virtual cable. With audio arriving, the two-stage extract-then-map model (“extract DSP data from audio and map it to visual parameters”) frames your first exercise: drive one obvious parameter — size from loudness — and feel how curve and scaling choices change the result. Then generalize: the insight that any channel can drive any parameter (“any CHOP channel can drive any operator parameter”) turns reactivity into a wiring-and-remapping habit, which is why that mapping move is the part-task drill — you’ll repoint sources to destinations dozens of times per set. Note that these ideas were first articulated in different tools (p5.js for extract-then-map, TouchDesigner for channel interchangeability), but they are the same model wearing different clothes: apply them in whichever visual environment your patch lives in. Finally, the eye-versus-ear asymmetry concept gives you the judgment to notice when a mapping is technically correct but perceptually wrong, exactly the repair the capstone demands.

Every required atom gates the capstone directly: without routing there is no signal, without the mapping model and channel interchangeability there are no three distinct mappings, and without the perceptual lens you cannot diagnose the out-of-step moment. There are no supporting atoms here — this module is all load-bearing.

Walkthrough

You already know how to get a bin reacting (the FFT lesson). This lesson is about which feature drives which parameter so it reads as musical — and how to fix a mapping that feels out of step. Open the Hydra editor, route your DAW in (loopback device, then the mic icon), and run each patch with Ctrl-Shift-Enter. Each fence is complete; the reactive value lives in a () => … thunk, so any number can be an audio feature ([[audio-reactive-loopback-routing]]).

1 — one feature, one parameter. Start with the most legible pairing: overall low-end loudness (a.fft[0]) → brightness. The image swells with level. This is the extract-then-map model at its simplest — a DSP number steering a visual knob ([[audio-reactive-dsp-data-mapping]]).

osc(30, 0.05, 0.6)
  .brightness(() => a.fft[0] * 0.8)          // bass loudness → brightness
  .out(o0)

2 — band-split into three senses. The spectrum isn’t one number — split it. Bass (fft[0]) drives size, mids (fft[1]) drive colour, highs (fft[3]) drive symmetry. Three features, three parameters, one patch ([[chop-channel-drives-any-parameter]]).

osc(20, 0.1, 0.5)
  .scale(() => 1 + a.fft[0])                  // bass → zoom
  .color(() => 0.5 + a.fft[1], 0.6, 1)        // mids → colour
  .kaleid(() => 3 + a.fft[3] * 5)             // highs → symmetry
  .out(o0)

3 — match the feature to the sense. Not every pairing feels right. The craft rule: bass → weight (scale, zoom, intensity), mids → texture/motion, highs → sparkle/edges. Heavy things should move to heavy sound. Here bass warps the whole field’s scale while a high-driven shape sparkles on top ([[audio-reactive-color-map]], [[geometric-reactivity-high-mid]]).

noise(2, 0.3)
  .modulateScale(osc(6), () => a.fft[0] * 0.4)      // bass → weighty warp
  .add(shape(3, 0.1).scale(() => 1 + a.fft[3]), 0.4) // highs → a small sparkling accent
  .color(0.9, 0.5, 0.3)
  .out(o0)

4 — smoothing is per-feature. a.setSmooth low-passes the values over time. Slow, weighty features (bass, pads) want heavy smoothing so they float; fast transients (hats) want light smoothing so they snap. Mismatch here is the #1 cause of un-musical reactivity ([[tempo-locked-change]]).

a.setSmooth(0.9)                              // heavy: the bass-driven zoom floats
osc(18, 0.08, 0.6)
  .scale(() => 1 + a.fft[0] * 0.6)
  .color(() => 0.4 + a.fft[3], 0.7, 1)        // (a snappy hat map would want LOW smoothing)
  .out(o0)

5 — diagnose and fix an out-of-step mapping. The seeing-vs-hearing gap: a patch can be technically reactive yet feel wrong. Symptom — highs jitter the zoom (the biggest, slowest thing) while the bass does nothing visible. Fix — put the weighty feature on the weighty parameter, and the fast feature on a small, fast one ([[seeing-hearing-difference-av]], [[glitch-reactivity-highs-band]]).

a.setSmooth(0.85)
osc(20, 0.1, 0.5)
  .scale(() => 1 + a.fft[0] * 0.8)            // FIX: bass → scale (weight follows weight)
  .thresh(() => 0.5 - a.fft[3] * 0.3)         // FIX: highs → edge crispness (fast → small/fast)
  .out(o0)

6 — three features + a beat flash (the capstone). The full musical mapping: mids colour the field, bass zooms it, a high-mid band warps it, and a threshold on the bass fires a discrete flash on the kick — continuous motion plus one on-beat event ([[organic-reactivity-low-mid-warp]], [[visual-pulse]]):

a.setSmooth(0.82)
a.setScale(6)
osc(20, 0.08, () => 0.4 + a.fft[1])          // mids → colour/detail
  .scale(() => 1 + a.fft[0] * 0.7)            // bass → zoom
  .modulate(noise(2), () => a.fft[2] * 0.3)   // high-mid → warp
  .brightness(() => (a.fft[0] > 0.6 ? 0.25 : 0))  // bass threshold → beat flash
  .out(o0)

What good looks like. Musical reactivity feels composed with the track, not merely triggered by it: weighty sound moves weighty visuals, the flash lands on the kick, and something stays stable between hits so the eye has a reference. The tell of a bad mapping is the whole frame reacting to everything at once — jittery, exhausting, no hierarchy. When it feels off, ask which feature is driving the biggest parameter and whether that’s the slowest feature. Keep one element deliberately un-reactive. (Skill map: live-visualist Domain C2 — mapping audio to visuals musically, the hard part.)

Now make it yours. Move the flash to a high band (a.fft[3] > 0.5) so it fires on hats. Raise setSmooth to 0.95 for slow pads, drop it to 0.6 for snappy percussion. Add a second, un-reactive layer as a stable bed. Square a feature (a.fft[0] * a.fft[0]) so only peaks read — an onset-ish response.

Runnable examples

Generated from the context/ instrument corpus by concept (redistributable idioms only). Do not edit — regenerate with gen-module-examples.mjs.

audio-reactive-map

updateAudio(); scale(1 + amp * 0.01)

_{p5live-0044 · CC0-1.0}

voronoi(() => 4 + a.fft[0] * 10).out()

_{hydra-0035 · CC0-1.0}

modulation-warp

osc(4).modulate(src(o0), 0.6).out(o0)

_{hydra-0022 · CC0-1.0}

s0.initP5(); src(s0).modulate(noize(), 0.3).out()

_{p5live-0038 · CC0-1.0}

feedback-trail

osc(4).modulate(src(o0), 0.6).out(o0)

_{hydra-0022 · CC0-1.0}

function draw(){ fill(0, 20); rect(0, 0, width, height); circle(mouseX, mouseY, 40) }

_{p5live-0003 · CC0-1.0}

scale-pulse

uv *= 1.0 + 0.3 * sin(u_time * 2.0);

_{glsl-0030 · public-domain}

updateAudio(); scale(1 + amp * 0.01)

_{p5live-0044 · CC0-1.0}

grain-glitch

col += (h21(st + fract(u_time)) - 0.5) * 0.15;

_{glsl-0026 · public-domain}

flow-field

let ang = noise(x*0.01, y*0.01) * TWO_PI

_{p5live-0008 · CC0-1.0}

spectral-band-split

[ihi, imid, ilo] >> rgb

_{punctual-0041 · CC0-1.0}

chromatic-aberration

col = vec3(texture2D(u_tex0,st+vec2(.005,0)).r, texture2D(u_tex0,st).g, texture2D(u_tex0,st-vec2(.005,0)).b);

_{glsl-0036 · public-domain}

visual-pulse

zoom (1 ~~ 2 $ osc 0.25) (circle 0 0.3) >> add

_{punctual-0022 · CC0-1.0}