A performer's live audio plus tracked position can drive visuals anchored to their body in real time

An audio-visual performance can bind projected visuals to a performer by combining two live inputs: the sound they produce (captured from a wireless microphone and processed in real time) and their location on stage (obtained from a computer-vision tracking system). Knowing where the performer’s head is lets the visuals be positioned relative to their body rather than floating abstractly, while processing their sound lets the visuals respond tightly to what they are doing vocally. Levin used exactly this pairing in a 2004 ARS Electronica piece with two vocalists, projecting speech-and-song visualizations on a large screen behind them. The general lesson is that combining an audio feature stream with positional tracking yields visuals that read as belonging to the performer.

Examples

Two vocalists making unusual mouth sounds are each tracked by camera and miked wirelessly; the processed audio drives on-screen forms while the head positions keep those forms attached to each singer, so the performers appear to ‘wear’ their sound on the screen behind them.

Assessment

Sketch the input-to-output flow for a performer-anchored visualization: what audio input, what positional input, what visual output. Explain what positional tracking adds over an audio-only visualization, and why the two streams must be combined rather than used separately.