J · Audio-visual integration & sync

Five terms — visual music, expanded cinema, live cinema, VJing, and live AV performance — divide the audiovisual-performance field

Hydra is a browser-based, JavaScript live-coding language for networked video-synth visuals

Live cinema is the simultaneous real-time creation of sound and image by sonic and visual artists on equal terms

The demoscene emerged from software cracktro culture in the 1980s as a realtime audiovisual art form

A signal can be reconstructed only if sampled above twice its highest frequency

Additive (Linear Dodge) blend mode sums pixel values, producing glowing brightness that clips to white on overflow

An OSC message has two parts: an address pattern (the parameter name) and one or more typed arguments (the values)

Any live cinema performance can be analysed through five tool-independent essential elements

Audio-reactive synthesis generates visuals from sound data rather than triggering pre-recorded clips

Dick Higgins' 'intermedia' names a new entity that emerges from merging two art forms rather than merely adding them

Every OSC message carries a type tag string that encodes the data type of each argument

Frequencies above the Nyquist frequency fold back into the audio band as inharmonic aliases

glslViewer in this rig has no audio input — `a.fft` does not exist and reactivity must use `u_time`

Hydra can use webcams, screen capture, video files, and images as source buffers alongside generated visuals

Hydra FFT bins stay at 0 until a Strudel pattern is tagged `.analyze('hydra')`

Hydra has four independent output buffers (o0–o3) that render separately and can be cross-mixed

In music, pitch corresponds to line width and melodic contour traces a literal line in visual space

In this rig, Hydra's `a` object is a 4-bin shim fed from Strudel, not the native mic FFT

Live audiovisual performance is both an umbrella generic term and a specific category for work that fits none of the other four labels

Liveness in AV performance means the bodily co-presence of performer and audience, not merely that something is happening in real time

Liveness, intermediality, performativity, and cinematicity are the four axes shared across all AV performance practices

MIDI is a serial control protocol carrying numbered performance messages, never audio waveforms

Non-narrative cinema uses visual rhythm and montage structure rather than story to create meaning

One E1.31 universe carries 512 DMX channels, controlling up to 170 RGB LEDs at 3 channels each

Open Sound Control is a transport-independent network protocol carrying typed, address-patterned messages between devices

OSC connections require both a destination IP address and a port number to route messages to the correct application

OSC's 32-bit float arguments provide far higher control resolution than MIDI's 7-bit (0–127) integer range

p5.sound wraps the Web Audio API in a Processing-style interface for audio analysis and playback

P5LIVE audio-reactivity has three incompatible sources (p5.sound, HY5, embedded Strudel) and none is the rig's 4-bin a.fft contract

Punctual's lo/mid/hi analyse the program's own audio output; ilo/imid/ihi analyse microphone input — reading lo with no audio output gives 0

Realtime technology is what makes live AV performance possible by enabling simultaneous capture and manipulation of sound and image

Repetition of a point or element is a source of elementary rhythm and a means of heightening inner vibration

Sonic Pi cannot feed the rig's 4-bin FFT bridge for audio-reactive visuals — it runs in its own IDE with a bundled SuperCollider server

Strudel patterns must include .analyze('hydra') for visuals to receive FFT data — an untagged pattern plays audio but the visuals see a.fft = 0

SuperCollider audio-analysis UGens are not wired to this rig's AV bridge and cannot drive its visuals

The Hydra rig shim exposes only 4 FFT bins — onset, tempo, RMS and spectral centroid do not exist

Visual music is defined by the quality of its audiovisual combination, not by any single medium, context, or presentation form

A Clip Slot is the container; firing it via OSC triggers play/pause regardless of whether a clip exists

A delay line is a circular buffer read behind a rotating write pointer

A Max for Live device bridges an external OSC control surface to Live via the LOM

A reactive Hydra parameter must be written as a () => … thunk — a bare expression is evaluated once at eval time and frozen

A reactive visual parameter that spends most of its time pinned at its min or max fails the sanity check and needs its base, gain, or scale retuned

A rising audio energy arc can be mapped to rising visual motion rate or scale so that both domains accelerate together during a build

A single 'energy' scalar driving multiple motion parameters makes the whole image rise and fall coherently

A spectrogram is a time-frequency power map created by computing the STFT frame-by-frame and plotting energy per bin

A Strudel voice must be tagged `.analyze('hydra')` to contribute to the FFT — an untagged voice is inaudible to the visuals even if loud in the speakers

A visual pulse is amplitude-following, not beat-locked, and will drift from the musical grid

a.bins gives Hydra's un-normalised FFT values and a.prevBins gives the previous frame's, enabling delta-based reactivity

a.fft is updated once per rendered frame at ~60 Hz — sub-frame transients are averaged away, making onset detection impossible

a.fft values are not clamped to 0–1 and can exceed 1 under normal use, so bounded visual parameters must be clamped in the sketch

a.setScale() and a.setCutoff() map FFT bin values to 0–1 via a noise gate and a ceiling threshold

a.setSmooth() prevents strobe artifacts by exponentially averaging consecutive FFT frames in Hydra

Ableton Link synchronizes tempo across apps by group adoption rather than a central master clock

Ableton's Live Object Model exposes Live's internals to Max for Live for programmatic control

AbletonOSC is installed as a MIDI remote script that exposes Live's full API over OSC

AbletonOSC pushes an int beat number to the client on every beat via start_listen/beat

AbletonOSC uses a fixed port pair: commands arrive on 11000, replies go out on 11001

AbletonOSC's start_listen/stop_listen pattern subscribes to Live property changes and delivers them as push messages

All compositional relationships reduce to two principles: parallel (side-by-side reinforcement) and contrast (opposition)

All couplings requiring onset, tempo, beat-phase, or per-instrument signal share one root blocker and are not achievable with the 4-bin bridge

An envelope follower extracts a control signal that tracks a sound's amplitude contour over time

An exp-curve bass amplitude pump is the supported proxy for a kick-keyed visual sidechain, and it is not the same thing

Audio can drive emphasis in a composition but cannot move focus on the beat in the current rig

Audio-reactive texture intensity (band amplitude to grain/warp) is realizable now; onset-triggered glitch bursts are not possible in this rig

Audio-reactive visuals extract DSP data from audio and map it to visual parameters

Audio-visual coherence requires agreement on energy, spectral balance, and section — reactive motion alone does not guarantee it

AV mapping transfer functions are almost always affine (linear) or squared-exponential — linear for continuous motion, exp for onset-ish pop

Bass should drive large, slow visual elements and highs should drive fine, fast detail for maximum AV coherence

Beat alignment means any integral beat value on one Link participant maps to an integral beat on all others

Beat detection via amplitude threshold fires a visual event when RMS crosses a set level

Bela is an embedded platform achieving sub-millisecond action-sound latency by running audio at the hardware interrupt level

Bela reads analog sensors synchronously with the audio block, removing the polling lag of slow microcontrollers

Clip launch quantization in AbletonOSC maps integer codes to musical durations, from None to 1/32

Composition is the law-abiding organization of the tensions within elements

Driving a shape's radius or count from audio is realizable now; making a shape appear on-the-kick is not

Driving several unrelated visual parameters from the same FFT band reads as everything-pumping-together mud — one band should map to one dominant target

Driving visual motion from band energy rather than Hydra's clock produces a tighter in-time feel because energy rises and falls with the groove

E1.31 (sACN) sends DMX over multicast UDP, with universe numbers matching QLC+ numbers directly

Editing film to music's rhythm — including the body's biorhythm — produces a profound physiological impact on viewers

FFT decomposes audio into frequency bins that can be mapped to individual visual elements

GLSL uniforms pass values from the CPU to the GPU each frame

Hydra is a browser-based live coding environment for visual synthesis, enabling collaborative networked visual performance

Hydra models WebRTC browser streams as patchable modules, enabling live routing of video between remote peers

Hydra reads only the browser's default microphone input, not desktop audio; DAW output requires virtual audio routing

Hydra's `a` object exposes real-time FFT bins so any parameter can be driven by an audio frequency band

Hydra's a.fft array holds per-bin amplitude values produced by FFT analysis of the microphone input

Hydra's a.onBeat() callback fires whenever a.vol crosses a configurable threshold

Hydra's a.setSmooth sets smoothing for all bands globally — to smooth one target differently, write a per-value lerp in the thunk

Hydra's blend family composites two sources by arithmetic on their pixel colors

I/O vector size in Max/MSP sets the tradeoff between audio latency and CPU efficiency

In Max the [route] object both separates OSC messages by address pattern and strips that pattern off, dispatching each to its handler

In the 4-bin rig, audio bands can drive color brightness, saturation, or palette phase but not onset-triggered color events

In this rig, audio-reactive visual motion is envelope-following only — beat-locking and onset-triggering are not available

In TouchDesigner any CHOP channel can drive any operator parameter, so audio, MIDI and beat inputs are interchangeable

In visual composition, a line's length is a concept of time, and a curved line represents more time than a straight line of equal length

LedFx processes audio input in real time and pushes light effects to networked LED devices over Wi-Fi

Live cinema replaces film's narrative structure with a musical arch of tension, rhythm, and colour

Map the low-mid band to organic warp/flow depth so the field swells and eddies with the music's body

Mapping audio low-mid to fractal zoom rate or feedback gain risks runaway whiteout — keep gain in check

Mapping bass to feedback-trail zoom or symmetry scale makes the tunnel pulse with the low end

Mapping high-mid band to rotation speed or line thickness gives geometric visuals a crisp, articulate audio response

Mapping the highs band to glitch intensity makes corruption spike with hats and transients — the closest proxy to onset-triggered glitching

Max for Live devices come in three types — Audio Effect, Instrument, and MIDI Effect — each with distinct signal roles

Max Overdrive and Audio Interrupt tighten timing of MIDI and metro events against the signal vector

Max's transport object provides a global master clock syncing objects to musical time values

MIDI Channel Voice Messages carry per-note performance data including Note On, Control Change, and Pitch Bend

MIDI Clock sends 24 pulses per quarter note so slaved devices can synchronise tempo to a master sequencer

Music feedback-loop and visual feedback-trail are the same self-feedback mechanism — a signal fed back into itself with less-than-one gain — expressed in two domains

Network jitter in OSC streams can be mitigated by buffering, rate-limiting, or smoothing incoming values

Open Stage Control loads a JSON file that defines a portable OSC control surface

OSC (Open Sound Control) is a network-based messaging protocol for real-time musical control, designed to supersede MIDI's limitations

OSC targets parameters with a left-to-right hierarchical address path plus a value

Overt audio-reactivity breaks the stillness of a minimal visual — subtle smoothed bass or no reactivity at all

p5.FFT provides both frequency-domain analyze() and time-domain waveform() readings

P5LIVE is not part of the 4-bin Strudel+Hydra rig — a.fft[] does not exist in P5 and must never be emitted there

Pinning a free-running audio tremolo and a visual brightness-pulse to the same rate value makes them move in lockstep without a shared transport

Publishing an SDK as open source accelerates adoption of a protocol across diverse software

Punctual runs audio and visuals in one program, eliminating the need for an external AV bridge — audio reacts to its own sound natively

QLC+ transmits DMX over Ethernet using Art-Net UDP with universe-numbering offset by 1

QLC+ uses OSC over UDP with auto-assigned ports derived from universe numbers

Quantized launch makes apps wait for the next quantum boundary before starting, enabling tight ensemble starts

Reading a.fft[4] or higher returns undefined, causing NaN arithmetic and a frozen or black Hydra frame

Real-time in live cinema has multiple levels: mixing pre-recorded clips, generating visuals algorithmically, and processing live camera feed

Reducing bit depth adds harmonic quantization noise; dithering trades it for benign broadband noise

RMS amplitude gives a perceptually smooth loudness value between 0 and 1

Routing the highs band to kaleidoscope symmetry count produces a coherent coupling — fast bright content increases visual complexity

Section-level visual intensity must be edited to match the music arrangement because no section signal crosses the AV bridge

Sound reaches Hydra through exactly one path: a 4-element FFT array filled from Strudel's Web-Audio analyser — no MIDI, OSC, or per-instrument bus exists

Spatial montage presents multiple images simultaneously rather than sequentially, enabling parallel narratives and immersive environments

Squaring a band value produces a soft onset-ish punch that pops on loud hits without requiring true onset detection

Strudel and Hydra have no shared clock — Hydra's bpm and Strudel's transport are independent and will drift

Superimposing two live images creates a 'third space' that belongs to neither source

Tempo-locked visual changes are not achievable in this rig because the bridge exposes only FFT energy with no beat-phase or onset

The 4-bin layout is the shared surface between music frequency-budgeting and visual spectral-band-split: a well-budgeted mix yields a legible band-split visual for free

The AV shim exposes four per-band tuning controls — setScale, setCutoff, setSmooth, setBins — that adjust how a.fft responds without changing the sketch

The base term in a mapping transfer function is the visual floor at silence — it must produce an intentional frame when a.fft is zero

The Live Object Model organises Live's entire state as a tree: Song > Tracks > Clips/Devices > Parameters

The metamedium reframes intermedia for the digital era as an active mix of media, not a mere addition

The seed mapping uses bass for brightness, high-mid for rotation speed, and low-mid for warp depth — three of four bands are assigned, highs are unused

The Song API lets you start/stop playback, set tempo, and jump to cue points using OSC messages

The spectral centroid is the frequency-domain centre of mass of a sound spectrum and correlates with perceived brightness

The STFT window length trades time resolution against frequency resolution and cannot give both at once

The video loop is the fundamental unit of live visual performance, replacing the single-shot timeline of cinema

Timbre is a perceptual quality; spectrum is its physical correlate—they are related but not equivalent

To make software react to music playing on a computer you must route system audio back in as an input via a platform-specific loopback device

Using more than 3 E1.31 universes in WLED degrades frame rate below 40fps

VJing is structurally collaborative because it always visualizes something else — most often a DJ or live music act

When no clock signal crosses the audio-visual bridge, the performer matches visual phrase length to musical phrases by feel and manual edit timing

WLED Effect DMX mode allows remote control of built-in effects via 15 DMX channels without per-pixel data

WLED offers multiple DMX channel modes ranging from single-color control to per-pixel RGB addressing

WLED Pixel Grouping treats N consecutive physical pixels as one DMX channel to stretch limited universes

WLED supports Art-Net and DDP as E1.31 alternatives, but DDP always uses Multi RGB regardless of DMX mode

Wrapping the input coordinate into a periodic cell tiles one SDF into infinite copies at near-zero cost

A DAW's internal musical state, not just its audio, can drive visuals when exposed over OSC

A jitter buffer absorbs packet timing variation by introducing a small constant delay before playback

A jitter buffer must define policies for underflow (empty) and overflow (full) conditions

A MIDI 2.0 device must implement MIDI-CI-with-discovery plus a feature, or UMP-with-discovery plus a feature

A MIDI-CI Profile is a published set of messages and required responses that devices can advertise and negotiate

A pattern must be tagged .analyze('hydra') to be visible to the audio-visual bridge

A performer's live audio plus tracked position can drive visuals anchored to their body in real time

Ableton Link lets visual software lock video clips and generative patches to a shared musical clock

AbletonOSC exposes per-track output meter levels, enabling audio-reactive visuals driven by individual track energy

AbletonOSC implements OSC wildcard dispatch by converting * to a [^/]+ regex and matching all registered handlers

AbletonOSC lets you add, remove, and query MIDI notes in a clip programmatically over OSC

AbletonOSC supports OSC wildcard patterns using * to query all sub-addresses matching a path segment

AbletonOSC's Device API lets you read and write any synthesiser or effect parameter by index via OSC

AbletonOSC's MidiMap API creates persistent MIDI CC assignments to device parameters via OSC, without Max for Live

AbletonOSC's OSC server uses a non-blocking UDP socket to drain all queued messages per tick without blocking Live

AbletonOSC's property listener uses Live's add_<prop>_listener API and immediately sends the current value on subscribe

Abstract visuals resist narrative analysis but can be composed and critiqued using the perceptual principles of music

Abstraction in live AV defeats the audience's conditioned expectation to read images as narrative

Adaptive jitter buffers resize dynamically to stay as small as possible while preventing dropouts

An Ableton Link session does not require Ableton Live to be part of it

An audio ADSR envelope and a visual physics simulation share the same integrate-over-time structure — both accumulate an impulse into a changing state

An OSC bundle groups multiple messages with a shared timetag so they are dispatched atomically at the same time

Analysing sound features (frequency, tempo, gesture) as inputs to probabilistic rules generates visuals that respond to music without one-to-one mapping

Any numeric parameter in Hydra can be replaced with a function, enabling audio or MIDI to modulate visual parameters

Audio FFT data can be passed to shaders as a 4-band uniform for audio-reactive visuals

Audio round-trip latency has multiple additive sources and can be quantified with a loopback pulse test

Autocorrelation detects pitch by finding the lag at which a signal most resembles itself

Automation and Clip Envelopes in Ableton Live can generate discrete visual output cues without audio processing

AV improvisation between visualist and musician requires real dialogue; in practice the visual artist often follows rather than co-creates

AV sync ranges from fine-grained per-parameter control for soloists to coarse cue-passing for visualist collaborators

Bidirectional communication is the core innovation of MIDI 2.0, letting devices discover and auto-configure for each other

Centralising all sensor inputs through a single OSC/WebSocket bridge (MAMI pattern) lets TouchDesigner focus on interaction design rather than device plumbing

ChucK Unit Analyzers extract audio features from a signal for real-time analysis

CineCer0's natural/every/round/chop functions align video playback to Estuary's cycle grid

Cross-synthesis imposes the spectral envelope of one sound onto another by multiplying their spectra

Each Live scene can carry its own tempo and time signature, settable via OSC

Feeding an analog video signal into an audio amplifier makes a fixed-pitch drone whose overtones encode the image

Feeding the previous frame back with an FFT-driven amount makes visual trails grow with the audio build

Gestural interfaces require a visible, immediate correspondence between the performer's movements and the audiovisual output

Gibber maps audio objects directly to visual properties using bias and scalar for range control

Humans localise sound using interaural time, level, and spectral cues from the pinnae

Hydra can use other browser windows as live video sources via WebRTC peer-to-peer streaming

Interactive music systems react to performer input in real time, ranging from deterministic score-following to autonomous improvising agents

Jitter buffers use per-packet timestamps to reorder out-of-sequence audio packets

Jitter is variation in latency over time, not latency itself

Link keeps apps in time by relating independent timelines, not by forcing one shared timeline

Link start/stop synchronization only follows explicit user actions and is not auto-applied when joining a session

Live cinema is a hybrid of opera's theatrical gesture, painting's intimate craft, and the concert hall's performative presence

Live's embedded Python cannot use threads; AbletonOSC polls the OSC socket once per 100ms tick using Live's scheduler

Logarithmic and 1/3-octave FFT scaling match frequency display to human pitch perception

Many self-timed audio voices and many self-timed visual agents are the same concurrent-agents structure expressed in two domains

Mapping a per-track OSC envelope to visual parameters turns a music sequencer into a precise visual accent machine

MIDI 2.0 extends MIDI 1.0 rather than replacing it, so existing devices and messages stay valid

MIDI-CI is MIDI 2.0's discovery-and-negotiation layer, delivering profiles, property exchange, and process inquiry

MIDI-CI Property Exchange uses JSON over SysEx to get and set device data with no bespoke software

MIDI-CI runs over any MIDI 1.0 transport because it is carried in System Exclusive messages

MIR uses content-based audio feature extraction and machine learning to automate musical analysis and similarity search

Network latency above 20-30 ms disrupts tight rhythmic ensemble playing but can be worked around with free improvisation or asynchronous structures

nn~ can output audio-analysis features as control-rate signals by using a large out_ratio

Onset detection locates the start of new sound events in an audio signal by finding rapid increases in energy or spectral flux

Pink (1/f) noise has equal energy per octave and models the long-range correlations found in many natural signals including music

Pre-rendered audio analysis APIs supply timestamped beats and pitch for offline sync

QLC+ maps MIDI beat clock start/stop and beat signals to special channels 530–531 for BPM-sync

Robust beat detection uses a decaying cutoff plus a hold window to debounce triggers

Some quality checks are permanently not-possible on this rig because the 4-bin FFT cannot supply the required feature

Sonification maps non-audio data to sound parameters to convey information through the ear in SuperCollider

Splitting incoming audio into named bands lets a VJ hook each band to a different visual parameter

Stepping kaleidoscope symmetry up on a downbeat gives a visual drop paired with an audio peak

Stretching a video clip to bar length couples visual rhythm directly to musical tempo

SuperCollider's Pitch and Amplitude UGens extract frequency and loudness from audio input in real time

Swapping the FFT index driving a visual parameter aligns the visual energy with the new section's audio driver

t-SNE embeds high-dimensional feature vectors in 2D so perceptually similar items cluster together

TDAbleton enables bidirectional MIDI and audio-feature communication between TouchDesigner and Ableton Live, allowing visuals to drive audio and audio to drive visuals

The /live/song/get/track_data command queries multiple tracks and clips in a single OSC message

The AbletonOSC Python client uses a threading.Event to block on an OSC reply, creating a synchronous query interface

The Audio Trigger widget maps audio spectrum bands and volume to DMX channels, functions, or VC widgets

The auditory system resolves frequency within critical bands: components within one band interact, components in separate bands do not

The Link quantum value sets the loop length for phase synchronization, aligning loop boundaries across apps

The MIDI 2.0 Protocol raises resolution on all Channel Voice Messages and adds per-note expression

The overarching taste check compares each intended intent-vector field against a perceived estimate within tolerance

The strongest section changes fire an audio and a visual transition on the same downbeat

The track_data handler uses a dot-namespace format to query track, clip, clip_slot, or device properties in a single call

The Universal MIDI Packet is one container for all MIDI, adding 16 Groups of 16 channels (256 total) and jitter-reduction timestamps

TidalCycles uses Ableton Link by default for BPM synchronization with other performers and applications

Toggling a color inversion or hard posterize on the drop downbeat is a single-frame visual hit

Two MIDI 2.0 devices reach full communication through an ordered discovery sequence ending in normal MIDI use

Tying scroll or scale speed to a rising FFT band accelerates the visual motion through a build

Vary the type of energy (density, brightness, space) not only the loudness level

Vision and hearing process time and density differently — AV artists must account for these asymmetries to achieve coherence

VPT emits OSC loopreport and cliptime events so external tools can sync to video progress

VPT's built-in LFOs automate any routed parameter with sine, ramp, triangle or square waves

VPT's router maps a numbered controller to any parameter via rows keyed by controller number

VPT's SoundTrigger app maps left/right audio channels to VPT controllers for audio-reactive visuals

VPT's Video Trigger detects motion in camera zones by background subtraction to fire events

Ableton Link enables multi-artist AV jams where several visual and music machines share one clock

Ableton Link synchronizes beat, tempo, and phase across multiple apps or devices over a local network without a master clock

AbletonOSC supports hot-reloading its own handler code by sending /live/api/reload, avoiding a full Live restart during development

Estuary maintains a single shared tempo (CPS/BPM) that all languages and ensemble members follow

In a live rig, sync only the elements that must be locked and leave the rest free-running for a more musical, less rigid result

In abstract AV art, audiences engage through resonance with personal experience and self-reference, not object recognition

Link Audio channels are demand-driven: a sink only transmits when at least one source subscribes

Link Audio shares named audio streams (channels) between peers aligned to the shared Link timeline

Link session state modifications should only happen on the audio thread to achieve sample-accurate timing

Link session state must be captured before reading and committed after writing to ensure consistency

MFCC UGen extracts 13 spectral shape coefficients from an FFT frame for timbre classification in SuperCollider

OSCdef triggers a function on incoming OSC and can be re-evaluated live to remap external data to sound

Adding .analyze('hydra') to a Strudel voice feeds it to the a.fft array that Hydra's audio-reactive mapping reads

Hydra's audio FFT array has exactly 4 bands (indices 0..3); reading index 4 or higher produces NaN

P5LIVE has three distinct audio reactivity sources and no automatic 4-bin a.fft rig bridge

Punctual's lo/mid/hi analyse the program's own audio output — they are zero if no audio statement is running

Strudel voices must call .analyze('hydra') for audio data to reach Hydra's a.fft array

The terminal-rig glslViewer has no audio-reactivity bridge; use u_time for motion instead