openFrameworks, audio/video I/O, and hardware bridges

This module is where sketches stop being toys and become instruments. In a live audiovisual set — a VJ rig reacting to a DJ’s mix, or a solo laptop performance where visuals breathe with the music — you need one app that ingests media, listens to sound, takes a knob or sensor as input, and leaves behind a loopable video for promo and documentation. openFrameworks (with Processing as a fallback) is the C++ workhorse for exactly this: real I/O, real hardware, real frame rates. Be aware the module deliberately mixes toolchains: the app core is openFrameworks, but the serial bridge and frame-sequence video export are taught on the Processing side — on the pure-OF path you apply the same patterns via OF’s equivalents (ofSerial, per-frame grabScreen saves), or you switch to Processing for those two steps.

The arc starts fully supported: wire up a project with addons via projectGenerator or addons.make, then follow the setup/draw lifecycle to get an image on screen (“ofImage loads and draws image files”) and a sound looping (“ofSoundPlayer loads and plays a sound file from bin/data”). Add an ofxGui panel with a float slider so a parameter goes live under your fingers — drill that load-once, draw-every-frame, tweak-live loop until it is automatic. Next, make the app listen: envelope following turns raw amplitude into a smooth control signal. Then reach outward over USB serial to a microcontroller sensor — this atom is Processing’s Serial library, and the two-sketch model it teaches carries over directly. Close the loop by capturing output: one-shot grabScreen stills in OF, then frame-sequence export with a frameCount-bounded run for an exact looping clip — again taught in Processing, with the save-every-frame-then-exit pattern transplanting cleanly.

Every required atom is a gate: skip addon integration and the GUI never builds; skip envelope following and the visuals cannot react; skip frame export and there is no deliverable loop. The supporting atoms enrich rather than gate — p5.sound supplies the audio-analysis vocabulary (amplitude, FFT, peaks) in browser form, 3D model loading extends the same addon workflow, the unit-generator model explains what your amplitude signal is downstream of, and gesture recording reframes your slider moves as performance data.