K · AI & real-time generative AV

The live-coding community values human struggle and failure over AI-generated output

A community Google Colab notebook enables RAVE training without local GPU hardware

Deep learning represents the world as a hierarchy of concepts, each built from simpler ones

Hand-coding world knowledge in formal rules failed, motivating machine learning from data

MIDI note numbers map to Hz via a tuning formula centered on A4=440 Hz

Pre-trained RAVE streaming models are available for immediate use without training

RAVE requires a CUDA GPU with 5–32 GB VRAM depending on config, and hours of audio

RAVE training requires CUDA verified via nvidia-smi and a dedicated conda environment

Stable Diffusion is a pipeline of three neural networks, not a single monolithic model

Stable Diffusion runs in two modes: text-to-image and image-plus-text (img2img)

WebGPU has three shader stages: vertex computes positions, fragment computes colors, compute runs a function N times

A diffusion model learns to reverse a fixed noise-adding process by training a neural network to denoise step-by-step

A diffusion model steers noise toward the training distribution, not toward exact stored images

A model's depth can be measured either as computation-graph length or as concept-hierarchy depth

A nn~ model's methods each define a distinct processing pipeline with its own inlet/outlet count

A VAE generates sound by encoding a spectrogram to a latent point, decoding it, and inverting the STFT

A variance schedule beta_t controls how quickly noise accumulates across DDPM timesteps

A WebGPU uniform is a shader global held constant for every invocation of one draw call

CFG scale in ComfyUI's KSampler balances prompt adherence against image quality

CFG scale in Deforum controls prompt adherence — too high saturates and distorts, too low drifts off-prompt

CLIP jointly trains text and image encoders so matching pairs get high cosine similarity

ComfyUI embeds the full workflow JSON in generated PNG files

ComfyUI models a diffusion pipeline as a directed graph of typed nodes

ComfyUI only re-executes nodes whose inputs have changed since the last run

ComfyUI supports {option|option} wildcard syntax in text prompts for random variation

ComfyUI weights a prompt term with (term:weight) syntax to strengthen or weaken it

ComfyUI's security scope is limited to localhost by design — --listen exposes the server to the network at user's risk

ComfyUI's smart memory manager offloads models to CPU when GPU VRAM is insufficient

Converting a generated log-spectrogram to audio requires denormalizing, de-logging to amplitude, then an iSTFT

Ctrl+B bypasses a node in ComfyUI as if it were removed with wires reconnected through

DDPM inference reverses the diffusion process: starting from noise, iteratively subtract predicted noise for T steps

DDPM normalizes pixel values from [0,255] to [-1,1] so the network operates on a fixed input range matching the Gaussian prior

DDPM training samples a random timestep per example, corrupts it, and minimizes noise-prediction loss

DDPM uses a U-Net with skip connections as the denoising network, taking noisy image and timestep as input

DDSP makes DSP components differentiable so neural networks can drive classical synthesizers

DDSP training requires preprocessing raw audio into TFRecord files with precomputed f0, loudness, and audio chunks

DDSP uses the CREPE pitch detector to extract frame-rate f0 and confidence from audio for training conditioning

DDSP's FilteredNoise synthesizer shapes white noise with a learned frequency-domain magnitude envelope

DDSP's Harmonic synthesizer sums band-limited sinusoidal harmonics weighted by a learned distribution

DDSP's Reverb processor can use either a predicted or a single learned impulse response for convolution

Deforum keyframe schedules interpolate parameter values linearly between defined frame:value pairs

Deforum motion operators apply per-frame and accumulate, so small values compound into large movement over an animation

Deforum parameters are scoped to an animation mode — a parameter effective in one mode has no effect in another

Deforum's strength_schedule sets how much the previous frame constrains the next and also fixes the effective step count

Demucs automatically rescales output stems to prevent clipping but this breaks relative stem loudness

Demucs separates any audio file from the command line with a single command

Demucs ships multiple model variants trading speed, quality, size, and stem count

Denoise strength below 1.0 in KSampler enables image-to-image sampling

Diffusion inference iteratively removes predicted noise from a latent tensor until an image emerges

Flok enables multiple live coders to share and edit a single browser interface in real time

Freesound's content-based similarity search returns sounds that are acoustically alike, not just similarly tagged

Griffin-Lim reconstructs a plausible phase from a magnitude spectrogram, so its output sounds robotic

Harmonic synthesizers must zero out partials above the Nyquist frequency to prevent aliasing

Latent diffusion runs the denoising process in a compressed latent space instead of pixel space, cutting compute cost

LoRA strength in ComfyUI scales the weight delta additively and can be negative or greater than 1

Music source separation splits a stereo mix into isolated stems (drums, bass, vocals, other)

nn~ exposes RAVE encode, decode, and forward as Max/MSP or Pure Data audio-rate methods

nn~ is a translation layer that runs any TorchScript (.ts) model as a live Max/MSP or Pure Data object

nn~ model attributes are model-defined, live-controllable parameters set via 'set NAME VALUE' messages

nn~'s circular buffer amortizes neural model compute across time, at the cost of added latency

Perceptual loudness in DDSP uses A-weighting to match human hearing sensitivity across frequencies

Processing audio in short segments reduces GPU memory at the cost of segment-boundary artefacts

q_sample implements the 'nice property' — corrupting an image to any noise level in one operation

Random horizontal flipping during training improves sample quality in image diffusion models

RAVE dataset quality needs a balance between homogeneity and diversity

RAVE is a variational autoencoder that encodes audio into a compact latent space and decodes it back in realtime

RAVE models must be exported with --streaming to avoid click artifacts in realtime hosts

RAVE perceptual quality must be judged by ear, since adversarial loss values do not track it

RAVE requires hours of homogeneous audio preprocessed into a chunked database before training

RAVE supports mute, compress, and gain augmentations to improve generalization on small datasets

RAVE training is monitored via TensorBoard distance, fidelity, and adversarial-loss logs

RAVE uses gin-config to define and override model hyperparameters without modifying code

RAVE's generate script applies a model to large collections of audio files offline in batch mode

RAVE's lazy dataset mode avoids disk conversion by loading raw audio files at training time

Representation learning discovers useful features from data instead of hand-designing them

Stem separation is a practical on-ramp to building personal sample banks from any released track

The DDSP Processor separates unconstrained network outputs (inputs) from physically valid synthesizer controls

The exp_sigmoid nonlinearity maps network outputs to strictly positive amplitudes with a controllable range

The noise predictor is trained by adding known noise to images and having it predict that noise

The RAVE VST loads a .ts model in any DAW as an audio effect that re-timbres incoming audio

TorchScript (.ts) files are the portable, runtime-independent format for deploying PyTorch models without Python

Transfer learning reuses a pre-trained network as a feature extractor to train a classifier from few examples

VAEs support two generation modes: reconstruction from an encoded input and sampling from the latent prior

Waveform-to-waveform nn~ models have equal in_ratio and out_ratio and pass audio through uncompressed

WebGPU draws or computes via a pipeline that chains shaders to GPU resources through bind groups

WGSL declares data with var (mutable storage), let (immutable value), and const (compile-time constant)

A biodata module turns any biological signal into Eurorack CV, gates, and audio, making living organisms into performers

A ComfyUI custom node is a Python class with INPUT_TYPES, RETURN_TYPES, FUNCTION, and CATEGORY

A DDSP ProcessorGroup chains processors as a DAG, configurable via gin without Python code changes

A deep source separation model can be wrapped in nn~ to split live audio into stems in real time

A GrooveTransformer ML model can predict a full drum pattern from a sparse hit/velocity/offset input vector

A screen region or webcam feed can be the continuous image source for real-time img2img diffusion

A single modulated delay line produces chorus, flanger, and vibrato depending on its delay-time settings

A warmup phase fills the pipeline's latent buffer and JIT-compiles kernels before timing starts

An autoencoder compresses input to a latent code whose interpolations generate smooth variations

Angular cumsum avoids phase drift in long synthesis by chunking cumulative sums and resetting with modular arithmetic

Any PyTorch model can become a nn~ object by subclassing nn_tilde.Module and registering methods and attributes

Arithmetic on a GAN's latent vectors edits generated images semantically

Averaging predictions over multiple time-shifted inputs (the shift trick) improves separation quality

Classifier-free guidance combines conditional and unconditional model outputs at inference to steer generation without a separate classifier

Classifier-guided diffusion steers generation by adding classifier-score gradients to the noise prediction

ComfyUI's HTTP API accepts workflows as JSON with nodes keyed by numeric IDs and links as [node_id, slot_index] arrays

Compiling UNet and VAE to TensorRT engines delivers the largest throughput gain in StreamDiffusion

ControlNet adds spatial conditioning to a frozen diffusion U-Net via a trainable copy connected with zero-convolution layers

DDIM makes diffusion sampling deterministic by setting the stochasticity parameter eta to zero, enabling far fewer steps

DDPM keeps the reverse-process variance fixed and only learns the mean, simplifying the training target

DDPM ResNet blocks inject the timestep embedding via scale-and-shift (FiLM-style) conditioning

DDPM's training objective is to predict the noise added to an image rather than the original image directly

DDSP timbre transfer re-synthesizes audio from one instrument using a model trained on a different instrument

DDSP's frequency_filter designs FIR filters from frequency-domain magnitude curves using IRFFT and windowing

DDSP's RnnFcDecoder runs independent FC stacks per conditioning input, concatenates, then runs an RNN

DDSP's Wavetable synthesizer reads a learned single-cycle waveform at a time-varying phase

Deforum's 3D FOV scales how fast translation_z moves the canvas, with defined edge cases at 0, 180 and negative values

Deforum's anti-blur applies an unsharp mask to counteract the progressive blurring that builds during long animations

Deforum's Perlin noise injection adds organic, spatially coherent variation to frames rather than uniform random noise

Demucs exposes a Python API for integrating stem separation into scripts and pipelines

Demucs uses a weighted ensemble (BagOfModels) of individually trained checkpoints for best performance

Demucs' internal separation output is a 4-D tensor shaped [batch, sources, channels, time]

Diffusion models are attractive because they are simultaneously analytically tractable and flexible

Diffusion models learn to reverse a Markov chain that progressively corrupts data with Gaussian noise

Diffusion models trade sampling speed for stable training and broad mode coverage relative to GANs and VAEs

Exponential Moving Average of weights produces smoother RAVE models by averaging checkpoints over time

Extending stem separation to guitar and piano is harder than the standard four-stem split

Feature matching loss in RAVE aligns intermediate discriminator activations between real and generated audio

Frequencies softmax maps network logits to Hz by weighting a log-spaced frequency grid

Fusing LCM-LoRA into an SD model enables 2–4 step diffusion inference without retraining the base model

Gin's @gin.register limits side-effects by only injecting defaults when a function is used as an argument

Group normalization divides channels into groups and normalizes within each group, working well with small batch sizes unlike batch normalization

Hybrid source separation processes audio in both waveform and spectrogram domains simultaneously

img2img and txt2img mode in StreamDiffusion differ in their latent initialization and CFG constraints

in_ratio and out_ratio in nn~ method registration define the temporal compression between audio and model outputs

Key DDPM follow-up works include improved variance learning, cascaded generation, classifier guidance, and classifier-free guidance

Linear attention scales O(n) in sequence length — making it practical for image feature maps without the O(n^2) cost of full attention

Manipulating individual latent dimensions of a RAVE model morphs continuously between audio-effect and synthesizer behavior

mc.nn~ processes multiple audio channels through one RAVE model instance to cut CPU and RAM

mc.nn~ runs several audio streams through one model as a batch, saving CPU and RAM versus duplicating nn~

mcs.nn~ packs all of one instance's inlets/outlets into a single multi-channel connection, enabling per-batch latent operations

Multi-scale SpectralLoss compares audio at multiple FFT sizes to balance time and frequency resolution

Multiple LoRA weights can be loaded and fused into StreamDiffusion before inference for style mixing

MUSDB-HQ is the standard benchmark dataset for music source separation research

nn~ can output audio-analysis features as control-rate signals by using a large out_ratio

nn~ is a Max/PureData external that bridges trained neural audio models (RAVE, vschaos2) into a patching environment

nn~'s void/lazy mode lets you fix inlet/outlet count before attaching a model

Pre-computing KV-cache for the text prompt eliminates repeated text-encoder work across frames

Quantization maps high-precision weights to lower-precision formats using a scaling factor to preserve dynamic range

Raising RAVE's discriminator update period fixes phase-2 instability when the discriminator is too strong

RAVE architecture configs trade reconstruction quality, GPU memory, and control modality

RAVE can be exported to ONNX format for deployment in environments that do not support TorchScript

RAVE ramps the KL regularization weight over a warmup schedule to avoid posterior collapse

RAVE trains in two sequential phases: reconstruction first, adversarial refinement second

RAVE uses Pseudo Quadrature Mirror Filters to split audio into sub-bands before encoding

RAVE v3 uses Adaptive Instance Normalization to transfer timbre from one audio stream to another

RAVE's causal convolution mode lowers latency at the cost of quality by removing future context

RAVE's discrete mode quantizes latent vectors using Residual Vector Quantization

RAVE's exported latent size is chosen by a fidelity threshold on the PCA explained-variance curve

RAVE's phase 1 length is a fixed step count, not a quality-based stopping criterion

RAVE's temporal receptive field sets the minimum audio context and chunk length it can process

RAVE's variational encoder reparametrizes latent samples using the mean and log-variance trick

Replacing the standard VAE with TinyVAE (TAESD) reduces encode/decode latency at small quality cost

Residual CFG approximates classifier-free guidance at near-zero extra cost by recycling a stored noise residual

SDR (signal-to-distortion ratio) is the standard metric for evaluating stem separation quality

Separating image generation and display into different OS processes prevents Python's GIL from throttling GPU throughput

Splitting RAVE encode and decode in nn~ lets performers process individual latent dimensions live

Stable-Fast compiles the full diffusion pipeline with Triton + CUDA graphs for an intermediate speed tier

StreamDiffusion achieves 106 fps txt2img and 93 fps img2img on RTX 4090 with SD-Turbo plus TensorRT

StreamDiffusion achieves real-time AI image generation by batching denoising steps and skipping redundant frames

StreamDiffusion can process offline video frame-by-frame with img2img to produce a stylised output video

StreamDiffusion requires CUDA GPU, Python 3.10, PyTorch 2.1, and optional TensorRT for optimal performance

StreamDiffusion's IO queues decouple the input capture rate from the diffusion throughput via multiprocessing

StreamDiffusion's stream batch runs multiple denoising steps in parallel to sustain real-time frame rates

StreamDiffusionWrapper provides a high-level production interface; the raw StreamDiffusion class gives full control

t-SNE embeds high-dimensional feature vectors in 2D so perceptually similar items cluster together

TAESD provides fast high-quality latent previews without running the full VAE

Text conditioning is injected into the UNet by attention layers placed between ResNet blocks

The 'nice property' lets you sample any diffusion timestep directly from the original image without iterating the forward process

The DDPM ELBO decomposes into per-timestep KL divergences between Gaussians, each computable as an L2 loss

The DDPM U-Net assembles encoder, bottleneck, and decoder stages using ModuleList, with skip connections via concatenation

The DDSP autoencoder encodes loudness and f0 plus a learned latent z, then decodes to synthesizer parameters

The denoising U-Net uses sinusoidal position embeddings to communicate the current noise level (timestep) to every layer

The forward diffusion process adds scheduled Gaussian noise over T steps until the signal becomes isotropic noise

The forward method runs a neural model as an audio effect: audio in, neural-transformed audio out

The reverse diffusion process is a neural network that approximates the intractable denoising posterior at each timestep

The Stochastic Similarity Filter skips GPU work probabilistically when consecutive frames are nearly identical

The StreamDiffusionTD operator wraps StreamDiffusion as a TouchDesigner node for diffusion-based real-time visuals inside a TD network

The t_index_list selects which diffusion timesteps to apply, controlling quality–speed trade-off

TouchDesigner can host real-time AI/ML pipelines for generation, tracking, and stylisation in a live AV rig

U-Net's symmetric downsampling-upsampling structure with skip connections makes it a standard diffusion backbone

unCLIP generates images by mapping text to a CLIP image embedding, then decoding that embedding to pixels with diffusion

WebGPU compute threads are grouped into workgroups and addressed by a global invocation id

xformers memory-efficient attention reduces VRAM usage in diffusion UNets at modest speed cost

A cosine noise schedule gives gentler early-stage transitions than the linear schedule, improving likelihood

A Markov chain trained on pitch and rhythm sequences generates new music with the same statistical patterns

A RAVE prior is a model trained on latent sequences that enables generation without audio input

BPM- and pitch-aligned stem cross-mixing creates more realistic training data than random remixing

Bridging a genetic-algorithm simulator to TouchDesigner via Python enables real-time visualisation of evolutionary computation

Cascaded diffusion chains models at increasing resolutions, using noise-conditioning augmentation between stages

Consistency models map any point on a diffusion trajectory directly to the trajectory origin, enabling single-step generation

Custom separation model bags let you combine multiple checkpoints with per-source weights

DDPM trains a network to predict the added noise epsilon rather than the clean sample or the reverse mean

DDSP's inverse synthesis approach detects pitch without pitch labels by reconstructing synthetic audio

Diffusion Transformers replace the U-Net backbone with a Vision Transformer operating on patchified latents

Dora manages ML experiment configurations via content-addressed signatures rather than manual naming

Dropping per-timestep loss weights and training with uniform MSE on noise prediction improves DDPM in practice

Every KL term in the diffusion VLB compares two Gaussians and is therefore analytically tractable

Imagen conditions image generation on a frozen large language model, and scaling that encoder matters more than scaling the U-Net

Learning the reverse-process variance as an interpolation between two fixed endpoints improves likelihood

Live AV systems range from fully autonomous algorithmic performance to semi-autonomous systems guided from a higher level

On-the-fly pitch and tempo shift augmentation improves separation model generalisation across musical keys and tempos

Progressive distillation halves the required sampling steps by training a student to match two teacher steps in one

Randomly remixing stems from different songs during training forces the model to learn true source priors

Score-based models learn the gradient of the log data density and generate samples via Langevin dynamics

Some live coding systems fold machine learning and machine listening into the performance, letting the coder train and steer models live

The diffusion training objective decomposes into per-timestep KL divergence terms plus a reconstruction term

The reparameterization trick lets you sample a noisy x_t at any timestep t directly without simulating the full chain