home/ modules/ engineering-ddsp-timbre-transfer

Engineering DDSP Timbre Transfer Internals

L3Craft 5h K · AI & real-time generative AV 13 atoms

Learning objectives

learner can trace the DDSP autoencoder from loudness/f0/latent encoding to synthesizer parameters
learner can diagnose harmonic/noise rendering artifacts using the DSP internals: frequency softmax, FIR filter design, angular cumsum, multi-scale spectral loss
learner can configure a ProcessorGroup DAG via gin and explain label-free pitch detection by inverse synthesis

Capstone — one whole task that evidences the objectives

Run a pretrained DDSP model to re-synthesize a supplied vocal line as a saxophone and deliver the annotated render: enable oscillator_bank.use_angular_cumsum=True for the long take, adjust the ProcessorGroup DAG in the gin config, diagnose one phasey or buzzy artifact by pointing to the responsible internal (frequency softmax, FIR frequency-filter design, or a multi-scale spectral-loss scale), and annotate the render with the autoencoder + RnnFcDecoder signal path plus how the same model recovers pitch without labels via inverse synthesis.

Prerequisite modules

Building Differentiable DSP Instruments with DDSP

Timbre transfer is DDSP’s signature trick — sing into a microphone and hear a saxophone sing back — and it is increasingly a live tool: performers route a vocal or contact-mic feed through a pretrained model as a mutant “instrument” in an electronic set. To use it beyond the Colab demo, or to debug why a transfer sounds phasey, buzzy, or pitch-drifty, you need to understand the machine end to end. This module builds toward exactly that: actually running a vocal-to-saxophone transfer, tuning its gin configuration, and diagnosing its artifacts — with an annotation layer that explains every stage that makes it work.

Start supported: with the differentiable-instruments prerequisite fresh, trace the encode–decode–synthesize pipeline using the DDSP autoencoder atom, then open up the decoder with the RnnFcDecoder atom to see how loudness, f0, and the latent z each get their own FC stack before the GRU merges them. Next drill into the DSP internals as JIT how-tos — the frequency-softmax atom for differentiable Hz prediction, the frequency-filter FIR atom for the IRFFT-plus-Hann design, and the angular-cumsum atom for why long renders stay in tune. Finish the arc with the multi-scale spectral loss that trains it all, the ProcessorGroup DAG that wires it in gin, and inverse-synthesis pitch detection — then produce the capstone render, configuration, and annotation unassisted.

Each required atom gates a named piece of the capstone; without it the render, the gin adjustment, the artifact diagnosis, or a section of the annotation collapses. The supporting atoms enrich: the harmonic and filtered-noise synthesizers ground the processor group’s leaf nodes, the wavetable synthesizer shows an alternative oscillator, and the gin registration conventions explain how the DAG stays code-free.