DDSP timbre transfer re-synthesizes audio from one instrument using a model trained on a different instrument

Timbre transfer in DDSP works by running the encoder (f0 extraction + loudness + z) on a source audio signal (e.g., a voice or a scratch), then feeding those conditioning signals to a decoder trained on a target instrument (e.g., violin or flute). The result is a synthesis that follows the pitch and dynamics of the source but has the timbre of the target instrument. This works because f0 and loudness are instrument-agnostic; only z carries instrument-specific information, which can be held fixed or discarded. The Timbre Transfer Colab is the primary demo: users upload audio or sing into their microphone and hear it re-synthesized as various instruments.

Examples

Load a pretrained model and transfer timbre:

from ddsp.training import inference
model = inference.AutoencoderInference(ckpt_path)
# Extract f0, loudness from source audio
# Decode with target instrument decoder
audio_out = model(source_audio_features)

Assessment

Explain why you can use CREPE f0 from a human voice to drive a violin synthesizer. What would you do with the z latent to maximize timbral transfer vs. retaining some source characteristics?