The DDSP autoencoder encodes loudness and f0 plus a learned latent z, then decodes to synthesizer parameters

The DDSP autoencoder follows an encode-decode-synthesize pipeline. The preprocessor extracts loudness (A-weighted dB) and pitch (f0 via CREPE) from the input audio. A MFCC-based encoder produces a latent z that captures timbral variation unexplained by f0 and loudness. An RNN+FC decoder (RnnFcDecoder) maps the three conditioning signals (loudness, f0, z) through separate fully-connected stacks, merges them, runs an RNN to model temporal dependencies, and outputs synthesizer parameters (amplitudes and harmonic_distribution). These drive a Harmonic+FilteredNoise+Reverb processor group. The multi-scale SpectralLoss trains the whole system. With ~10 minutes of a solo instrument, the model learns to re-synthesize that instrument’s timbre.

Examples

Train a model on your own audio:

ddsp_prepare_tfrecord --input_audio_filepatterns=/path/to/*.wav \
  --output_tfrecord_path=/path/to/dataset.tfrecord
ddsp_run --mode=train --gin_file=models/solo_instrument.gin \
  --gin_file=datasets/tfrecord.gin

Assessment

Trace the data flow from raw audio to synthesized audio in the DDSP autoencoder. What role does the latent z play that f0 and loudness do not capture?