DDSP's inverse synthesis approach detects pitch without pitch labels by reconstructing synthetic audio

The ICML 2020 workshop paper introduced self-supervised pitch detection using DDSP by inverse audio synthesis: a transcribing autoencoder is pretrained on a large synthetic dataset of harmonic notes (10M examples, generated with known f0 by ddsp_generate_synthetic_dataset) so it learns to predict pitch without any real-audio pitch labels. It is then fine-tuned on small real-instrument datasets. Because the synthetic data is created by DDSP’s own differentiable synthesizer, the ground-truth f0 comes for free, side-stepping expensive manual pitch annotation. The reproduction recipe is a three-step pipeline: generate the synthetic dataset, pretrain the model on it, then fine-tune on real recordings.

Examples

Self-supervised pitch training pipeline:

# 1. Generate synthetic dataset
ddsp_generate_synthetic_dataset ...
# 2. Pretrain on synthetic
ddsp_run --gin_file=papers/icml2020/pretrain_model.gin ...
# 3. Fine-tune on real audio
ddsp_run --restore_dir=/pretrained_ckpt --gin_file=papers/icml2020/finetune_model.gin ...

Assessment

Why does pretraining on synthetic data (with known pitch labels) enable pitch detection on real audio without real pitch labels? Why is DDSP’s own differentiable synthesizer a natural source of that labeled synthetic data?