DDSP uses the CREPE pitch detector to extract frame-rate f0 and confidence from audio for training conditioning

DDSP requires fundamental frequency (f0) as a conditioning signal for the harmonic synthesizer. It uses CREPE (a pretrained convolutional neural network) to estimate per-frame pitch and a confidence score from raw 16kHz audio. compute_f0() pads the audio, runs CREPE’s pitch prediction, and returns f0_hz and f0_confidence arrays. Viterbi decoding (default: True) uses an HMM to smooth noisy pitch tracks. CREPE is non-differentiable and runs at pre-processing time; during training, f0 is treated as a fixed conditioning signal. The confidence score is important: low-confidence frames (background noise, silence) can be downweighted in the loss.

Examples

import ddsp
f0_hz, f0_confidence = ddsp.spectral_ops.compute_f0(
    audio,          # numpy array, 16kHz
    frame_rate=250, # Hz
    viterbi=True)

Assessment

Why is CREPE run at pre-processing time rather than as part of the training graph? When should you weight frames by f0_confidence in your loss?