Hybrid source separation processes audio in both waveform and spectrogram domains simultaneously

Early neural separation models chose one audio representation: waveform (raw samples in time) or spectrogram (frequency-over-time magnitude). Hybrid Demucs runs parallel branches — one temporal branch operating on raw waveform, one spectral branch operating on STFT magnitude — and fuses them in a U-Net encoder/decoder. Each domain captures complementary information: the waveform branch is sensitive to fine timing and transients; the spectral branch captures harmonic content more efficiently. The v4 Hybrid Transformer Demucs adds cross-domain Transformer layers between the encoder and decoder, allowing self-attention within each domain and cross-attention across them. This hybrid approach consistently outperforms pure-domain models at equivalent parameter budgets.

Examples

The HTDemucs architecture diagram shows a dual U-Net: one branch for temporal domain, one for spectral domain, connected by a cross-domain Transformer Encoder at the bottleneck.

Assessment

Explain the role of each domain branch (waveform vs spectrogram) in a hybrid model. Why is cross-attention across domains beneficial at the bottleneck?