home/ atoms/ rave-lazy-dataset

RAVE's lazy dataset mode avoids disk conversion by loading raw audio files at training time

Normal preprocessing decodes and resamples all audio into fixed-length chunks stored on disk before training begins — fast training, high disk usage. Lazy mode instead stores only file paths and metadata; decoding happens on-the-fly during training. This dramatically reduces disk requirements for large corpora that would not fit on a hard drive when uncompressed. The tradeoff: CPU load during training increases by a large margin, especially on Windows. Use rave preprocess --lazy to enable.

Examples

rave preprocess --input_path /audio --output_path /db --lazy. Useful when a large mp3/ogg corpus would balloon if expanded to wav.

Assessment

A user has 200 GB of flac files and limited disk space. Should they use lazy preprocessing? What is the performance cost and on which OS is it worst?

“lazy dataset loading will increase your CPU load by a large margin during training, especially on Windows”
corpus · rave-realtime-audio-variational-autoencoder-train-your-own-n · chunk 1