ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models
2026-07-10 • Sound
SoundArtificial Intelligence
AI summaryⓘ
The authors studied a method called representation alignment in diffusion models and found it can mix up important information, limiting how well the models create sounds. To fix this, they developed ReGen, a system that generates multiple representations at once to better handle the data inside a single model. They also improved a technique called generalized flow matching to make the model work better with different conditions. Their approach significantly improved audio quality in tasks like waveform generation and text-to-speech, even when using small datasets and faster training. They demonstrated fast and efficient speech synthesis with good clarity and voice similarity.
representation alignmentdiffusion transformerslatent representationsgeneralized flow matchingwaveform diffusionneural audio codectext-to-speechlatent diffusion modelspeaker similarityreal-time factor
Authors
Sang-Hoon Lee, Ha-Yeong Choi
Abstract
Representation alignment (REPA) has been investigated to accelerate diffusion training, but we observe that regularizing intermediate representations in diffusion Transformers (DiT) may implicitly entangle latents and limit generative capacity. To address this issue, we propose ReGen, a hierarchical multi-prompt representation generation framework that jointly estimates multiple vector fields for both representations and data within a single diffusion model. We further introduce generalized flow matching (GFM) to improve the generalization of conditional flow matching (CFM). We validate ReGen on single-stage waveform diffusion models including neural audio codec and Wave-VAE. ReGen significantly improves waveform generation quality from highly compressed latent representations at 12.5 Hz. We also present ReGenVoice, a latent diffusion model (LDM)-based text-to-speech model that achieves strong speech intelligibility (WER) and speaker similarity (SIM) with a small dataset. Moreover, operating the LDM at 6.25 Hz with rich semantic and acoustic latent representation enables efficient training and sampling, requiring only 1 day of training on 4 GPUs and fast inference with an RTF of 0.08. Audio samples are available at https://regenvoice.github.io/demo/.