Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
2026-07-12 • Sound
SoundMultimedia
AI summaryⓘ
The authors address the challenge of creating music that matches dance movements, focusing on using human joint positions to represent the dance. Because there isn't much good paired data of dance and music together, they develop a method that uses both paired and unpaired data to train their system. Their approach combines existing models for music and motion and aligns their understanding with special training guided by beats. Tests show their method better matches music to dance movements and keeps good sound quality compared to previous methods.
dance-to-music generationhuman joint positionspaired dataunpaired datacontrastive pretrainingControlNetdiffusion modelbeat alignmenttext-to-audio synthesisAIST++ dataset
Authors
Ryota Kimura, Sangheon Park, Natalia Polouliakh, Taketo Akama
Abstract
Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .