Local Multimodal Music Alignment from Global Supervision
2026-07-10 • Sound
SoundMachine LearningMultimedia
AI summaryⓘ
The authors created a method called FuSiLi to better understand how pieces of music relate across different types of data, like audio and images of sheet music. Normally, it's hard to match small parts precisely because we only have overall matches between whole audio and images. FuSiLi works by softly matching tiny parts of audio with small parts of images, using a mathematical technique called Sinkhorn alignment. The authors showed that their method learns these local connections well using only broad supervision, while still doing well on global matching tasks.
multimodal learningcontrastive learningSinkhorn algorithmlocal alignmentcross-modal retrievalsheet music recognitionaudio segmentationCLIP encoderCLAP encodersoft alignment
Authors
Irmak Bukey, Zachary Novack, Jongmin Jung, Dasaem Jeong, Chris Donahue
Abstract
Understanding music requires understanding localized relationships across data modalities, e.g., how time in performance audio maps onto position in a score image. Yet supervision for such local correspondences is difficult to obtain-in practice, we often only have access to coarser global supervision like paired segments of audio and images. To address this gap, we propose FuSiLi (Fused Sinkhorn-Localized Similarity), a similarity score for multimodal contrastive learning operating directly on local image patch and audio frame features via Sinkhorn-based soft alignment. We show that FuSiLi (i) effectively learns local relationships, (ii) requires only global supervision, and (iii) retains the global alignment capabilities of conventional contrastive approaches. We fine-tune pretrained CLIP and CLAP encoders on pairs of raw sheet music images and audio using a hybrid contrastive objective combining FuSiLi with conventional global similarity. We evaluate on cross-modal retrieval and frame-level alignment tasks against a range of global and local baselines, showing that our approach outperforms them on local alignment while remaining competitive on retrieval.