AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionSound
AI summaryⓘ
The authors created AV-SyncBench, a new way to test how well audio and visual features work together in videos. Unlike older tests, their benchmark separates checking if sounds match the right visuals (semantic) from checking if the timing between audio and video is correct (temporal). They built the test using real videos with verified sounds on screen, covering voice, music, and other sounds across multiple scenarios. They also tested five models to see how good they are at syncing audio and video. The dataset and code are openly available for others to use.
audio-visual synchronizationfeature extractionsemantic matchingtemporal offset detectionmultimodal understandingbenchmarkin-the-wild videosaudio-visual alignmentdatasetmachine learning models
Authors
Tianhong Zhou, Mingyang Han, Boyu Li, Yuxuan Jiang, Jiaxin Ye, Dongxiao Wang, Haoxiang Shi, Kunpeng Wang, Jun Song, Cheng Yu, Bo Zheng
Abstract
Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for feature extraction models exhibit dimensional bias, typically focusing on either semantic matching or temporal offset detection. Moreover, their data construction remains coupled, preventing independent assessment of temporal and semantic consistency. We propose AV-SyncBench, the first benchmark to fully separate temporal and semantic evaluation for audio-visual synchronization. Built from in-the-wild videos, it spans Voice, Music, and Sound across 10 scenarios and 5 challenge tasks. Data are automatically filtered and manually verified to ensure on-screen sound sources. The benchmark contains 3,269 videos and 38,390 samples, and we evaluate five representative models to quantify feature quality for alignment and downstream tasks. The code and dataset are available at: https://fgt7t6g.github.io/AV-SyncBench.