Benchmarking Single-Factor Physical Video-to-Audio Generation
2026-05-28 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMultimediaSound
AI summaryⓘ
The authors created FlatSounds, a new test to check if video-to-audio models really understand physical events in videos, not just make realistic sounds. They designed special video pairs that change one physical thing at a time and tests to see if the sound changes correctly with the video. Their tests show current models often depend more on text captions than the actual video to guess the sounds, which helps with accuracy but can mess up timing. The authors suggest that future models should learn physical details directly from the visuals. They also found their physical reasoning tests match well with what people prefer when judging sound.
Generative modelsVideo-to-audio synthesisPhysical reasoningCounterfactual pairsTemporal alignmentSemantic accuracyControlled interventionsVisual streamText captionsBenchmark evaluation
Authors
Tingle Li, Siddharth Gururani, Kevin J. Shih, Gantavya Bhatt, Sang-gil Lee, Zhifeng Kong, Arushi Goel, Gopala Anumanchipalli, Ming-Yu Liu
Abstract
Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emphasize perceptual realism and overlook physical correctness under controlled interventions. In this paper, we introduce FlatSounds, a benchmark that audits the physical reasoning of V2A models through: 1) controlled counterfactual pairs in which a single physical factor is varied, and 2) single-video pattern tests that probe internal consistency and directional trends. These settings test whether the generated audio correctly reflects specific physical properties and timings. Our evaluation of state-of-the-art models reveals a consistent trade-off: models rely more on text captions than the visual stream to infer physics and semantics. Captions generally improve physical and semantic accuracy, but paradoxically degrade temporal alignment. Our results highlight the need to move beyond audio quality toward learning physical processes directly from pixels. Finally, we find that our physics-based metrics correlate strongly with human preference tests on our own data. Project webpage: https://research.nvidia.com/labs/cosmos-lab/flatsounds/