$C^3$ASD: Multi-Level Consistency-Driven Representation Learning
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionSound
AI summaryⓘ
The authors study how to better detect when someone is speaking in a video, even if the audio or video is messy or unclear. They find that existing methods fail because they don't ensure the audio and video parts work well together. To fix this, the authors introduce a system called C3ASD that teaches the model to keep audio and video signals consistent at different levels. Their experiments show this approach improves detection in noisy or corrupted conditions, without losing accuracy on clear videos.
Active Speaker DetectionAudio-Visual FusionRepresentation LearningContrastive LearningKnowledge DistillationMultimodal ConsistencyRobustnessSpeech DetectionSemantic AlignmentCorrupted Data
Authors
Jin Hong, Jisoo Park, Junseok Kwon
Abstract
Active Speaker Detection determines whether a visible person in a video is speaking at each moment. While recent audio-visual fusion methods perform well on clean data, they degrade under real-world corruptions such as background noise, occlusion, or simultaneous modality degradation. We attribute this limitation to the absence of explicit consistency constraints that promote robust, semantically aligned representations across modalities. Without such guidance, models tend to learn fragile modality-specific shortcuts that fail under corrupted conditions. We propose $C^3$ASD, a multi-level consistency-driven framework with three complementary constraints: embedding-level inter-modality consistency aligns audio-visual representations during speech; sequence-level intra-modality consistency separates speaking and non-speaking clusters via track-aware contrastive learning; and prediction-level consistency stabilizes fusion through knowledge distillation. Extensive experiments demonstrate significant improvements under diverse audio, visual and joint corruptions, while maintaining competitive performance on clean data.