MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
2026-07-12 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors developed a method called MRUF to improve how computers understand emotions from speech, videos, and text together. They noticed that sometimes one type of information (like audio or video) can be unclear or noisy, so MRUF learns to trust clear information more and less reliable details. It does this by estimating how uncertain each type is and adjusting the importance of each before combining them. Their tests on popular emotion datasets showed MRUF works better than older methods, especially by giving less weight to uncertain inputs.
Multimodal sentiment analysisModality reliabilityUncertainty calibrationMulti-granularity routingFusion methodsContrastive alignmentLeave-one-out errorCMU-MOSI datasetCMU-MOSEI dataset
Authors
Haoran Ma, Yinfeng Yu, Liejun Wang
Abstract
Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights.