AI summaryⓘ
The authors studied ways to combine different types of data (like video and audio) to recognize emotions or sentiment. They focused on a method called XAI-guided adaptive fusion (XGAF), which mixes input from separate experts based on explainable AI (SHAP) values. They found that how the SHAP values are combined matters, especially when different experts use data of varying sizes. Their results show that using the sum of absolute SHAP values helps keep the important cross-modal experts active, leading to accuracy nearly as good as early fusion methods and better than late fusion. They concluded that adding cross-modal experts is more important than complicated sample-based routing for improving performance.
multimodal fusionearly fusionlate fusionexplainable AISHAP valuescross-modal expertsemotion recognitionsentiment recognitionTreeSHAPadaptive fusion
Abstract
Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors. Early fusion can be accurate but monolithic, while late fusion is modular but may lose cross-modal interactions. This paper revisits XAI-guided adaptive fusion (\xgaf), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes. We focus on the effect of SHAP attribution reduction when experts have unequal feature dimensionalities. In this setting, mean-abs and median-abs reductions can suppress high-dimensional cross-modal experts, whereas sum-abs reduction preserves total attribution mass. On MELD 7-class emotion recognition, sum-abs \xgaf{} nearly matches early fusion across three face-sequence aggregators; the Transformer variant reaches 0.5983 \wf{}, compared with 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs \xgaf{} and early fusion on MELD ($p=1.000$), while \xgaf{} remains significantly better than late fusion ($p<0.0001$). On CMU-MOSEI 3-class sentiment recognition, sum-abs \xgaf{} reaches 0.6519 \wf{}, slightly exceeding early fusion (0.6485) and late fusion (0.5696). Ablation studies show that the main gain comes from adding cross-modal experts, especially the trimodal expert, rather than from complex per-sample routing. Diagnostics further show that mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate on the trimodal expert. Thus, the main contribution is a transparent empirical analysis of how SHAP reduction, expert dimensionality, and cross-modal expert design affect modular multimodal fusion.