Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction
2026-07-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors created a new way for computers to use three types of medical data—numbers from health records, doctors' notes, and chest X-ray images—to make predictions about patient health. Their method lets them see exactly how each type of data helps in those predictions, making it easier to understand the model's decisions. They also tested what happens when some data types are missing, without needing to retrain the model. By checking these changes, they learned how different data sources influence the results for various health conditions.
Electronic Health RecordsMultimodal LearningClinical PredictionInterpretabilityMedical ImagingNatural Language ProcessingChest X-raysMIMIC-IVPhenotype PredictionICU Mortality Prediction
Authors
Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta
Abstract
Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.