MIND: Multimodal Intent-Driven Network via Diffusion Transformers for Medical Image Fusion
2026-07-30 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a new method called MIND that combines medical images from different sources in a smarter way by understanding the purpose of diagnosis and specific disease features. They used an AI tool, BioMedGPT, to guide this combination based on diagnostic intentions. To keep the image details accurate, they designed special modules that maintain spatial and semantic information during processing. Their tests showed better image fusion quality and improved brain tumor detection, suggesting this method could help doctors make better decisions.
medical image fusiondiagnostic intentdiffusion transformersBioMedGPTspatial continuitysemantic consistency lossbrain tumor segmentationmultimodal imaginglatent adapterintent-driven fusion
Authors
Yunzhan Fu, Xiangyu Shen, Yifei Sun, Yuhan Chen, Jian Wu, Hongxia Xu
Abstract
Medical image fusion aims to integrate complementary information from diverse imaging modalities to support clinical diagnosis. Existing methods typically apply uniform fusion rules globally, lacking a deep understanding of diagnostic intents and pathological structures. To address these limitations, we propose MIND, a Multimodal Intent-Driven Network via Diffusion Transformers (DiTs) for medical image fusion. Specifically, we utilize BioMedGPT to generate intent-driven fusion texts from source images, guiding the fusion process with pathology-aware diagnostic intents. To combat the loss of 2D spatial continuity caused by 1D sequence flattening in DiTs, we design a Multi-scale Latent Adapter. This module explicitly extracts source image features before serialization, injecting them into the network via strict dimensional alignment to effectively supplement image features. To resolve the semantic shift caused by decoupling image outputs from diagnostic intents, we design a medical semantic consistency loss. This loss ensures deep semantic locking between fused images and fusion texts while maintaining the stability of the underlying physical manifold reconstruction. Comprehensive experiments on the Harvard, BraTS, and GFP datasets reveal that MIND delivers superior fusion quality, significantly improves downstream brain tumor segmentation accuracy, and enables flexible interactive fusion, holding significant promise for intent-driven intelligent clinical decision support systems.