Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
2026-06-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors found that existing ways to measure how well vision-language models understand medical images and their descriptions aren't good at showing which part (image or text) causes problems. They created a new score called the Spectral Alignment Score (SAS) that looks at how each part aligns separately and shows differences clearly. Testing 15 models, they saw that medical images have more detailed structure than the related doctors' notes, which older metrics missed. SAS also matches well with how well the models retrieve information, making it useful for checking these models in healthcare.
Vision-Language ModelsMedical ImagingRepresentation AlignmentSpectral Alignment ScoreEigenbasisCross-modal DegradationRetrieval PerformanceDirectional MetricsBiomedical TextBenchmarking
Authors
Alessandro Gambetti, Qiwei Han, Cláudia Soares, Hong Shen
Abstract
Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited. Existing representation alignment metrics are symmetric, collapsing both modalities into a single score and hiding which modality drives cross-modal degradation. We introduce the Spectral Alignment Score (SAS), an asymmetric metric that projects both modalities onto the principal eigenbasis of an anchor modality and computes eigenvalue-weighted per-eigenmode correlations, resulting in directional scores whose difference quantifies modality information imbalance. We embed SAS within a benchmarking framework evaluating 15 VLMs across natural and medical image-text datasets alongside 6 alignment metrics and bidirectional retrieval. Our experiments show that medical images retain richer structural information than their paired clinical reports, a directional asymmetry invisible to all competing metrics, and that SAS achieves the strongest zero-label correlation with retrieval performance in the medical domain, positioning it as a practical diagnostic tool for clinical deployment. Code is available at this URL: https://github.com/iamalegambetti/medical-vlms-assessment.