Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
2026-03-19 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors explore using a different type of visual processing model called a state space model (SSM) instead of the usual transformer models for vision-language tasks. They find that SSMs can perform better or just as well as transformers on tasks like answering questions about images and locating objects, even when using smaller models. Training the models on tasks like detection and segmentation improves both types. They also find that bigger models or higher accuracy on standard image tests don't always mean better vision-language performance. Finally, they suggest ways to make these models more stable, showing SSMs as a promising alternative for combining vision and language.
vision-language modelsstate space model (SSM)transformerImageNet-1Kvisual backboneVQA (visual question answering)grounding/localizationdetection trainingsegmentation trainingmodel stability
Authors
Shang-Jui Ray Kuo, Paola Cascante-Bonilla
Abstract
Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to transformer-based vision encoders in VLMs.