MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
2026-08-11 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
AI summaryⓘ
The authors point out that current multimodal language models learn by linking whole images to long text descriptions, which can be confusing when multiple objects are involved. To fix this, they introduce MultiModal Code-Switching (MMCS), which mixes visual objects directly into text, helping the model understand exactly which object matches which word. They created a large dataset with clear object-to-word links and showed that MMCS learns more efficiently and improves the model's ability to recognize and locate objects in images. Overall, their method makes training smarter and more precise without needing tons of data.
Multimodal Large Language ModelsModality AlignmentVisual GroundingCode-SwitchingObject-Level SupervisionImage-Text PairsData SynthesisSemantic Grounding
Authors
Changhao Xiang, Shangyu Xing, Zhen Wu, Jianbing Zhang, Xinyu Dai
Abstract
Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions. However, this image-level alignment suffers from referential ambiguity: models struggle to infer the correspondences between multiple visual objects and textual entities from the global representation, leading to data inefficiency and suboptimal semantic grounding. To address this, we propose MultiModal Code-Switching (MMCS), a novel pretraining paradigm that provides explicit object-level supervision. Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding. We further develop a scalable data synthesis pipeline to generate a pretraining dataset of 773K samples with accurate object-entity correspondences. Experiments show that MMCS is highly data-efficient: with only 50K samples, it matches or surpasses models trained on 600K image-text pairs. Furthermore, MMCS consistently improves visual grounding and perception capabilities across varying model scales.