AI summaryⓘ
The authors study how large models that understand both images and text can work well with new types of images they haven't seen before. They treat different image types as just different views of the same real-world scenes and teach models to focus on meaning rather than appearance differences. Their method, VVM-Tuning, uses fake images with various looks to help models learn to separate what an image shows from how it looks, and then guides the model to recognize these differences using special prompts. They also created a new test called VVM-Bench to check if models really understand different image types. Their experiments show that their approach helps models better handle unseen visual styles without extra training on those styles.
Large Multimodal ModelsVisual ModalitiesModality-Agnostic PerceptionZero-Shot AdaptationModality SynthesisInstruction TuningSemantic PerceptionMultimodal BenchmarkScene SemanticsVisual-Language Alignment
Authors
Shihao Yuan, Yuanze Li, Ruyi Zhang, Ming Liu, Wangmeng Zuo
Abstract
Despite the advancements of Large Multimodal Models (LMMs) in RGB vision, their ability to generalize to unseen visual modalities remains a largely unexplored challenge. We argue that different visual modalities are merely distinct samplings of the same physical world. Therefore, effective generalization requires models to possess both modality-agnostic perception of scene semantics and the adaptability to modality-specific characteristics. To achieve this, we propose a training framework, VVM-Tuning, to equip LMMs with these capabilities through modality synthesis and modality contexts. Specifically, we synthesize diverse appearance-varied images from RGB scenes, training the model to disentangle invariant semantics from varying visual appearances, and align these appearances with language for visual concepts decoupled from modalities. We then introduce modality contexts in the prompt and use instruction tuning to assist the model in mapping these appearance variations back to modality-related attributes, enabling zero-shot adaptation to unseen modalities during inference. To facilitate research in this direction, we introduce VVM-Bench, a comprehensive benchmark featuring 6 real and synthetic modalities to evaluate semantic perception and modality understanding. Experiments demonstrate that, via our training on synthetic modalities, 5 tested models exhibit consistent improvements on both real-world and novel synthetic modalities without in-modality training. Source code and data will be publicly available at https://github.com/Hunter-Will/VVM-Tuning.