Visual Instruction Tuning Aligns Modalities through Abstraction
2026-06-02 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs) learn to understand images when tuned for visual tasks. They found that visual information mainly connects with the LLM’s middle layers, which handle meaning, rather than early layers that process text alone. By focusing on these middle layers, the model better aligns visual and text information, improving performance on tasks involving both images and language. They also showed that tuning only these layers can save training time without losing accuracy. This suggests that mixing visual and textual understanding happens mostly in the LLM’s core abstraction layers.
Large Language Model (LLM)Visual Instruction TuningMultimodal LearningSemantic LayersFine-tuningVision-Language ModelsIntermediate LayersRepresentation AlignmentCausal InterventionAbstraction Phase
Authors
Luis Palacios, Lorenzo Basile, Diego Doimo, Alberto Cazzaniga
Abstract
Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text. Yet, it remains unclear how visual features are embedded into the layer-wise hierarchy of abstractions of the LLM backbone. Across a diverse set of vision-language architectures, we show that instruction tuning primarily serves as a bridge, embedding visual features directly into the intermediate semantic layers of the LLM, bypassing the early layers devoted to unimodal processing. With probing analyses and causal interventions, we show that these intermediate layers are the semantic core of vision-language processing and play a critical role in the performance on a broad set of multimodal benchmarks. In addition, by comparing the geometry of semantically equivalent visual and textual representations, we find that fine-tuning extends and strengthens the existing abstraction phase, aligning visual features with pre-existing textual ones. Finally, we confirm the functional role of this localized alignment by restricting fine-tuning to intermediate layers alone: this strategy preserves the performance of full fine-tuning on vision-centric benchmarks while reducing training time. Our results suggest that multimodal integration is a localized phenomenon driven by the repurposing of the internal abstraction engine of the LLM.