ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs
2026-08-04 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and Language
AI summaryⓘ
The authors propose a new way to improve multimodal large language models by running vision and language parts in parallel using the same underlying model parameters multiple times. This method, called ParVL, helps balance how much computation is used for visual analysis versus language processing without increasing model size or slow-down. They tested different ways to split computation between the vision and language components and found that the best balance depends on the task. Their approach performs better than traditional single-branch models when trained on a large dataset.
Multimodal Large Language ModelsVision Transformer (ViT)Large Language Model (LLM)Parallel ComputationFine-tuningComputation AllocationPrefix ParametersSupervised LearningModel ScalingMultimodal Performance
Authors
Yang Yang, Qinyu Zhao, Mouxiang Chen, Xiaohui Li, Lixin Gu, Wenhai Wang, Hongjie Zhang, Wenwei Zhang
Abstract
Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.