Overloading Large Vision-Language Models for Jailbreaking
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionCryptography and Security
AI summaryⓘ
The authors studied big computer models that understand both pictures and words and found a new way to trick these models into ignoring safety rules. They do this by combining complicated images with lots of text in a special pattern that confuses the model. This makes the models more likely to give unsafe answers when they should refuse. Their technique works better than previous ones and affects many different models. The authors also looked inside the models and found that this trick makes the models less sure about saying no.
Large Vision-Language ModelsJailbreak AttacksMultimodal ProcessingInformation OverloadingCross-Modal InteractionModel Safety AlignmentText-Image IntegrationAttack Success RateModel TransferabilityEmbodied Agents
Authors
Haoyu Zhang, Yangyang Guo, Mohan Kankanhalli
Abstract
Large Vision-Language Models (LVLMs) exhibit remarkable vision-language capabilities and are increasingly deployed in real-world applications such as personal assistants, document analysis systems, and embodied agents. However, their dual-modal attack surfaces make them vulnerable to jailbreak attacks. Existing LVLM jailbreaks rely on simple designs, e.g., short text and out-of-distribution images. Nevertheless, recent advancements in both large language model backbones and multimodal mechanisms undermine these attacks, particularly their transferability among model architectures. To overcome this limitation, we propose a novel information overloading method that is equipped with both extensive text and multi-dimensional image attacks. These components are arranged in recursion-based image-typography layouts to exponentially increase multimodal information complexity. This overloading approach amplifies the cross-modal processing required, which undermines the safety alignment in LVLMs. Extensive experiments on both open-sourced and commercial LVLMs establish our method as a new state-of-the-art LVLM jailbreak attack. On open-source models, our method achieves an average ASR of 88.6%; on commercial LVLMs, it reaches an average ASR of 84.0%, exceeding the best baseline by 48.7%. Moreover, our prompts optimized on open-source surrogate models transfer effectively across model families. Beyond empirical results, we probe the safety-critical information flows within victim LVLMs. Our observations reveal that complex image-typography compositions induce intensified cross-modal processing and reduce the model's certainty in generating refusal responses. Together, these findings highlight information overloading as a practical and emerging safety risk for real-world LVLM deployments, underscoring the need for stronger defenses against complex multimodal jailbreak inputs.