Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
2026-09-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied models that do both image understanding and image generation together. They found that doing both tasks in one model can actually help each other at the feature and task levels, but if forced to share the exact same parts of the model, one task can overpower the other. To fix this, they suggest separating some parts of the model while still letting the tasks share important information. Their experiments showed that models trained end-to-end with this approach work better on complex tasks than models that do understanding and generation separately. Overall, the authors show that combining these tasks well needs careful design beyond just having one model do both.
unified multimodal modelsvisual understandingimage generationrepresentation learningtask decouplingvision-language alignmentend-to-end optimizationtransfer learningplanner-executor pipeline
Authors
Penghao Wu, Haiwen Diao, Weichen Fan, Lewei Lu, Dahua Lin, Ziwei Liu
Abstract
While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.