Representation Forcing for Bottleneck-Free Unified Multimodal Models
2026-05-29 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address a limitation in unified multimodal models, which typically use a separate, fixed part (a VAE) for generating images, causing a bottleneck. They introduce Representation Forcing (RF), a method that has the model predict intermediate visual representations before generating pixels, helping it learn better without relying on an external system. Their approach improves both image generation and understanding, matching or exceeding previous models that depended on VAEs. This work moves toward models that process images end-to-end without built-in bottlenecks.
unified multimodal modelsvariational autoencoder (VAE)image generationrepresentation learningautoregressive predictionpixel diffusionlatent spaceimage understanding
Authors
Yuqing Wang, Zhijie Lin, Ceyuan Yang, Yang Zhao, Fei Xiao, Hao He, Qi Zhao, Zihan Ding, Fuyun Wang, Shuai Wang, Youliang Zhang, Haoqi Fan, Xihui Liu
Abstract
Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.