MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address problems in Visual AutoRegressive (VAR) models where different image scales are not well learned, causing early mistakes to affect final images. They propose a new approach using a special mechanism called scale-aware token-routed Mixture of Experts (MoE), which lets the model learn representations separately for each scale. They also improve early-stage semantic understanding by carefully combining external features using a residual aggregation method. Their experiments show this method trains faster and produces better quality images than previous VAR models, especially on ImageNet. Larger training times further improve their results.
Visual AutoRegressive modelingmulti-scale representationMixture of Experts (MoE)scale-aware learningsemantic modelingself-supervised featuresresidual feature aggregationImageNetFID scoretraining efficiency
Authors
Nuoyan Zhou, Zhijun Tu, Lei Yu, Kun Cheng, Jie Hu, Nannan Wang, Xinghao Chen
Abstract
Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation. However, VAR still suffers from inherent deficiencies in multi-scale representation learning. Specifically, lower scales primarily capture global semantics, while higher scales focus on fine-grained details. Employing a shared architecture across scales induces optimization conflicts. Moreover, due to the causal autoregressive process, inaccurate semantics at early scales can propagate and significantly degrade the final output. To address these issues, we introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection, thereby facilitating decoupled representation learning across scales. In addition, we enhance semantic modeling at early scales by incorporating external self-supervised features. Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the VAR paradigm. Extensive experiments show that our method significantly improves both training efficiency and generation quality. On the ImageNet 256*256 benchmark, our model achieves a superior FID compared to the dense baseline while requiring only half of the default training epochs and a smaller parameter budget, with a merely marginal increase in training cost. Moreover, the performance gap further widens with larger training epochs.