AI summaryⓘ
The authors present Chimera, a new model framework for generating high-resolution images and long videos that avoids the heavy computational cost of traditional full attention methods. Chimera processes different types of data—text, images, and videos—in a unified way using a mix of efficient attention and convolution techniques, plus a flexible scaling method called HeteroP. Their experiments show Chimera is more compute-efficient than previous models, can handle longer videos without special retraining, and reveals useful patterns about balancing model size and training data. Overall, their work provides a practical approach for building scalable visual generation systems that manage long context efficiently.
diffusion modelsattention mechanismKimi Delta Attention (KDA)Multi-head Latent Attention (MLA)Sparse Mixture-of-Experts (MoE)HeteroP scalingpretrainingChinchilla scaling lawsFID (Fréchet Inception Distance)long-context modeling
Authors
Chongjian Ge, Hanwen Jiang, Tianyu Wang, Jiuxiang Gu, Yiran Xu, Ziwen Chen, Shaoteng Liu, Jing Shi, Yicong Hong, Zefan Cai, Hailin Jin, Hao Tan
Abstract
Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.