AI summaryⓘ
The authors studied a type of AI model for generating images called AIGC Foundation Models (AFMs), which often use a method called sparse experts. Instead of just making these models bigger, the authors tried adapting techniques from large language models that improve efficiency and training speed. They created ModernMOE (MMOE), which combines different expert routing ideas within diffusion transformers to balance quality and cost better. Their experiments showed MMOE learns faster and produces better results than similar models while using a reasonable computing budget. This suggests that image-generating AI can benefit from efficient scaling strategies used in language models, not just from increasing size.
AIGC Foundation Modelsdiffusion transformerssparse expertsMixture of Experts (MoE)efficiency in scalingFID scorerouting mechanismslarge language modelsGPU trainingconvergence
Authors
Yanhao Jia, Jiepeng Wang, Haibin Huang, Chi Zhang, Erik Cambria, Xuelong Li
Abstract
Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.