Rethinking Data Mixing from the Perspective of Large Language Models
2026-04-09 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors study how mixing different types of data affects the training of large language models like GPT-2. They explore what makes a data 'domain' and whether humans and models think about domains the same way. By connecting how models learn (gradient dynamics) to the distribution of domains, they create a theory explaining the role of domains during training. Using this, they develop DoGraph, a method that improves training by smartly reweighting data using graph optimization. Their experiments show DoGraph consistently works well across different GPT-2 sizes.
large language modelsdata mixing strategydomain distributiongradient dynamicsdata reweightinggraph optimizationmodel generalizationGPT-2
Authors
Yuanjian Xu, Tianze Sun, Changwei Xu, XinLong Zhao, Jianing Hao, Ran Chen, Yang Liu, Ruijie Xu, Stephen Chen, Guang Zhang
Abstract
Data mixing strategy is essential for large language model (LLM) training. Empirical evidence shows that inappropriate strategies can significantly reduce generalization. Although recent methods have improved empirical performance, several fundamental questions remain open: what constitutes a domain, whether human and model perceptions of domains are aligned, and how domain weighting influences generalization. We address these questions by establishing formal connections between gradient dynamics and domain distributions, offering a theoretical framework that clarifies the role of domains in training dynamics. Building on this analysis, we introduce DoGraph, a reweighting framework that formulates data scheduling as a graph-constrained optimization problem. Extensive experiments on GPT-2 models of varying scales demonstrate that DoGraph consistently achieves competitive performance.