DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
2026-07-27 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors introduce DataOrchestra, a system that customizes how each piece of data is cleaned and prepared before training large language models, instead of using one fixed method for all data. Their system decides whether to keep, drop, or clean each data chunk and, if cleaning is needed, it chooses specific editing steps and generates instructions for those edits. They tested this approach by training models of various sizes from scratch and found it consistently improved performance on multiple tests while using less computing by skipping needless steps. DataOrchestra also worked well for specialized math data training and was better than other strong data processing methods.
Large Language ModelsPretrainingData ProcessingData CleaningProgrammatic EditingLLM-based RewritingInstruction GenerationModel TrainingCompute EfficiencyBenchmarking
Authors
Zhen Huang, Yikun Wang, Shijie Xia, Pengfei Liu
Abstract
Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.