An Empirical Study on Influence-Based Pretraining Data Selection for Code Large Language Models
2026-04-09 • Software Engineering
Software Engineering
AI summaryⓘ
The authors studied how to pick better training examples to improve programming language models, which are AI systems that understand and write code. They focused on a method called data-influence-score filtering, which scores training data by how much it helps the model perform on coding tasks. By training a large model on filtered code data, they found that using this score can boost programming performance. They also noticed that what counts as 'helpful data' changes depending on the specific programming challenge and training phase.
Code Large Language ModelsPre-training DataData Influence ScoreData FilteringGenerative Programming TasksValidation Set LossModel PerformanceTraining DatasetProgram SynthesisLoss Function
Authors
Chengli Xing, Zhengran Zeng, Gexiang Fang, Rui Xie, Wei Ye, Shikun Zhang
Abstract
Recent advancements in code large language models (Code-LLMs) have demonstrated remarkable capabilities in resolving programming related tasks. Meanwhile, researchers have recognized that the quality of pre-training data is crucial for improving LLM performance. However, most of the existing research on pre-training data filtering has focused on general datasets, and little attention for programming datasets. In this paper, we aim to address this gap by exploring the effectiveness of a widely used general data filtering technique, i.e., data-influence-score filtering, within the context of programming-related datasets. To this end, we first introduce a method for calculating data-influence-score for generative programming tasks which involves transforming a variety of downstream coding tasks into validation sets and using the models loss on these sets as a performance metric. Next, we pre-train a Code-LLMs with 1 billion parameters from scratch on a dataset of 100 billion code tokens. Based on it, we conduct an extensive empirical study to evaluate the effectiveness of data-influence-score filtering methods. Specifically, we examine how well this technique improves model performance, investigate how the characteristics of beneficial training data vary across different training stages and programming tasks, and assess the feasibility of prediction-based data-influence-score filtering method. Our findings show that data-influence-score filtering based on validation-set-loss can enhance models programming performance. Moreover, we observe that the criteria of beneficial training data differ significantly across various downstream programming tasks.