RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
2026-06-02 • Machine Learning
Machine Learning
AI summaryⓘ
The authors tested different ways to teach large language models (LLMs) by mixing reinforcement learning (RL) and supervised fine-tuning (SFT) earlier in training, rather than just after all initial training. They found that using RL early can work well and sometimes as good as doing SFT first then RL. The type of data used before RL also strongly affects how well RL works, even more than making the model bigger. RL applied early helps the model try more varied answers, while the usual RL-after-SFT makes the model's answers more focused. Combining RL and SFT objectives together worked best, keeping the model's general abilities intact while improving performance.
Large Language ModelsReinforcement LearningSupervised Fine-TuningPre-trainingCheckpointModel ScaleData CompositionDistribution SharpeningParallel AveragingReasoning Accuracy
Authors
Rachit Bansal, Clara Mohri, Tian Qin, David Alvarez-Melis, Sham Kakade
Abstract
The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT$\to$RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by parallel averaging, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.