AI summaryⓘ
The authors study how to best split a limited amount of labeled data between two training methods, supervised fine-tuning (SFT) and reinforcement learning (RL), when improving large language models. Instead of finding a single perfect split, they look for a range of splits that perform almost as well as the best one. They find this range is quite large, especially for bigger models, and that insights from small models apply well to larger ones. This means experiments on smaller models can guide efficient data allocation for bigger models without costly trials. Their findings hold across different tasks, model types, and RL approaches, and they also consider how different annotation costs affect the results.
supervised fine-tuningreinforcement learninglarge language modelsannotation budgetmodel scalingoff-policy RLon-policy RLpreference datareward supervisiondata allocation
Authors
Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen, Daniela Rus, Bryan Kian Hsiang Low
Abstract
How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhaustive large-scale search. Our results hold consistently across tasks, model families, and both preference-based off-policy and reward-supervision on-policy RL methods. We further analyze how the asymmetry in annotation costs between SFT and RL data shifts the near-optimal region.