A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design
2026-06-09 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors look at supervised fine-tuning (SFT), a way to teach models by making them copy exact example tokens, and suggest this might not always be best because example tokens can be uncertain or not always correct. They propose a new framework called Q-target that treats the training as deciding how much to trust the given token and how to spread remaining trust elsewhere. This method lets the training be more flexible and improves performance on reasoning tasks. Their work offers a new way to think about and improve fine-tuning models.
supervised fine-tuningtoken likelihoodtarget distributionQ-target frameworktraining objectivemodel priorone-hot targetloss objectivereasoning datasetsprobability allocation
Authors
Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, Cho-Jui Hsieh
Abstract
Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match. We introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit choices of the target distribution Q. Building on this view, we propose Target-SFT which constructs the training objective directly from the desired target distribution. This method consistently outperforms across the ten reasoning dataset-model settings evaluated, showing the effectiveness of this target-based approach. Overall, our formulation reveals a more fundamental design principle for SFT training and opens a broader search space for SFT objectives.