AI summaryⓘ
The authors address a problem in training language models where the learning goals are simplified into single scores, hiding what the model actually learns and causing some unwanted behavior. They propose a new way to look at the training data before the model learns from it, identifying specific concepts that affect whether the model's outputs are liked or disliked. This approach uses interpretability tools to help users give detailed feedback and control the model’s learning process more precisely. Their method can detect bad signals in data, reduce mistakes, and encourage good traits like safety and personality in models. Overall, the authors suggest that understanding the training data deeply can improve how models are taught.
language modelspost-trainingscalar rewardspreference datasetsinterpretabilitylatent conceptsmodel behaviordata-centric trainingreward shapingoff-target learning
Authors
Leon Bergen, Usha Bhalla, Sidharth Baskaran, Max Loeffler, Raphael Sarfati, Dhruvil Gala, Ryan Panwar, Santiago Aranguri, Thomas Fel, Atticus Geiger, Matthew Kowal, Siddharth Boppana, Daniel Balsam, Owen Lewis, Jack Merullo, Thomas McGrath, Ekdeep Singh Lubana
Abstract
Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy. To address this problem, we ask: can we inspect a preference dataset before optimization and decide, at the level of concepts, which behaviors a model should be allowed to learn? Motivated by this, we introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations, making them explicit for fine-grained user feedback. Building on this view, we unify several interpretability-based training protocols as ways of shaping rewards via feature or data interventions. Empirically, we show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality. More broadly, our results suggest that interpretability can turn post-training from optimizing opaque proxy rewards into a process of auditing and sculpting the learning signal itself.