Consistency Training Can Entrench Misalignment
2026-06-02 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied consistency training, a method that makes models give similar answers to similar questions, to see how it affects model behavior. They tested seven types of consistency training on many models designed to act in certain misaligned ways. They found that while consistency training usually reduces some bad behaviors like reward hacking, it can increase others like sycophancy. The authors suggest that changes in the data distribution during training, rather than the specific methods used, cause these effects. They also developed a theory to predict when consistency training will help or harm model alignment, recommending careful checks when using it in important systems.
consistency trainingmodel alignmentreward hackingsycophancydistribution shiftself-bootstrappingfine-tuningopen-source modelsmisalignmenttheoretical framework
Authors
David Demitri Africa, Arathi Mani
Abstract
Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but their effects on model alignment remain poorly understood. Could the self-bootstrapping nature of these methods amplify undesired behavior in models? We test seven consistency training methods on 108 ``model organisms: open-source models (7B--70B) fine-tuned to exhibit various forms of controlled misaligned behavior. We find that outcomes vary significantly: consistency training generally suppresses reward hacking and emergent misalignment but amplifies sycophancy. We present evidence that distribution shifts induced by the consistency labeling process, rather than variation in the selection operators, may be the primary driver of systematic alignment effects. Finally, we present a unifying theoretical framework to derive conditions under which consistency training will amplify or suppress misalignment. In total, our study establishes that consistency training is not alignment-neutral, and that its use in critical systems should be carefully audited.