Unbiased Alignment for Large Language Models with Noisy Preferences
2026-07-03 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address a problem where training language models to match human preferences can be tricky because the feedback data is often noisy or messy. They propose new mathematical methods called URM loss and UDPO loss that can correct this noise automatically during training. Their theory shows these methods handle noisy data well and work reliably with existing model setups. Experiments confirm their approaches work better than previous techniques on various datasets.
large language modelsreinforcement learning from human feedbackpreference noisereward modeldirect preference optimizationnoise toleranceclassification calibrationmodel alignment
Authors
Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li
Abstract
The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.