In-Context Reward Adaptation for Robust Preference Modeling
2026-05-28 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors point out that current methods for teaching AI to follow human preferences usually use one fixed model, which struggles when people's values change or are different. They propose a new method called In-Context Reward Adaptation that lets the AI learn and adjust to new human preferences quickly by looking at a few examples. Their method uses a transformer model and includes how fast people respond as extra information to improve learning. This makes the AI better at handling different and new human preferences without needing expensive retraining.
Reinforcement Learning from Human Feedback (RLHF)reward modelstransformersin-context learninghuman preferencespreference modelingdistribution shiftresponse timehuman-AI alignment
Authors
Zhenyu Sun, Zheng Xu, Ermin Wei
Abstract
Reinforcement Learning from Human Feedback (RLHF) typically relies on static reward models to align Large Language Models with human preferences. However, human values are inherently diverse and heterogeneous, and a single reward model often lacks the robustness required to generalize to unseen preference domains. While existing multi-reward frameworks attempt to address this, they are often restricted to a fixed set of known domains and fail to adapt to unseen human distributions without costly retraining. In this work, we propose In-Context Reward Adaptation, a transformer-based framework designed to model diverse and unseen human preferences on the fly. By leveraging the in-context learning capabilities of transformers, our approach adaptively infers the underlying reward structure from a small set of preference demonstrations. We demonstrate that while a standard transformer architecture is insufficient for this task by characterizing an asymptotic bias to the ground-truth, incorporating human response time as an auxiliary input signal enables the model to successfully adapt to preferences from previously unseen domains. Our findings show that this approach provides a more robust foundation for preference modeling, allowing for the representation of heterogeneous rewards and preference distribution shift, and offering a scalable path toward more flexible human-AI alignment.