Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
2026-05-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explore if large language models (LLMs) can understand and mimic the unique ways different people explain their labeling decisions in tasks like finding sentence meaning relationships. They find that individual explanation styles are hard to see in single cases but clearer when looking at multiple examples per person. They test different methods and introduce a new one called CAPO, which helps models better capture these unique explanation styles by comparing one person's explanations to others. Their work suggests it's possible to teach models to learn and produce personalized explanations based on past annotator behavior, not just labels.
Human Label VariationLarge Language ModelsNatural Language InferenceParaphrase JudgmentPromptingSupervised Fine-TuningCross-Annotator Preference OptimizationAnnotation AggregationExplanation-Based Annotation
Authors
Beiduo Chen, Pingjun Hong, Ziyun Zhang, Benjamin Roth, Anna Korhonen, Barbara Plank
Abstract
Free-text explanations extend human label variation (HLV) beyond label disagreement by revealing the reasoning and preferences behind annotators' decisions. We study whether large language models (LLMs) can learn and reproduce such annotator-specific label-explanation behavior. Using two sentence-pair tasks with four annotators each -- natural language inference and paraphrase judgment -- we first analyze whether annotators exhibit stable individual patterns. We find that such patterns are weak at the single-annotation level due to strong input-content effects, but become detectable after input-content reduction and annotator-level aggregation. We then compare prompting and supervised fine-tuning (SFT) baselines and propose cross-annotator preference optimization (CAPO), which contrasts a target annotator's response with other valid but less target-specific annotations for the same input. Experiments show that prompting is limited and unstable, SFT better captures annotator-specific behavior, and CAPO further improves aggregation-aware imitation and judge-based attribution while preserving target-specific reasoning patterns under human validation. Overall, our results show that HLV can be learned as annotator-specific label-explanation behavior, suggesting a path toward scalable explanation-based annotation grounded in annotator histories rather than labels alone.