Are We Really Making Progress in Group Recommendation? Unmasking the Tie-Breaking Illusion
2026-08-11 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors found that some recent group recommendation methods seemed better mainly because of a testing bias, not because they truly learned group preferences better. They discovered that how ties in recommendation scores are handled during evaluation can hugely affect results, especially when a certain sigmoid step is added during training. By using a fairer, tie-aware evaluation method, they showed that many claimed improvements become much smaller or change in ranking. They also proposed a safer alternative training method that keeps benefits without causing too many ties. Overall, the authors stress the need for careful evaluation to measure real progress in group recommendations.
group recommendationevaluation biastie-breakingBPR objectivesigmoid transformationHR@KNDCG@Ktie-aware evaluationtemperature scalingmargin smoothing
Authors
Song-Duo Ma, Pu-Jen Cheng
Abstract
Recent group recommendation methods have reported strong improvements on standard benchmarks, but it remains unclear whether these gains always reflect genuine advances in modeling group preferences. In this paper, we show that several recent methods are affected by a systematic evaluation bias caused by the interaction between training-time score compression and evaluation-time deterministic tie-breaking. Specifically, an additional sigmoid transformation before the BPR objective can greatly increase tied top scores, making top-K metrics such as HR@K and NDCG@K highly sensitive to how ties are resolved. We revisit recent representative methods and their baselines on CAMRa2011 and Mafengwo under both group and user recommendation settings, and evaluate them with a tie-aware protocol that computes the exact expectation of HR@K and NDCG@K under uniform random tie-breaking. Our results show that many previously reported improvements shrink substantially under tie-aware evaluation, and the relative ranking of methods can change markedly. We further show that the additional sigmoid may act as implicit margin smoothing during optimization, and that temperature-scaled BPR can retain much of this benefit without inducing severe tie inflation. Overall, our findings highlight the importance of tie-aware evaluation for establishing reliable progress in group recommendation. The code is available at https://github.com/songduoma/TieAwareGroupRec.