APeB: Benchmarking Personalization Ability of Large Language Model Agents
2026-07-03 • Artificial Intelligence
Artificial IntelligenceHuman-Computer Interaction
AI summaryⓘ
The authors studied how language models (LLMs) help personalize product search when users give vague or unclear requests. They created a new test called APeB that uses real user data, including messy interaction histories and unclear queries, to see how well these models understand what users really want. Their tests showed that current models do well with clear requests but have trouble when they need to guess user intent from messy histories. They also found that adding a simple step to better use past user history improved performance, suggesting future models should focus more on understanding user history.
LLMspersonalizationproduct searchuser intentinteraction historyquery refinementbenchmarksagent workflowsnatural language processingrecommendation systems
Authors
Garry Yang, Zizhe Chen, Xinru Chen, Yongqiang Chen, Jianxiang Wang, Deyu Zou, Linyi Ding, Jialiang Wu, Yunzhong He, Yu Gong, James Cheng, Huaixiao Tou
Abstract
LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.