Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models
2026-04-02 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors point out that typical tests for large language models (LLMs) only check if the model can respond correctly to a question, but don't see if the model understands how a conversation might continue. They suggest a new test where the model tries to generate what a user would say next, to see if it truly grasps the interaction. Their experiments with various models show that a model's ability to give correct answers doesn't necessarily mean it is good at imagining natural follow-ups. They found that certain techniques can reveal hidden awareness of conversation flow, and that extra training focused on collaboration can improve this ability.
large language modelsbenchmarkinguser-turn generationinteraction awarenessGSM8K datasettemperature samplingQwen3.5conversation modelingpost-trainingtask accuracy
Authors
Sarath Shekkizhar, Romain Cosentino, Adam Earle
Abstract
Standard LLM benchmarks evaluate the assistant turn: the model generates a response to an input, a verifier scores correctness, and the analysis ends. This paradigm leaves unmeasured whether the LLM encodes any awareness of what follows the assistant response. We propose user-turn generation as a probe of this gap: given a conversation context of user query and assistant response, we let a model generate under the user role. If the model's weights encode interaction awareness, the generated user turn will be a grounded follow-up that reacts to the preceding context. Through experiments across $11$ open-weight LLMs (Qwen3.5, gpt-oss, GLM) and $5$ datasets (math reasoning, instruction following, conversation), we show that interaction awareness is decoupled from task accuracy. In particular, within the Qwen3.5 family, GSM8K accuracy scales from $41\%$ ($0.8$B) to $96.8\%$ ($397$B-A$17$B), yet genuine follow-up rates under deterministic generation remain near zero. In contrast, higher temperature sampling reveals interaction awareness is latent with follow up rates reaching $22\%$. Controlled perturbations validate that the proposed probe measures a real property of the model, and collaboration-oriented post-training on Qwen3.5-2B demonstrates an increase in follow-up rates. Our results show that user-turn generation captures a dimension of LLM behavior, interaction awareness, that is unexplored and invisible with current assistant-only benchmarks.