Communicate-Predict-Act: Evaluating Social Intelligence of Agents
2026-04-09 • Computers and Society
Computers and Society
AI summaryⓘ
The authors created a game setup where different language models (like chatbots) interact with each other in both cooperative and competitive ways to study social intelligence. They tested eight language models of various sizes and found that simple rankings don't fully capture social smarts. By looking closely at how the models act during games, the authors identified measures like influence and adaptability as key to success, more so than deep planning or understanding other minds. Their work suggests social intelligence is complex and involves several measurable skills.
large language modelssocial intelligencemulti-agent gamescommunicative influenceElo ratingTheory of Mindstrategic reasoningadaptabilityAUC ROCsociocognitive metrics
Authors
David Shoresh, Sarit Kraus, Yonatan Loewenstein
Abstract
As large language model (LLM) agents become more prevalent in real world social settings, social intelligence will play an increasingly critical role. But social intelligence is still a poorly defined construct, for humans and artificial agents. We introduce a multiplayer arena of mixed cooperative and competitive social games to study LLM social intelligence. The controllability of LLM based agents enables systematic evaluation, which also supports broader inferences about social intelligence per se. We evaluated eight diverse LLMs (24B to 1T parameters) using a Communicate Predict Act (COMPACT) interaction protocol and fine grained probing of social dynamics. Elo style ratings reveal consistent performance differences across models, but this scalar measure provides only a partial characterization of social intelligence. To address this limitation, we analyze gameplay traces to extract sociocognitive metrics capturing action prediction, communicative influence, strategic reasoning, and tradeoffs under conflicting interests. These sociocognitive metrics exhibit strong intramodel consistency and they reliably predict pairwise agent advantage in game outcomes (AUC ROC = 0.82). Feature importance analysis indicates that surprisingly, influence, transparency, and adaptability are more predictive of success than Theory of Mind inference or deep planning. Together, our results advance a testable, multidimensional conception of social intelligence and provide empirical insights into the capacities that underpin it.