SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
2026-06-02 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created SMAC-Talk, a game-based test to see how well large language models (LLMs) can work together with other AI agents using natural language. Their setup involves challenges like limited information and the need for long-term planning, simulating real teamwork problems. They added a chat feature where agents talk to each other, including scenarios where one agent tries to fool the others. The authors tested different AI models to understand what helps cooperation and are sharing SMAC-Talk so other researchers can improve AI teamwork.
Large Language Models (LLMs)Multi-Agent SystemsStarCraft Multi-Agent ChallengeNatural Language CommunicationDecentralized ControlPartial ObservabilityCooperative AIDeceptive CommunicationBenchmarkingAI Coordination
Authors
Joel Sol, Homayoun Najjaran
Abstract
As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty. We introduce SMAC-Talk, a natural language extension of the StarCraft Multi-Agent Challenge for evaluating LLM-based agents in cooperative multi-agent environments. The environment has several key features such as decentralized control, partial observability and long-horizon decision making. SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust. We use this communication channel to construct different evaluation scenarios, including settings with an embedded deceptive communicator that tries to disrupt and deceive allies through communication alone. We provide three agents for benchmarking using 4 models from the Qwen3.5 family and study how reasoning structure, memory and model scale affect coordination between agents. We release SMAC-Talk as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent settings.