MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
2026-07-13 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created MM-ToolSandBox, a test system to evaluate AI agents that use visual information to decide which software tools to call. Their system includes over 500 tools and handles complex tasks with multiple images and conversations, reflecting real-world use. They tested 12 AI models and found that none performed very well, mainly because many struggled to accurately understand details in images rather than planning their actions. The authors note that smaller models have trouble figuring out what to do, while bigger ones often misinterpret visual inputs, suggesting different ways to improve these models.
tool-calling agentsvisual groundingstateful execution environmentmulti-turn tasksscenario generationinformation extractionAI model evaluationplanning vs. perceptioninteractive UI applications
Authors
Kaixin Ma, Di Feng, Alexander Metz, Jiarui Lu, Eshan Verma, Afshin Dehghan
Abstract
We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox