Benchmark Everything Everywhere All at Once
2026-06-04 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created Benchmark Agent, a fully automated system that builds tests to evaluate language models without much human help. This system handles everything from understanding what kind of test is needed to making sure the test questions are good quality. They tested it by creating 15 different benchmarks that check various skills, like reading text, understanding images and words together, and specialized reasoning. Their experiments show the system makes good tests and reveal that current models still find some complex reasoning tasks difficult. The authors think these evolving benchmarks will help researchers better measure progress.
BenchmarkLLMMLLMData AnnotationQuality ControlDomain-specific ReasoningAutonomous AgentEvaluation MetricsHuman EvaluationContinual Evaluation
Authors
Shiyun Xiong, Dongming Wu, Peiwen Sun, Yuang Ai, Bokang Yang, Wencheng Han, Xiao-Hui Li, Xiangyu Yue
Abstract
Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.