AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
2026-07-31 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created a new test called AgentHPOBench to see how well AI agents can run and improve machine learning experiments step-by-step. Unlike previous tests that only looked at one-time code or final results, this test checks if agents can learn from past results and decide on better settings for future trials. They tried this out on 12 popular AI agents and found that while these agents show some ability to optimize, they still struggle with long-term improvements and understanding complex experiment details. This work helps highlight where AI needs to get better to work more like real scientific researchers.
Large Language ModelsHyperparameter OptimizationSequential BenchmarkMachine Learning ExperimentsExperimental Evidence InterpretationAgentHPOBenchAutonomous Scientific AgentsPerformance MetricsIterative RefinementLog Diagnosis
Authors
Tianyu Huai, Tingshuo Fan, Xinchi Chen, Yining Zheng, Yuxin Wang, Shuang Chen, Jie Zhou, Xuanjing Huang
Abstract
As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration. We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol. The results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.