LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution
2026-07-01 • Software Engineering
Software EngineeringArtificial IntelligenceProgramming Languages
AI summaryⓘ
The authors created LLVM-Bench, a large set of real bug-fixing tasks from the LLVM compiler project, to test how well language models can fix compiler issues. They also built LLVM-Gym, a system to automatically test whether the fixes work by rebuilding and running the compiler. Their study found that current language models often produce fixes that don't work or cause build failures. They then combined fixes from different models with LLVM-Ens to improve success rates, achieving about 22% of issues resolved. This shows combining multiple models can help, but there's still room for improvement in fixing complex compiler bugs automatically.
LLVMcompiler infrastructurelarge language modelsissue resolutionbenchmarkautomatic patchingbuild failureensemble methodssoftware testingbug fixing
Authors
Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang, Junjie Chen
Abstract
LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.