PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
2026-07-30 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors studied common benchmarks that test how well large language models (LLMs) can fix software issues by matching pull requests (PRs) to issue reports. They found that about 13.6% of these PR-issue pairs are incorrectly matched in various ways. To solve this, they created PAIChecker, a system that uses multiple steps and agents to better detect mismatches between PRs and issues. Their tests showed PAIChecker improved accuracy in identifying correct PR-issue pairs on two benchmark datasets.
Large Language ModelsPull RequestIssue TrackingBenchmarkingSoftware EngineeringPR-Issue AlignmentMulti-Agent SystemsCode ValidationSoftware RepositoriesSWE-bench
Authors
Manyi Wang, Junjielong Xu, Pinjia He
Abstract
SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.