SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
2026-09-03 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors created SWE-Gate, a new benchmark that tests coding tools not just on whether their fixes pass functional tests, but also if they follow review rules often required in real software projects. They built SWE-Gate using examples from 75 open-source Python projects that include both functional tests and review constraints. Their experiments showed many fixes that passed functional tests still failed to meet these review constraints, meaning testing only functionality misses important quality checks. This helps better evaluate how well coding tools work in realistic software repair situations.
software engineering benchmarkscoding agentsfunctional testsreview constraintspull requestsrepository-level repairopen-source softwarelarge language modelssoftware patching
Authors
Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li
Abstract
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.