Code Review Agent Benchmark
2026-03-24 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors created a new dataset called c-CRAB to test how well AI agents can review code changes, especially those generated by other AI or humans. They used real human code reviews to build tests that check if AI reviews catch similar issues. When they tested current AI review tools, these could solve only about 40% of the tasks, showing there is room for improvement. They also found that AI reviews focus on different points than humans, suggesting people and AI could work together on code review. The dataset also provides a way to measure the quality of AI-generated reviews using tests.
code reviewAI agentspull requestcode quality assurancedataset c-CRABhuman code reviewcode generationtest generationbenchmark
Authors
Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, Abhik Roychoudhury
Abstract
Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.