Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild
2026-07-03 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors studied how developers react to automatic code review comments made by an AI tool called CodeRabbit on GitHub pull requests. They found that developers accepted about a third of these AI comments but rejected over half, often because the suggestions were wrong, repetitive, or didn’t fit the developers' intentions. The AI focused more on how code works than on how it can be improved over time but was more likely to make mistakes in those areas. The authors also tested methods to predict when AI comments would be rejected and found some promising results, indicating ways to make these AI reviews better.
agentic code reviewCodeRabbitpull requestsGitHubcode review feedbackfalse positivesdeveloper intentfunctional concernsLLM-based predictionF1 score
Authors
Hong Yi Lin, Mingzhao Liang, Kla Tantithamthavorn, Patanamon Thongtanunam
Abstract
Agentic code review, where autonomous agents provide code review comments on pull requests, is increasingly integrated into development workflows, yet there is limited empirical evidence on how developers respond to such comments in practice. In this paper, we present an empirical study of agentic code reviews using CodeRabbit as a case study. Through an empirical study of 31,073 pairs of code reviews and developer feedback from 10,191 pull requests across 239 GitHub repositories, our results show that agentic reviews receive mixed reception: 36.4% were accepted and 7.3% triggered discussion, while 56.3% were rejected. Rejections were primarily associated with invalid suggestions that were false positives, redundant, or out of scope, as well as misalignment with developer intent and coding practices. We further found that agentic reviews tend to focus more on functional concerns than evolvability-related comments, yet they were more likely to be invalid. To improve effectiveness in review practices, we explored various LLM-based approaches for predicting review rejection. We found that lightweight learning-based methods achieve up to 76% F1 score, suggesting learnable patterns exist between code reviews and their corresponding feedback. Our results highlight the current state of CodeRabbit's agentic code reviews, showing opportunity gaps for improvement, as well as shortcomings hindering its effectiveness.