Detecting Behavioral Changes in Python Refactoring Implementations with Foundation Models
2026-08-10 • Software Engineering
Software Engineering
AI summaryⓘ
The authors studied how automated tools that improve Python code, called refactoring tools, sometimes accidentally change how the code works. They created a method using a foundation model to look at code changes and detect when the behavior of the program was unintentionally altered. Testing it on existing Python refactoring tools revealed several bugs, most of which were confirmed and accepted by developers. This shows that current Python refactoring tools need to be improved to avoid breaking code during automatic updates.
PythonRefactoringAutomated RefactoringFoundation ModelGit DiffSoftware EvolutionCode TransformationBugsSoftware MaintenanceRope Library
Authors
Jonhnanthan Oliveira, Rohit Gheyi, Márcio Ribeiro, Alessandro Garcia
Abstract
Python is a widely adopted programming language, valued for its simplicity and flexibility. However, automated refactoring for Python remains challenging, even though refactoring is an essential practice in software evolution aimed at improving internal code structure without changing external behavior. Understanding how behavioral changes are introduced during refactoring is crucial, as such issues can compromise software reliability and reduce developer productivity. We propose an approach based on a foundation model oracle that analyzes git-style diffs to identify behavioral changes introduced by Python refactorings. We evaluated our technique on Rope refactoring implementations, reusing 1,152 refactoring attempts from a prior study and analyzing 217 resulting transformation pairs with the oracle. Our model-based analysis uncovered 13 distinct bugs among the seven refactoring types studied. All reported bugs were submitted to the respective developers, and 12 of the 13 resulting issue reports were accepted according to issue-tracker evidence. These results highlight the need to improve the robustness of current Python refactoring tools to ensure the correctness of automated code transformations and support reliable software maintenance.