Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection
2026-08-21 • Software Engineering
Software Engineering
AI summaryⓘ
The authors studied how large language models (LLMs) affect humans inspecting software requirements for problems called 'smells.' They found that using LLMs made people less accurate at spotting these smells but did not change how well they judged their seriousness or how long they took. Also, people learned less effectively if they started inspecting with LLM help. Overall, the authors show that LLMs might not always help beginners in this kind of quality checking and could even slow down learning.
Requirements inspectionRequirements smellsLarge language modelsCollaborative assistanceInspection effectivenessSeverity classificationBayesian regressionCrossover experimentSkill acquisitionSoftware requirements engineering
Authors
Giovanna Broccia, Julian Frattini, Chetan Arora, Maurice H. ter Beek, Alessandro Fantechi, Andreas Vogelsang, Alessio Ferrari
Abstract
Background. Requirements inspection (RI) is a well-established practice for detecting potential defects in requirements artifacts early in the software lifecycle. Recent advances in large language models (LLMs) have stimulated interest in their potential to support requirements engineering (RE) tasks. However, empirical evidence on the effects of LLMs when used as collaborative assistants in human-performed RI remains scarce. Aims. We aim to investigate the impact of LLM support on human-performed RI, considering inspection effectiveness in terms of smell identification and severity classification (i.e., nocuous vs innocuous), as well as inspection duration. Method. We conducted a controlled crossover design experiment with 34 participants, who inspected textual specifications with and without LLM support, identifying and classifying requirements smells while recording inspection time. We analyzed the data using one Bayesian regression model per outcome variable, accounting for validity threats induced by the crossover design as well as covariates and mediators. Results. Results show that LLM support negatively affects smell detection accuracy but has no significant effect on smell classification or task duration. A learning effect is present across experimental periods, but reduced when RI is first performed with LLM support. Conclusions. Our findings provide empirical evidence that LLM support does not necessarily improve performance and may, instead, hinder it for novice inspectors. Moreover, the results suggest that learning RI with LLM-support from the beginning may slow down the skill acquisition process, implying threats for LLM-supported learning.