Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study
2026-07-03 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors studied if Large Language Models (LLMs) can create diverse versions of software to reduce the chance that all versions fail the same way. They compared traditional human-written programs with many LLM-generated programs across different models, settings, and programming languages. Their tests found that using LLM-generated code, especially when combining diverse types, can improve software reliability. However, this improvement depends on factors like the programming language and how the code is generated. Overall, the authors suggest LLMs offer a cost-effective way to produce diverse software that could be more dependable.
Software diversityCommon-mode failuresLarge Language Models (LLMs)Code generationReliabilityProgramming languagesTest suitesDiverse redundancyHeterogeneous populationsSoftware validation
Authors
Gabriel Almeida, Ilir Gashi, Vladimir Stankovic, João R. Campos
Abstract
Software diversity has been extensively studied as a means of reducing the risk of common-mode failures. Classic work showed that the central issue is whether failures of diversely redundant components overlap in ways that limit the reliability gains. Traditional software diversity is costly to obtain, since it requires multiple implementations as well as the corresponding validation, maintenance, and deployment effort. Recent advances in Large Language Models (LLMs) may change this. LLMs enable inexpensive code generation: they produce many candidate implementations of the same specification quickly, across different models, decoding settings, and programming languages. This raises a natural question: can LLMs serve as practical generators of software diversity, and how much reliability improvement can that diversity actually provide? In this paper, we extend classical empirical studies of software diversity in human-written programs to LLM-generated code. We study three specifications using both historical human-written programs and large pools of LLM-generated ones evaluated under a common compilation, sandboxing, and exhaustive test suite. We explore LLM diversity along multiple axes, including model family, generation temperature, and programming language. Reliability improvement is evaluated in a 1-out-of-2 configuration across both homogeneous and heterogeneous program populations, including within-LLM pairings and pairings across programming languages and across LLM-generated and human-written programs. The results show that combining LLM-generated programs, especially in heterogeneous settings, can yield reliability gains, although this is partly conditioned by the programming language and generation setting. Taken together, these findings suggest that LLMs provide a scalable source of comparatively low-cost programs whose diversity can be leveraged for reliability improvement.