AI summaryⓘ
The authors studied LoRA, a method that fine-tunes large neural networks by updating many small matrices, but found that not all these matrices are equally important. They discovered that matrices with bigger condition numbers (a measure related to how imbalanced they are) play a bigger role in improving performance. Based on this, they created a method called appa-LoRA, which only updates the most important matrices, cutting the number of parameters and computational cost roughly in half. Their experiments showed that this selective updating keeps accuracy similar while making training faster and cheaper. They also found that this works because the chosen matrices become more balanced during training, improving effectiveness.
Low-Rank Adaptation (LoRA)fine-tuningcondition numbersingular valueneural networksparameter efficiencyspectral rebalancingcompute costmemory costmodel adaptation
Authors
Jianghui Wang, Silong Yong, Francesco Orabona, Marco Canini, Katia P. Sycara, Yaqi Xie
Abstract
Low-Rank Adaptation (LoRA) has become a widely adopted technique for efficient neural network fine-tuning, decomposing model updates into low-rank matrices. However, LoRA remains computationally costly because it updates all matrices uniformly, regardless of their actual contribution to adaptation. This cost is especially prohibitive for large-scale models with billions of parameters and for resource-constrained settings such as edge deployment and on-device fine-tuning. We show for the first time that not all LoRA matrices are equally worth tuning: matrices with smaller condition numbers (the ratio of largest to smallest singular value) are already well-balanced across directions and contribute only marginally to adaptation, whereas matrices with larger condition numbers contain underdeveloped directions that span richer subspaces and drive most of the performance gains. This observation itself is a key contribution of our work, and it motivates a more selective approach to fine-tuning. Building on this insight, we propose \k{appa}-LoRA, a method that optimizes LoRA by focusing updates on the matrices with the largest condition numbers, which capture the most informative directions of change. By restricting LoRA updates to the top 50% of weight matrices ranked by condition number, \k{appa}-LoRA halves the trainable parameter count and correspondingly reduces compute and memory cost. Extensive experiments across multiple benchmarks show that this design cuts fine-tuning time by 16.2% on average while matching the accuracy of standard LoRA and reducing memory cost by 4.5%. Further analysis reveals that the condition numbers of the selected matrices consistently decrease over training, suggesting that \k{appa}-LoRA's effectiveness stems from targeted spectral rebalancing rather than parameter selection alone.