Parameter-Efficient Fine-Tuning with Learnable Rank
2026-06-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study a way to fine-tune large language models by updating only small parts called adapters, which normally have a fixed size called rank. They introduce a method, LR-LoRA, that lets the model learn the best adapter size for each layer during training instead of using the same size everywhere. They find that different layers prefer different adapter sizes, especially in transformer models. Their approach works better than other similar methods on language tasks, showing that letting the rank vary is helpful.
Low-Rank Adaptation (LoRA)parameter-efficient fine-tuning (PEFT)adapter layersranktransformer modelslanguage understandingcommonsense reasoningMLP layersattention layers
Authors
Arpit Garg, Simon Lucey, Hemanth Saratchandran
Abstract
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning. We introduce *Learnable Rank LoRA (LR-LoRA)*, a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences. Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.