Grow, Don't Overwrite: Fine-tuning Without Forgetting
2026-03-09 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address a problem where adapting large pre-trained models to new tasks can cause them to forget what they originally learned. They propose a new method that makes the model bigger by copying some parts but adjusts things so the model starts off exactly the same as before. This means the model can learn new tasks without losing old abilities. Their approach also lets them improve just parts of the model, saving computation while keeping performance high.
pre-trained modelscatastrophic forgettingfine-tuningtransformermodel expansionplasticitystabilityparameter replicationscaling correctionmodularity
Authors
Dyah Adila, Hanna Mazzawi, Benoit Dherin, Xavier Gonzalvo
Abstract
Adapting pre-trained models to specialized tasks often leads to catastrophic forgetting, where new knowledge overwrites foundational capabilities. Existing methods either compromise performance on the new task or struggle to balance training stability with efficient reuse of pre-trained knowledge. We introduce a novel function-preserving expansion method that resolves this dilemma. Our technique expands model capacity by replicating pre-trained parameters within transformer submodules and applying a scaling correction that guarantees the expanded model is mathematically identical to the original at initialization, enabling stable training while exploiting existing knowledge. Empirically, our method eliminates the trade-off between plasticity and stability, matching the performance of full fine-tuning on downstream tasks without any degradation of the model's original capabilities. Furthermore, we demonstrate the modularity of our approach, showing that by selectively expanding a small subset of layers we can achieve the same performance as full fine-tuning at a fraction of the computational cost.