Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
2026-06-05 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address a problem in teaching large language models new skills without forgetting old ones, known as the plasticity-stability dilemma. They propose a method called SETA, which splits the model into special task-focused parts and shared parts to keep new and old knowledge separate. This approach uses smart rules to protect important shared knowledge and helps the model pick the right parts when working on different tasks. Tests show that their method works well on various benchmarks and helps the model remember earlier skills better than other methods.
continual learninglarge language modelsplasticity-stability dilemmacatastrophic forgettingsparse expert moduleselastic regularizationgating networkknowledge retentionbackward transferparameter isolation
Authors
Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari
Abstract
Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through adaptive elastic anchoring and a routing-aware regularization that jointly protect shared knowledge at both the weight and routing levels and enable a unified gating network to automatically retrieve the correct expert combination during inference. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, with particularly strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.