One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors address class-incremental learning, where a model must learn new classes over time without forgetting old ones or accessing past data. They propose FACET, a method that uses a single adapter dynamically adjusted by task information to efficiently keep features distinct for different tasks. FACET also introduces a special loss to prevent forgetting without needing to replay old data. Their approach is both parameter-efficient and works well on both short and very long sequences of tasks.
Class-incremental learningPretrained modelsAdapter modulesFeature transformationCatastrophic forgettingTask-conditioned learningLoRAReplay-free methodsParameter efficiencyFeature consistency loss
Authors
Yunxiang Fu, Meng Lou, Yizhou Yu
Abstract
Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while preserving the ability to recognize all seen classes. Recently, pretrained-model-based approaches have become prevalent by adapting a frozen backbone with additional lightweight trainable modules. Existing methods, however, exhibit limitations: task-specific adapters learn explicit per-task representations but are parameter- and computation-inefficient, while LoRA-based merging methods combine per-task LoRA parameters into a single model whose static aggregated weights cause representation interference during inference. To address these problems, we present \textbf{FACET}: task-conditioned \textbf{F}e\textbf{A}ture transformation with \textbf{C}ondition\textbf{E}d feature consis\textbf{T}ency, achieving excellent parameter efficiency while producing highly discriminative features during inference. When continually trained on a task sequence, FACET learns a single shared adapter that employs a dynamic task-conditioned feature transformation, shaping the overall feature distribution of the adapter into a mixture of overlap-reduced task-specific components. On the other hand, we propose an efficient replay-free task-conditioned feature consistency loss, aiming to mitigate catastrophic forgetting of the learned mixture distribution in the adapter's feature space. Even when maintaining only a single adapter, FACET demonstrates robust scalability. On both very long task sequences (e.g., 200 tasks) and standard short task sequences (e.g., 20 tasks), our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs. The code will be made open source upon acceptance.