CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of teaching vision-language models to continually learn new anomaly detection tasks without forgetting previous ones. They propose CL-Anomaly, a method that separates task-specific knowledge to avoid mixing up different tasks, while also sharing useful information across tasks through special shared components. Their approach dynamically chooses which shared parts to update, helping the model perform well across multiple types of anomaly detection challenges. Experiments show this method works better than previous approaches for learning across different classes, domains, and modalities.
Multimodal Large Language ModelsContinual LearningAnomaly DetectionParameter-efficient Fine-tuningSemantic EntanglementPrivLoRAKnowledge TransferCross-domain LearningCross-modal Learning
Authors
Wen Dong, Zhao Wang, Shuangqing Zhang, Kai Sun, Ben Li, Guo-Sen Xie, Caifeng Shan, Fang Zhao
Abstract
Multimodal Large Language Models (MLLMs) excel in diverse vision tasks, but full-parameter retraining is computationally expensive as real-world knowledge evolves. Existing continual learning methods often suffer from semantic entanglement in parameter spaces across tasks, impeding the continuous deployment of models. This challenge is especially pronounced in Anomaly Detection (AD), which exhibits triple heterogeneity across modalities, domains, and defect scale variability, significantly complicating multi-task knowledge transfer. In this paper, we propose CL-Anomaly, a parameter-efficient fine-tuning framework based on an isolation-sharing collaboration to enable continual learning for anomaly detection with MLLMs. We introduce the task-private expert PrivLoRA, which physically isolates task-specific subspaces in the parameter space to prevent semantic entanglement of anomaly knowledge in diverse scenarios. The Layer-Adaptive Shared Experts maintain cross-task representations within a unified feature space, enabling knowledge sharing between previous and new tasks. Furthermore, we propose a Layer-Adaptive Knowledge Transfer strategy that automatically selects and dynamically updates the layer-wise key shared experts of each task via a momentum-based mechanism, promoting effective knowledge transfer across related anomaly detection tasks. Extensive experiments across three continual learning scenarios for anomaly detection, including class-incremental, cross-domain, and cross-modal, demonstrate that CL-Anomaly outperforms state-of-the-art methods. Code is available at https://github.com/WenDongyp/CL-Anomaly.