DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation
2026-06-30 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors show that when pre-trained text-to-image models are changed to add or remove visual concepts, the changes can unintentionally harm other unrelated concepts. Common metrics like FID and KID may not detect this damage unless it becomes very severe, by which point the model is nearly unusable. To catch these hidden problems early, the authors created DriftScope, a tool that compares two versions of a model and finds which visual concepts have changed the most without needing extra data. This helps reveal subtle shifts in the model that typical evaluations miss.
text-to-image diffusion modelsconcept adaptationFIDKIDzero-shot classificationmodel driftautoencoder analysisprompt engineeringmodel evaluationvisual concept unlearning
Authors
Héctor Laria, Yiping Han, Julian D. Santamaria, Kai Wang, Bogdan Raducanu, Joost van de Weijer, Alexandra Gomez-Villa
Abstract
Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone. We argue this framing is incomplete. Through sparse autoencoder analysis and zero-shot classification, we demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface: when damage is severe enough for FID and KID to respond, the model is already nearly unusable; when the model remains functional, FID and KID stay flat while specific classes silently suffer worst-case zero-shot accuracy drops of up to 18.9 points and concept-level distributions shift dramatically. This pattern appears at both ends of the adaptation spectrum (concept customization and concept unlearning), suggesting it is a systematic consequence of weight-level modification rather than an artifact of any particular method. To surface this hidden drift before deployment, we introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them. DriftScope optimizes a soft prompt to attribute drift at the token level without requiring access to real data or model internals. The result is an interpretable, concept-level audit that aggregate evaluation cannot provide.