AI summaryⓘ
The authors created CloakDiff, a method to protect images from privacy attacks on Vision Language Models (VLMs) that try to guess sensitive details from pictures. Unlike previous methods that make images look bad or only work for visual tasks, CloakDiff hides these details without hurting image quality and lets users fully recover the original image later. It does this by subtly changing both the image pixels and internal model features to block attacks across different models and text prompts. The authors also developed a new way to guide these changes for better image quality. Tests on various datasets showed CloakDiff effectively protects privacy while keeping images clear and reversible.
Vision Language Modelsadversarial examplesprivacy attacksdiffusion modelsinvertible networkslatent cross attentionadversarial editingmultimodalimage recoveryEDM Heuristic Sampling
Authors
Qi Lu, Ziqi Zhou, Yufei Song, Zijing Li, Lulu Xue, Minghui Li, Shengshan Hu, Leo Yu Zhang
Abstract
Vision Language Models (VLMs) offer powerful multimodal ability but also expose users to text-based privacy attacks where adversaries crawl online photos and query VLMs to extract sensitive attributes. Existing reversible adversarial example (RAE) methods protect images in purely visual tasks but fail in multimodal settings, and current adversarial examples on VLMs rely on high frequency noise that severely degrades visual quality. We propose CloakDiff, the first framework for reversible, high fidelity privacy protection against text-based query attacks in VLMs. CloakDiff produces imperceptible adversarial examples by combining diffusion based adversarial editing with an invertible network that embeds the original image for lossless recovery. It perturbs both pixel space embeddings and manipulates latent cross attention maps to ensure strong cross-model and cross-prompt transferability while preserving global visual structure. To further enhance fidelity, we design EDM Heuristic Sampling, a principled diffusion schedule for adversarial guidance. Experiments on multiple datasets and VLMs demonstrate that CloakDiff delivers multimodal privacy preservation with high visual quality and reversibility.