DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation
2026-04-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed DiffuSAM, a new method that helps segment medical images without needing special user prompts. They built it on top of an existing model called SAM2 but changed it to better handle medical images by generating mask-like features using a diffusion process. This also allows the method to keep segmentations consistent across slices of 3D scans. Tested on CT and MRI datasets, their approach performs well with minimal extra training. Overall, their work aims to simplify and improve medical image segmentation without extensive expert input.
SegmentationSegment Anything Model (SAM)Diffusion modelsMedical imagingImage segmentationDomain adaptationCT scansMRIZero-shot learningFew-shot learning
Authors
Tal Grossman, Noa Cahan, Lev Ayzenberg, Hayit Greenspan
Abstract
Segmentation models such as Segment Anything Model (SAM) and SAM2 achieve strong prompt-driven zero-shot performance. However, their training on natural images limits domain transfer to medical data. Consequently, accurate segmentation typically requires extensive fine-tuning and expert-designed prompts. We propose DiffuSAM, a diffusion-based adaptation of SAM2 for prompt-free medical image segmentation. Our framework synthesizes SAM2-compatible segmentation mask-like embeddings via a lightweight diffusion-prior from off-the-shelf frozen SAM2 image features. The generated embeddings are integrated into SAM2's mask decoder to produce accurate segmentations, thereby eliminating the need for user prompts. The diffusion prior is further conditioned on previously segmented slices, enforcing spatial consistency across volumes. Evaluated on the BTCV and CHAOS datasets for CT and MRI under Source-Free Unsupervised Domain Adaptation (SF-UDA) and Few-Shot settings, DiffuSAM achieves competitive performance with efficient training and inference. Code is available upon request from the corresponding author.