Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift
2026-04-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors studied whether changing the text prompts can help vision-language models understand satellite images better, especially for identifying clouds. They found that no matter how they worded the prompts, performance was worse than just using the model as-is without any prompt. However, when they used even a small amount of labeled satellite images for training, the model got a lot better, much more than prompting could achieve. Their work suggests that for specialized images like satellites, collecting a bit of labeled data is more useful than trying to tweak prompts.
vision-language modelsremote sensingsatellite imagerypromptingcloud segmentationCLIPSegzero-shot learningfine-tuninglow-rank adaptationmIoU
Authors
Harshith Kethavath, Weiming Hu
Abstract
Adapting vision-language models to remote sensing imagery presents a fundamental challenge: both the visual and linguistic distributions of satellite data lie far outside natural image pretraining corpora. Despite this, prompting remains the dominant deployment paradigm, driven by the assumption that domain-specific language can guide frozen model representations toward specialized tasks. We test this assumption directly on a domain where the mismatch is prominent: cloud segmentation for satellite imagery. Using CLIPSeg on the CloudSEN12+ cloud segmentation benchmark, we evaluate 60 prompt variants spanning simple labels, domain terminology, appearance descriptors, and contextual cues, finding that every variant underperforms the zero-shot baseline (0.255 mIoU), with engineered prompts scoring as low as 0.07 mIoU. No amount of linguistic refinement bridges the gap between CLIP's natural image representations and satellite spectral imagery. In contrast, supervised fine-tuning with just 0.1% labeled data (~8 images) surpasses zero-shot performance overall, and 5-10% data recovers ~85% of maximum achievable mIoU. Full fine-tuning consistently outperforms low-rank adaptation by 0.03-0.09 mIoU, with the largest gaps for spectrally ambiguous classes, and at 0.5 to 1% labeled data, fine-tuning temporarily degrades performance on these classes before recovering, a supervision dip that aggregate mIoU can mask. For practitioners adapting vision-language models to specialized imagery, our results deliver a clear message: labeled data is not the expensive alternative to prompting; it is the worthwhile path.