AI summaryⓘ
The authors studied how well a big computer vision model called SAM 3 works on earth observation images like satellite photos without extra training. They tested it on tasks like classifying scenes, detecting objects, and segmenting images using only zero or one example. They modified SAM 3 to use part of its structure for zero-shot classification and tested how text and visual hints affect performance. They found that visual hints help the model understand complex shapes in these images, but text hints can confuse it by adding irrelevant ground-level meanings. The authors suggest that SAM 3 performs well without overfitting but needs fine-tuning to overcome some resolution and interpretation challenges for better use in geospatial tasks.
Foundation ModelsSegment Anything Model 3 (SAM 3)Earth Observation ImageryZero-shot LearningOne-shot LearningRemote SensingScene ClassificationObject DetectionInstance SegmentationMultimodal Decoder
Abstract
The deployment of large-scale foundation models, such as the Segment Anything Model 3 (SAM 3), promises a transition toward open-vocabulary, training-free computer vision. However, their capacity to generalize out-of-distribution to the complex, top-down geometric structures of Earth Observation imagery remains largely unquantified. Driven by SAM 3's performance disparities in highly specialized domains, we present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation under strict zero-shot and one-shot constraints. To achieve this, we introduce a structural adaptation of SAM 3 by repurposing its decoupled binary presence head into a standalone zero-shot classifier. Furthermore, by systematically isolating textual and visual prompt modalities across five configurations, we explicitly diagnose the alignment mechanics within the model's multimodal decoder. Our findings reveal severe cross-modal interference: while visual prompts successfully align the decoder to complex remote sensing geometry, textual prompts inject misaligned, ground-level semantic bias, actively degrading coordinate regression. To benchmark these capabilities without resource-intensive training, we formulate a novel training-free proxy evaluation protocol for Generalized Zero-Shot tasks (scene classification and instance segmentation). Ultimately, our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models, achieving high Harmonic Mean scores in segmentation tasks. However, it remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots, charting a definitive mandate for parameter-efficient geospatial fine-tuning of its multimodal decoder.