InstructSAM: Segment Any Instance with Any Instructions
2026-05-25 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present InstructSAM, a system that can find and outline multiple objects in images based on flexible instructions. They treat this task as predicting a set of instance queries, connecting a vision-language model with SAM3 through a special interface that understands both instructions and images. Their method helps avoid duplicate object detections and works efficiently in a single pass. They also created a new dataset called Inst2Seg to train and test their approach. Experiments show that InstructSAM performs well on complex instruction-driven segmentation tasks compared to previous methods.
Instance SegmentationVision-Language Model (VLM)SAM3Query PredictionHybrid AttentionMulti-instance SegmentationInstruction-driven SegmentationSet-structured PredictionDatasetReferring Segmentation
Authors
Yuqian Yuan, Wentong Li, Zhaocheng Li, Yutong Lin, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang, Wenqiao Zhang
Abstract
In this paper, we introduce InstructSAM, a unified and streamlined framework designed for multi-instance segmentation under arbitrary instructions. We formulates instruction-driven instance segmentation as a set-structured query prediction problem and propose an explicit reasoning-to-instance query interface that elegantly bridges a vision-language model (VLM) and SAM3. Specifically, a bank of learnable instance queries is injected into the VLM and contextualized with instruction and visual information, enabling each query to serve as an instance-aware slot. A hybrid-attention mechanism further promotes interaction among these queries, visual tokens, and instruction tokens, improving instance enumeration and reducing duplicate predictions. The resulting LLM-conditioned queries are projected into SAM3's detector query space to drive accurate multi-instance segmentation in a single forward pass. This design equips SAM3 with high-level instruction understanding, compositional reasoning, and instance-level set prediction without modifying its core architecture. To support training and evaluation, we further construct Inst2Seg, a high-quality and large-scale instruction-based instance segmentation dataset and benchmark that couples free-form instructions with instance-level masks. Extensive experiments show that only 2B-scale InstructSAM achieves strong results across complex instruction-driven and phrase-level referring segmentation benchmarks, outperforming prior end-to-end methods and SAM3's agentic pipeline while enabling efficient single-pass multi-instance prediction.