GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created GEAR-Seg, a system that breaks down the task of finding and understanding objects in images into separate, clear steps: identifying objects, describing them with detailed text, and then reasoning about them with a language model. This makes the process easier to understand and improves performance on tricky image tasks without needing extra training data. They also built a large dataset called GEAR-131K to help test and train systems on complex reasoning with images. Finally, they showed that smaller models trained using their method nearly perform as well as models trained on expensive human-labeled data.
Reasoning segmentationClass-agnostic segmentationSemantic descriptionLarge Language Model (LLM)Zero-shot inferenceData distillationMultifaceted taxonomyVisual groundingReferring segmentationAutomated data pipeline
Authors
Yanan Wang, Wen Li, Yibin Ying, Zhenghao Fei
Abstract
Reasoning segmentation requires localizing targets based on complex, implicit queries. Current end-to-end models typically entangle perception and deduction into an opaque black box, severely limiting interpretability and scalability. To address this, we propose GEAR-Seg (Grounded Explainable Agent for Reasoning Segmentation), an explicitly decoupled agent that shifts the paradigm by translating visual pixels into dense, attribute-rich text. By decoupling class-agnostic segmentation, semantic description, and Large Language Model (LLM) deduction, GEAR-Seg transforms implicit reasoning into an explicit, trackable logic chain. As a zero-shot inference framework, it achieves highly competitive performance across diverse reasoning and fine-grained referring segmentation benchmarks. Furthermore, GEAR-Seg inherently functions as a highly scalable data engine. Utilizing this engine, we construct GEAR-131K, a massive benchmark (over 38k images, 656k QA-mask pairs) introducing a multifaceted taxonomy tailored for complex real-world manipulation-oriented reasoning. Finally, distillation experiments demonstrate that lightweight models supervised exclusively by our automated pipeline closely match the upper-bound performance of costly human-annotated baselines.