Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning
2026-06-17 • Robotics
Robotics
AI summaryⓘ
The authors created a system that can follow language instructions to perform complex hand movements with objects, using ordinary cameras from different angles instead of special sensors. They use a language-and-vision model to find key points in 2D images, then combine these views to figure out the exact 3D positions. This helps their robot pick up, place, or use tools by matching actions to the objects and planning movements accordingly. Their approach works better than previous methods and can handle new objects and tasks without extra training.
zero-shot learningdexterous manipulationvision-language modelmulti-view fusion3D keypointspick-and-placetool-usegrasp affordancetask planningclosed-loop control
Authors
Jisoo Kim, Sangwon Baik, Taeksoo Kim, Sungjoo Kim, Junyoung Lee, Mingi Choi, Hanbyul Joo
Abstract
We present a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Rather than training an end-to-end policy, our system uses a vision-language model (VLM) to produce reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D via multi-view fusion. This lifting combines triangulation of view-wise VLM groundings with reference-view ray voting, which searches along a semantic camera ray for geometrically consistent candidates across neighboring views. The resulting 3D keypoints support both pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the scene; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines. We further demonstrate long-horizon manipulation through closed-loop status verification and replan, enabling zero-shot execution on unseen objects and tool-use tasks in novel scenes.