NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning
2026-02-23 • Robotics
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors created NovaPlan, a system that helps robots do long tasks by combining smart thinking with real physical actions. It uses language and video models to break down big tasks and imagine steps but also keeps track of what the robot is actually doing to fix mistakes. NovaPlan looks at important object points and hand movements from videos to guide the robot's actions, even when the robot has trouble seeing clearly. They tested it on complicated tasks and the Functional Manipulation Benchmark, showing it can work well without needing special training or examples beforehand.
Vision-language modelsVideo generation modelsHierarchical planningKeypoint extractionKinematic priorsClosed-loop controlLong-horizon manipulationError recoveryRobot executionFunctional Manipulation Benchmark
Authors
Jiahui Fu, Junyu Nan, Lingfeng Sun, Hongyu Li, Jianing Qian, Jennifer L. Barry, Kris Kitani, George Konidaris
Abstract
Solving long-horizon tasks requires robots to integrate high-level semantic reasoning with low-level physical interaction. While vision-language models (VLMs) and video generation models can decompose tasks and imagine outcomes, they often lack the physical grounding necessary for real-world execution. We introduce NovaPlan, a hierarchical framework that unifies closed-loop VLM and video planning with geometrically grounded robot execution for zero-shot long-horizon manipulation. At the high level, a VLM planner decomposes tasks into sub-goals and monitors robot execution in a closed loop, enabling the system to recover from single-step failures through autonomous re-planning. To compute low-level robot actions, we extract and utilize both task-relevant object keypoints and human hand poses as kinematic priors from the generated videos, and employ a switching mechanism to choose the better one as a reference for robot actions, maintaining stable execution even under heavy occlusion or depth inaccuracy. We demonstrate the effectiveness of NovaPlan on three long-horizon tasks and the Functional Manipulation Benchmark (FMB). Our results show that NovaPlan can perform complex assembly tasks and exhibit dexterous error recovery behaviors without any prior demonstrations or training. Project page: https://nova-plan.github.io/