SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies
2026-08-31 • Robotics
RoboticsArtificial Intelligence
AI summaryⓘ
The authors created a system called Kuafu that makes robots better at long tasks by combining clear, meaningful instructions with learned behaviors. They introduced SUN Programs, which keep the task goals consistent across different parts of robot control and learning, helping avoid confusion from vague rewards. Kuafu uses language and scene info to make these programs automatically, checks if they can work using control methods, and then trains the robot. Their experiments show Kuafu outperforms traditional methods and even works on real robot arms without needing lots of human examples. This approach helps connect step-by-step planning with data-based learning in robotics.
Model Predictive ControlReinforcement LearningSemantic ProgrammingRobotic ManipulationTask SemanticsSparse RewardsPolicy LearningTrajectory ScreeningSymbolic PlanningVision-Language Systems
Authors
Weiqi Wang, Zhi Li, Yudong Lei, David Martinez, Xiaofeng Gao, Yuxin Jiang, Chenfanfu Jiang, Yingnian Wu, Demetri Terzopoulos, Ran Gong
Abstract
Bridging model-based control and learned policies in long-horizon manipulation has harbored a silent disagreement: control executes specified objectives, learning amortizes that behavior into a reactive policy, yet existing protocols discard task semantics, leaving rewards hand-crafted and behavior drifting from what control verified.We introduce Semantically UNified (SUN) Programs, typed executables where geometric and contact relations are defined once and compiled into aligned Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. Our system, Kuafu, driven by large vision language systems, automatically synthesizes SUN Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. Across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines. At 8192-way scale, it generates 10.57x the successful trajectory time per hour of human teleoperation. With 500 trajectories per task, Kuafu data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka and Kinova robots. These results establish that simulation-screened task semantics can effectively amortize control into robust policies, without demonstrations or manual dense rewards, unifying symbolic planning and data-driven execution.