Generative Simulation for Policy Learning in Physical Human-Robot Interaction
2026-04-09 • Robotics
Robotics
AI summaryⓘ
The authors created a system that uses AI models to automatically build virtual scenarios where robots help people, like scratching or bathing. These scenarios are made from simple text prompts and include realistic human movements and robot actions. They used this virtual data to train robots to perform tasks in the real world without extra fine-tuning. Their trained robots worked well even when people moved unpredictably. This is the first system that fully automates making simulations, gathering data, and teaching robots for physical human-robot interaction.
physical human-robot interactionzero-shot learninglarge language modelsvision-language modelssimulationimitation learningpoint cloudssim-to-real transfersoft-body human modelsrobot motion trajectories
Authors
Junxiang Wang, Xinwen Xu, Tiancheng Wu, Julian Millan, Nir Pechuk, Zackory Erickson
Abstract
Developing autonomous physical human-robot interaction (pHRI) systems is limited by the scarcity of large-scale training data to learn robust robot behaviors for real-world applications. In this paper, we introduce a zero-shot "text2sim2real" generative simulation framework that automatically synthesizes diverse pHRI scenarios from high-level natural-language prompts. Leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs), our pipeline procedurally generates soft-body human models, scene layouts, and robot motion trajectories for assistive tasks. We utilize this framework to autonomously collect large-scale synthetic demonstration datasets and then train vision-based imitation learning policies operating on segmented point clouds. We evaluate our approach through a user study on two physically assistive tasks: scratching and bathing. Our learned policies successfully achieve zero-shot sim-to-real transfer, attaining success rates exceeding 80% and demonstrating resilience to unscripted human motion. Overall, we introduce the first generative simulation pipeline for pHRI applications, automating simulation environment synthesis, data collection, and policy learning. Additional information may be found on our project website: https://rchi-lab.github.io/gen_phri/