ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
2026-07-24 • Robotics
Robotics
AI summaryⓘ
The authors created ViTacWorld, a model that helps robots learn to use both sight and touch together for tasks that involve physical contact. Since collecting real tactile data is hard and limited, they combined real and simulated data to train their model. ViTacWorld can predict what a robot would see and feel after taking certain actions, allowing for better learning and testing of robot policies. Their approach improves robot performance by generating more data and checking how well the robot might act before trying it in real life.
robot manipulationtactile sensingvisuo-tactile learningworld modelsimulation-to-real transferpolicy evaluationdata augmentationrobot controltrajectory predictioncontact-rich tasks
Authors
Yunao Huang, Shiyu Sang, Haotao Lu, Suting Ni, Shijie Wu, Ziyang Guo, Ye Shi, Jingya Wang
Abstract
Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/