ViTacPhys: Physical Property-Aware Grasping from Human Visual-Tactile Demonstrations
2026-08-21 • Robotics
Robotics
AI summaryⓘ
The authors created ViTacPhys, a system that uses both vision and touch to estimate how heavy an object is, how slippery its surface is, and how squishy it is, based on watching humans handle objects. They trained it on many objects and showed it works well even on new objects. Then, they adapted the system for robots and used it to help robots grasp objects better by adjusting to these physical properties. Their robot system was more successful and mimicked human touch patterns more closely than a previous method. This work shows it’s possible and helpful to include physical property knowledge in robot grasping.
visual-tactile sensingobject mass estimationfriction coefficientstiffness measurementmultimodal fusionvision-language modelrobot graspinghuman teleoperationadaptive manipulationdata augmentation
Authors
Yiwen Liu, Yujun Zhu, Kui Jia, Zhao Liao, Yangwei You, Shuaijun Wang
Abstract
Recent vision-based action models have demonstrated strong capabilities in complex manipulation, but they rarely leverage explicit object physical properties to adapt their policies. We introduce ViTacPhys, a visual-tactile framework and data acquisition system that estimates object mass and friction-coefficient classes, together with continuous stiffness, from human manipulation demonstrations. Trained on data from 60 rigid and deformable objects, ViTacPhys combines temporal visual-tactile modeling, cross-attention multimodal fusion, and a semantic prior derived from a vision-language model. On seen objects, it achieves 97.2% mass classification accuracy, 98.8% friction-coefficient classification accuracy, and a stiffness mean absolute percentage error (MAPE) of 5.51%. On held-out objects from known categories, it achieves 87.5% mass accuracy, 97.5% friction-coefficient accuracy, and a stiffness MAPE of 9.08%. We transfer ViTacPhys from the human domain to the robot domain using limited robot teleoperation data, robot-style video augmentation, and human demonstrations with matched actions, and deploy it as an online module for adaptive grasping. The resulting physical-property-conditioned policy achieves total grasping success rates of 95.0% on in-distribution objects and 83.4% on out-of-distribution objects. For out-of-distribution objects successfully grasped by both methods, its force profiles are more consistent with human teleoperation than those produced by ACT. These results demonstrate the feasibility of explicitly estimating and conditioning on object physical properties for real-world adaptive grasping.