Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models
2026-07-06 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors created Deform360, a large dataset capturing how everyday soft objects move and change shape when touched or manipulated, using many cameras and tactile sensors. They used this data to compare different methods for predicting object movement: models that learn from regular 2D video versus those using detailed 3D shapes. Their work helps understand which approach works better for various tasks and shows how this dataset can assist robots in planning actions with soft objects. This offers a new standard for improving how robots understand and predict deformable objects in real life.
deformable objectsworld modelingrobotic manipulationvisuotactile sensing3D tracking2D video modeling3D particle modelsdatasetrobot planninggeometry extraction
Authors
Hongyu Li, Wanjia Fu, Xiaoyan Cong, Zekun Li, Binghao Huang, Hanxiao Jiang, Xintong He, Yiqing Liang, Rao Fu, Tao Lu, Srinath Sridhar, Kevin A. Smith, George Konidaris, Yunzhu Li
Abstract
Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz