T-Rex: Tactile-Reactive Dexterous Manipulation
2026-06-15 • Robotics
Robotics
AI summaryⓘ
The authors focus on improving robots' ability to respond to touch, which is important for delicate and skilled actions. They collected a large, new set of touch data by having robots perform simple repeated movements. To handle this complex touch information, they designed a new model that processes signals at different speeds using a special encoder. Their approach worked better than previous methods on tasks that need careful force and deformable object handling, showing notable improvement. This research helps robots use touch more effectively to manipulate objects.
tactile sensingrobotic manipulationVision-Language-Action (VLA) modelsMixture-of-Transformers (MoT)VQ-VAE encodertemporal encodingforce controldeformable object manipulationdata-efficient learningrobot dexterity
Authors
Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Fei-Fei Li, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Jim, Fan, Trevor Darrell
Abstract
The ability to react dynamically to tactile signals has long been considered crucial to agile human-level dexterity. Yet contemporary learning-based Vision-Language-Action (VLA) models for robotic manipulation generally either overlook the tactile modality or are limited to encoders with static cues, due in part to the scarcity of diverse training data and standardized evaluation, architectural constraints in current VLA models, and limitations of static tactile encoders. In this paper, we push the frontier of tactile-reactive manipulation by addressing all of these limitations. We propose a large-scale, 100-hour tactile-rich dataset collected via a novel, data-efficient recipe that prioritizes elementary motor primitives. To effectively exploit naturally high-frequency touch signals without sacrificing the existing capabilities of existing VLAs, we introduce a variable-rate Mixture-of-Transformers (MoT) architecture equipped with a novel temporal tactile VQ-VAE encoder. We demonstrate the effectiveness of tactile-reactive policies on 12 manipulation tasks requiring delicate force control and deformable object manipulation, achieving over 30% higher average success rate than the strongest baseline.