VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
2026-08-21 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors developed VT-MUSE, a system that learns to understand touch and sight together over time, rather than separately or just at one moment. Their method first aligns visual and touch information across time, then uses a special model to fill in missing visual data and predict touch changes, helping it learn better. This improved understanding is used by a robot control policy to perform manipulation tasks. Their approach showed better results than previous methods both in simulations and real-world tests.
visuotactile manipulationmultimodal learningtemporal alignmentmasked-view consistencyvariational latent modelTransformercross-attentionauxiliary decoderrobot policy
Authors
Congsheng Xu, Qiaochu Yang, Fangyuan Shi, Yifan Han, Baijun Chen, Yiming Wang, Haonan Zhao, Daolin Ma, Xiaokang Yang, Hesheng Wang
Abstract
We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.