VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds
2026-07-17 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors present VTLoc, a system that helps robots figure out exactly where they are touching an object by combining what they see and what they feel. Vision gives a big-picture view of the object, while touch provides detailed information at the points of contact. VTLoc uses a special method to match and align touch data with the visual shape of the object in 3D and then improves its guess about the touch location step-by-step. Testing on many real objects showed that their approach makes it easier for robots to understand where they've touched.
robotic perceptiontouch sensingvisual-tactile integration3D point cloudcontact localizationmulti-modal alignmentspatial alignmentiterative refinement
Authors
Zhiyuan Wu, Zhuo Chen, Shan Luo
Abstract
Vision and touch are complementary modalities essential for robotic perception and manipulation. While vision provides global object context, touch offers precise local information at contact points. Integrating these modalities for contact localization, i.e., predicting the location of touch on an object's surface, poses significant challenges due to the need for accurate spatial alignment between tactile data and visual geometry. To address this challenge, we propose VTLoc, a novel visual-tactile framework that localizes contact points from tactile readings using a 3D point cloud as visual input. VTLoc introduces two key components: a geometric multi-modal alignment module, which reconstructs a pseudo-point cloud from fused visual-tactile features and aligns it with the visual point cloud to enforce spatial consistencies across modalities; and an iterative localizing updater, which iteratively refines the predicted contact location using fused visual-tactile features. Evaluated on a new benchmark of 100 real-world objects, VTLoc improves single-touch contact localization by reducing local-to-global correspondence ambiguity.