TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction
2026-04-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed TAIHRI, a new system that helps robots better understand and locate important parts of a person's body in 3D during close-up interactions. Instead of focusing on the whole body, TAIHRI zeroes in on key body parts relevant to the robot's tasks by using a combination of visual and language information. It converts 3D points into a simpler space to predict important spots from 2D images and can follow motion commands or recover full body shapes. Tests show that TAIHRI is more accurate in estimating the positions of critical body parts from the robot's viewpoint. This work could improve how robots interact naturally and safely with humans nearby.
3D human keypointshuman-robot interactionegocentric cameravision-language modelmetric-scale localizationnext token predictiontask-relevant body partshuman mesh recoveryspatial coordinatesclose-range perception
Authors
Ao Li, Yonggen Ling, Yiyang Lin, Yuji Wang, Yong Deng, Yansong Tang
Abstract
Accurate 3D human keypoints localization is a critical technology enabling robots to achieve natural and safe physical interaction with users. Conventional 3D human keypoints estimation methods primarily focus on the whole-body reconstruction quality relative to the root joint. However, in practical human-robot interaction (HRI) scenarios, robots are more concerned with the precise metric-scale spatial localization of task-relevant body parts under the egocentric camera 3D coordinate. We propose TAIHRI, the first Vision-Language Model (VLM) tailored for close-range HRI perception, capable of understanding users' motion commands and directing the robot's attention to the most task-relevant keypoints. By quantizing 3D keypoints into a finite interaction space, TAIHRI precisely localize the 3D spatial coordinates of critical body parts by 2D keypoint reasoning via next token prediction, and seamlessly adapt to downstream tasks such as natural language control or global space human mesh recovery. Experiments on egocentric interaction benchmarks demonstrate that TAIHRI achieves superior estimation accuracy for task-critical body parts. We believe TAIHRI opens new research avenues in the field of embodied human-robot interaction. Code is available at: https://github.com/Tencent/TAIHRI.