AI summaryⓘ
The authors address the problem of figuring out where a person is looking in a scene, which is called gaze target estimation. Traditional methods either need lots of extra labeled data or focus too much on simple visual clues that don't always match where a person actually looks. They propose TextGaze, a new method that uses a large vision-language model to provide meaningful text cues that help predict gaze more accurately without needing extra annotations. Their approach combines visual features and text clues in a smart way and works well across different datasets without extra training. This shows that using vision-language models can improve gaze estimation while making the system simpler and more flexible.
gaze target estimationvision-language modelsvisual featurestransformer fusionheatmapscross-modal learningtextual cuesmulti-branch methodsannotation burdencross-dataset generalization
Authors
Junhui She, Fei Wang, Kun Li, Yiqi Nie, Yuxin Liu, Zhangling Duan, Xun Yang
Abstract
Gaze target estimation aims to infer the position of a person's gaze within a scene. Within mainstream design logic, multi-branch methods require extra supervision and annotations, while streamlined designs prioritize low-level visual saliency over true gaze intent. The former leads to a high annotation burden and hinders domain transfer, whereas the latter causes misalignment between predicted attention and actual gaze targets. To address this issue, we propose TextGaze, a unified cross-modal architecture that leverages a Large Vision-Language Model (LVLM) as scalable semantic guidance to balance the two design paradigms. The model extracts visual features from a frozen encoder and utilizes an LVLM to obtain gaze-aligned textual cues. We design a transformer-based fusion module with hierarchical text supervision to preserve task semantics. Lightweight decoding heads enable the joint prediction of gaze heatmaps and in-/out-of-frame status. We evaluate our method on four mainstream datasets, and the results show competitive performance across key metrics with robust cross-dataset generalisation without extra fine-tuning. Overall, we provide a streamlined alternative to traditional designs and highlight the potential of LVLMs as accessible auxiliary guidance for gaze estimation.