AI summaryⓘ
The authors created a large new dataset called Gesture Recognition in the Wild (GRW) to help computers understand the specific hand movements people make while talking. These gestures often relate closely to certain words, but previous models had trouble learning this because of limited training data. GRW has over 150,000 video clips with detailed annotations showing exactly when and what gestures happen linked to 150 different word categories. The authors used this dataset to teach models to identify whether a gesture is meaningful, to recognize the word connected with a gesture, and to pinpoint when the gesture occurs in the video. They also set up tests to measure how well models perform on these tasks.
co-speech gesturesmultimodal modelsgesture recognitionvideo annotationtemporal localizationsemantic classificationdataset benchmarkphysical actionsspatial descriptorsabstract concepts
Authors
Sindhu B Hegde, K R Prajwal, Andrew Zisserman
Abstract
While humans naturally gesture during speech, only a sparse subset of these movements are visually depictive and semantically linked to specific spoken words. Current multimodal models struggle to capture these semantic co-speech gestures, heavily bottlenecked by a lack of precisely annotated training data. To address this, we introduce the Gesture Recognition in the Wild (GRW) dataset, the first large-scale benchmark designed to map unconstrained human gestures to specific words with frame-accurate temporal boundaries. Comprising 156,688 manually annotated video clips, GRW spans a highly diverse 150-word taxonomy of physical actions, spatial descriptors, and abstract concepts. We leverage GRW to train video models to (a) classify gestures as semantic or not, (b) recognize the word corresponding to a co-speech gesture, and (c) temporally localize the gesture. We also use GRW to establish benchmarks for these three tasks.