EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment
2026-04-09 • Machine Learning
Machine Learning
AI summaryⓘ
The authors created EgoEverything, a new dataset for understanding long videos taken from a first-person (egocentric) view, focusing on augmented reality (AR) uses. Unlike other datasets, theirs uses eye gaze data to see what people pay attention to when forming questions about the video. This helps capture more natural human behaviors and makes the evaluation of video understanding more realistic. It includes over 5,000 multiple-choice questions covering more than 100 hours of footage.
egocentric videoaugmented realityhuman attentiongaze trackinglong-context reasoningvideo understandingquestion answering datasetmultiple choice questions
Authors
Qiance Tang, Ziqi Wang, Jieyu Lin, Ziyun Li, Barbara De Salvo, Sai Qian Zhang
Abstract
Long context egocentric video understanding has recently attracted significant research attention, with augmented reality (AR) highlighted as one of its most important application domains. Nevertheless, the task remains highly challenging due to the need for reasoning over extended temporal contexts and diverse, unstructured activities. Although several benchmarks exist, most egocentric datasets rely on human worn cameras and focus mainly on visual content, with limited consideration of underlying user behavior when forming video-related queries. EgoEverything is a benchmark that explicitly considers human behavior by leveraging human attention signals, abstracted from gaze data, when generating questions. It comprises over 5,000 multiple choice question answer pairs, spanning more than 100 hours of video. By integrating human attention signals during question generation, it more faithfully captures natural human behavior and offers a realistic evaluation setting for long-context egocentric video understanding in AR.