Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs
2026-07-02 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on understanding how humans interact with their environment by analyzing first-person videos. They created SG-Ego, a detailed dataset that shows how objects and people in a scene change over time using structured graphs. To handle these graphs, they designed GLEN, a model that links these changing relationships to actions and predicts how scenes evolve. They also posed a new challenge called A-GEF, which asks models to forecast scene changes based on ongoing actions. Their approach worked well on several tests, showing that explicitly modeling scene dynamics is useful for understanding activities in videos.
embodied AIfirst-person videosscene graphsspatio-temporal reasoninggraph neural networksactivity recognitionvideo understandingtemporal dynamicsstructured predictiongraph editing
Authors
Francesca Pistilli, Simone Alberto Peirone, Giuseppe Averta
Abstract
Understanding human behavior while interacting with the surrounding world is crucial for many applications of embodied AI. First-person videos are particularly informative for this problem, as they well capture how activities reshape the scene over time. However, existing approaches often rely on implicit visual or language-aligned representations, disregarding structured reasoning over the scene dynamic. We argue that explicit, compositional and editable representations of human-environment interactions can play a crucial role for rich grounded activity understanding. To this end, we introduce SG-Ego, a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. To reason over this representation, we propose GLEN, a graph-based model that operates over scene graph sequences to both align them with textual actions and model their temporal evolution. In addition, we formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations conditioned on ongoing actions, enabling explicit reasoning about how scenes change over time. We validate our approach across multiple downstream tasks, spanning retrieval benchmarks as EgoMCQ and EgoCVR, as well as long-horizon reasoning benchmarks as EXPLORE-Bench and the newly introduced A-GEF. GLEN achieves strong results compared to raw video baselines and it excels in reasoning settings, typically addressed only with MLLMs, while enabling controllable and structured predictions of scene dynamics driven by human activities. We believe our results establish spatio-temporal scene graphs, together with models that reason over them, as strong compositional and interpretable representations for video understanding and potentially beyond.