MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
2026-07-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed MotionForesight, a system that learns to predict how objects will move in 3D after watching short ordinary videos of people interacting with them. Instead of relying on special information about objects, it uses patterns already learned by video models about how objects typically move. They train the system by tracking object points in videos and teaching it to forecast future movements using only the initial video frames. Their approach works well even on new objects and scenes and beats bigger models that use much more data. This shows video understanding can be transformed into predicting physical object motion.
3D motion forecastingvideo predictionmonocular videoshuman-object interactionscene flowobject trackingpretrained video modelspseudo-ground-truthlatent representationout-of-distribution generalization
Authors
Homanga Bharadhwaj, Yash Jangir
Abstract
Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/