MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation
2026-06-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on creating videos that show new camera angles based on a single original video, keeping both the scene’s 3D shape and movement accurate. They point out that previous methods either struggled with dynamic object shapes or with keeping motion and geometry consistent. To fix this, the authors introduce a new approach called MVTrack4Gen that uses multi-view point tracking to help the model understand and follow motion across different views and times. Their method improves how well new videos match the original motion and maintain geometric accuracy better than earlier techniques.
novel-view video synthesismonocular videogeometric consistencymotion fidelitymulti-view point trackingdiffusion modelscamera conditioning3D reconstructionattention layersvideo generation
Authors
JoungBin Lee, Jaewoo Jung, Jongmin Lee, Tongmin Kim, Hyunsung Kim, Takuya Narihira, Kazumi Fukuda, Jahyeok Koo, Jisang Han, Yuki Mitsufuji, Seungryong Kim
Abstract
Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen (Multi-View point Tracking for Novel-View Generation), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy.