Video Generative Models as Geometry Learner
2026-08-28 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created a new method called GeoNeXt that uses pretrained video generation models to estimate 3D geometry like depth and surface normals from images. Unlike previous methods that either trained separate models for each geometry type or needed lots of labeled data to fine-tune one model, GeoNeXt treats the problem like predicting the next video frame, which helps it learn more efficiently. Their approach leverages the knowledge in video models to relate images and geometry together, requiring much less data. Tests show their method works well without extra training and beats earlier generative methods, performing nearly as well as top models that use much more data.
geometry estimationimage diffusion modelsvideo generative modelsmonocular depth estimationsurface normal estimationself-attentionzero-shot learningnext-frame predictiondata efficiencygenerative modeling
Authors
Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng
Abstract
Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.