Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations
2026-09-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied 3D Foundation Models (3DFMs), which are powerful AI models that understand 3D scenes from images. They found these models store useful hidden details about surfaces in a scene, even from views the model hasn't seen before. Using this insight, the authors created a method called Z3D that can generate realistic depth maps (which show how far things are) for new viewpoints by manipulating the internal model representations. Their work shows how to use existing 3DFMs for 3D reconstruction of scenes from new angles.
3D Foundation ModelsVGGTfeed-forward transformers3D vision3D reconstructionlatent diffusionpointmapsdepth maps
Authors
Denis M. Akola, David F. Fouhey
Abstract
3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-foward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, we investigate using their internal representations to infer 3D in the scene from new views. Our hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes. After showing that it is possible to decode hidden surfaces from internal 3DFM representations, we propose a method, Z3D, that estimates pointmaps in unseen views by doing latent diffusion on 3DFM representation. We show that Z3D can predict realistic depth maps for new views across multiple datasets.