REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image
2026-05-28 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a method called REST3D that turns a single photo into a 3D scene that behaves physically correctly, meaning objects don’t float or overlap unrealistically. They use a special process to understand how objects support each other under gravity and organize the scene like a tree. Then, they adjust the scene using physics rules to make sure everything is stable while still matching what the photo shows. Their method works better than previous ones for making scenes that can be used in simulations and virtual reality. They tested it on both made-up and real images and demonstrated its use in VR interactions.
3D scene reconstructionsingle RGB imagephysical stabilitygravity-support relationshipsscene-tree representationphysics-constrained optimizationsimulation stabilityimage-to-3D modelsvirtual realityhuman-object interaction
Authors
Xiaoxuan Ma, Jiashun Wang, Nicolas Ugrinovic, Yehonathan Litman, Kris Kitani
Abstract
Reconstructing physically stable 3D scenes from a single RGB image enables casual images to be converted into simulation-ready digital assets for applications such as immersive interaction and content creation. However, existing single-image reconstruction methods fall short in capturing the physical structure of a scene. As a result, they often produce geometrically plausible but physically inconsistent results, including object floating and penetration, which lead to unstable behavior in physics simulations. Image-conditioned scene generation methods improve physical plausibility but often rely on strong scene priors, yielding plausible yet inaccurate object arrangements that fail to match the input image. We propose REST3D, a single-image reconstruction framework that can reconstruct physically stable 3D scenes by integrating physical scene understanding with physics-constrained refinement. We first introduce an agentic physical scene understanding technique that constructs a scene-tree representation capturing object physical states and inter-object relationships from a gravity-support perspective, providing a structural prior for reconstruction. Leveraging this structure, we initialize the scene using image-to-3D models, followed by scene-tree-guided alignment and physics-constrained optimization to resolve physical violations while preserving visual consistency with the input image. Experiments show that our method significantly reduces physical errors and improves simulation stability on both synthetic and real-world datasets while maintaining strong reconstruction quality. We further demonstrate the reconstructed scenes in VR-based human-object interaction, showing their potential for immersive applications.