Privacy-Preserving Depth-Only Open-Vocabulary 3D Semantic Segmentation Via Uncertainty-Guided Test-Time Optimization
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionRobotics
AI summaryⓘ
The authors focus on understanding indoor 3D scenes without using RGB images to protect privacy. Since depth data alone lacks color and texture details, making accurate semantic guesses is hard. They created UTTO, a method that uses uncertainty in its predictions to improve results by leveraging existing knowledge from large pre-trained models. Tests show UTTO makes better 3D semantic segmentation in a privacy-safe way than current methods, without needing extra training.
3D semantic segmentationprivacy-preserving perceptiondepth dataopen-vocabularyuncertainty-guided optimizationtest-time optimizationfoundation modelsScanNet datasetsemantic priorsindoor scene understanding
Authors
Xuying Huang, Sicong Pan, Maren Bennewitz
Abstract
Privacy-preserving perception is a critical requirement for deploying 3D scene understanding systems in real-world indoor environments, yet it remains underexplored in open-vocabulary 3D semantic segmentation. Existing methods typically rely on obtaining rich semantic cues from RGB images, which may expose privacy-sensitive visual information. Depth-only 3D geometry provides a privacy-preserving alternative, but the absence of appearance-based semantic cues makes open-vocabulary predictions highly uncertain and less reliable. Under this setting, we propose to convert uncertainty into a guidance signal to identify unreliable semantic responses and use semantic priors from foundation models to regularize their refinement. We present UTTO, an uncertainty-guided test-time optimization framework for depth-only open-vocabulary 3D semantic segmentation. Without additional training, experiments on ScanNet20, ScanNet40, and ScanNet200 demonstrate that UTTO consistently improves depth-only open-vocabulary 3D segmentation and outperforms representative baselines under privacy-preserving conditions.