Toward Semantic Communication for Real-time Mobile 3D Reconstruction
2026-07-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMultiagent SystemsMultimedia
AI summaryⓘ
The authors developed a system for real-time 3D reconstruction using images sent from a moving device to a server. Their method sends both a compressed image and a confidence map showing which parts of the image are more reliable. This confidence information helps the system ignore bad data when estimating camera positions and 3D shapes, making the results more accurate even when the communication channel is noisy. Tests show their approach works better than previous methods in keeping image quality high and improving 3D reconstruction accuracy.
real-time 3D reconstructionsemantic communicationpose estimationbundle adjustmentRANSACconfidence mapmobile computingmulti-view consistencychannel noiseimage transmission
Authors
Fangzhou Zhao, Yao Sun, Xuesong Liu, Runze Cheng, Shang Kai, Yi Sun
Abstract
Real-time mobile 3D reconstruction is fundamental to many emerging applications such as autonomous navigation and digital twin construction, where a moving platform continuously captures an image stream and transmit to a computing server for scene understanding. Unlike offline reconstruction, camera poses and scene geometry are estimated on-the-fly during acquisition, making multi-view consistency a real-time requirement and rendering geometric estimation highly sensitive to communication-induced distortions. Semantic communication (SemCom) transmits compact semantic information, offering a promising way to preserve task-critical data over unreliable links. However, existing designs are optimized at the image or single-view level and without providing explicit reliability information for geometric estimation, limiting their applicability to real-time mobile 3D reconstruction. In this context, we propose a SemCom framework for real-time mobile 3D reconstruction. The framework includes a semantic transceiver that outputs a reconstructed image alongside a pixel-wise confidence map, quantifying the reliability of each region. We further introduce a confidence-guided geometric estimation method, incorporating confidence into RANSAC-based pose initialization and bundle adjustment to reduce the influence of unreliable regions and enhance robustness under noisy channels. Simulations show that, compared to existing SemCom and traditional seperate source and channel coding, our framework maintains high image quality while significantly improving pose estimation accuracy and 3D structural consistency.