ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models
2026-07-14 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors designed ViCo3D, a new way for connected vehicles to better detect objects using LiDAR sensors by combining 3D point cloud data with powerful image-based features from pretrained vision models. They convert 3D LiDAR data into bird's-eye-view images so an image model (DINOv2) can extract useful visual information, which they then merge with traditional geometric features. Their approach also shares information efficiently between multiple vehicles to improve detection accuracy. Tests show ViCo3D outperforms previous methods on standard datasets, especially in collaborative scenarios.
LiDARVehicle-to-Everything (V2X)Bird's-Eye-View (BEV)Vision Foundation Models (VFMs)DINOv23D Object DetectionPoint CloudCross-Agent FusionCollaborative PerceptionDAIR-V2X
Authors
Haojie Ren, Songrui Luo, Lingfeng Wang, Yan Xia, Yao Li, Jing Li, Lu Zhang, Jiajun Deng, Yanyong Zhang
Abstract
LiDAR-based collaborative 3D perception in Vehicle-to-Everything (V2X) systems typically relies on fusing bird's-eye-view (BEV) features across agents. However, current BEV representations, typically extracted by LiDAR backbones trained from scratch, are geometry-dominated and lack general semantic priors, inherently limiting the efficacy of feature-level collaboration. Meanwhile, vision foundation models (VFMs) pretrained on large-scale image data have demonstrated strong capability in learning general-purpose and informative visual representations for 2D tasks, and have the potential to enhance agent-wise LiDAR BEV representations for collaboration. Despite this potential, adapting VFMs to LiDAR-based 3D detection remains challenging due to the substantial image-point cloud modality gap. To bridge this gap, we propose ViCo3D, a collaborative 3D object detection framework powered by VFMs. Specifically, ViCo3D adapts VFMs to LiDAR-based collaborative perception from three aspects: First, ViCo3D projects point clouds onto the BEV plane as three-channel images, enabling DINOv2 to extract BEV-space visual features from LiDAR inputs. Besides, to effectively integrate these DINOv2-derived features with LiDAR geometric features, ViCo3D introduces a multi-scale BEV fusion module within the single-agent encoder. In addition, ViCo3D adopts an ego-centric cross-agent fusion strategy to aggregate complementary information from multiple agents. Experiments on DAIR-V2X and V2XSet demonstrate that ViCo3D achieves state-of-the-art 3D detection performance. Remarkably, it delivers up to 1.8x greater collaborative gains than prior methods on DAIR-V2X. The code will be made public available for future investigation.