X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras
2026-07-14 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors introduce X-lens, a small and fast model that estimates real-world distances from multiple camera views, including fisheye and regular lenses. It uses special tokens to align different camera types and a distortion bias to maintain consistency across views, which helps it work well with few parameters and high speed. X-lens predicts detailed depth with real-world scale directly, without extra tasks that slow training. The authors trained it on diverse datasets, including a large new synthetic one they created, and showed it performs better and more efficiently than previous methods on depth estimation from heterogeneous cameras.
metric depth estimationfisheye camerapinhole cameracross-attentionlearnable calibration tokensJacobian-parameterized distortioncross-camera consistencyOmniScene datasetdepth scale estimationsynthetic datasets
Authors
Heng Zhou, Shuhong Liu, Yonghao He, Bohao Zhang, Fa Fu, Chenhui Hou, Xianbao Hou, Lijun Han, Wei Sui
Abstract
We present X-lens, a compact feed-forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. To support real-time downstream perception, X-lens is built around a geometry-aware heterogeneous camera formulation with two key components. Learnable calibration tokens provide a coarse alignment between fisheye and pinhole projective spaces, while a Jacobian-parameterized distortion bias injected into cross-attention models local projection changes and promotes cross-camera consistency, enabling robust generalization with only 0.04B parameters and up to 41 FPS. The model predicts dense depth together with a global metric scale, avoiding auxiliary reconstruction targets that increase computation and optimization complexity. To learn such cross-camera generalization at scale and depth, X-lens is trained on multiple public datasets and OmniScene, our newly released large-scale synthetic dataset containing approximately 266K synchronized six-view frames, 1.7M individual images, and 103 indoor and outdoor scenes. Extensive experiments on both real-world and synthetic indoor and outdoor datasets demonstrate superior heterogeneous-camera metric depth accuracy, reducing AbsRel by 25.4\% on OmniScene-Full over the strongest baseline while using 88.9\% fewer parameters, with competitive performance on conventional fisheye-only and pinhole-only settings.