Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created Evita, a new kind of AI model that combines regular RGB camera frames with event-based camera data to improve perception, especially in difficult conditions. Unlike past methods that either use two separate systems or generic combined designs, Evita unifies them with special modules that fix alignment issues and better handle differences between the two data types. They also developed new training strategies and datasets to help Evita work well with various event data formats. Testing on standard benchmarks showed Evita achieves better accuracy and speed than previous approaches.
RGB framesEvent camerasMultimodal visionGeometric parallaxSparse and dense dataFrequency domainAttention mechanismPretrainingReal-time perceptionBenchmark datasets
Authors
Chenxu Peng, Chongtian zhou, Dicheng Liu, Bo-Wen Yin, Yimian Dai, Xialei Liu, Ming-Ming Cheng, Xiang Li
Abstract
Fusing standard RGB frames with asynchronous event streams has emerged as a definitive paradigm for robust perception in degraded environments. Although unified backbones have recently gained traction in multi-modal vision, adapting them to the RGB-Event domain remains fundamentally challenging. Existing architectures either resort to decoupled dual encoders that double computational overhead, or adopt generic unified designs that fail to resolve implicit geometric parallax and cross-spectral aliasing under the extreme representational divide between dense intensity grids and sparse kinematic spikes. To transcend these bottlenecks, we present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing. To achieve profound modal synergy, Evita explicitly embeds a suite of intrinsic co-learning modules directly into every encoder layer. Specifically, it features Geometric Parallax Rectification for adaptive spatial alignment, Harmonic Spectral Resonance for texture transfer exclusively in the complex frequency domain, and Transient Global Routing for event-driven asymmetric attention. To guarantee robust feature extraction against spatial misalignments and decouple representations from specific event encodings, we construct N-ImageNetV2 alongside a stochastic event representation mixing pretraining protocol, empowering the network to seamlessly accommodate arbitrary event formats in downstream tasks. Extensive evaluations across the DELIVER, DDD17, and DSEC benchmarks confirm that Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.The code are publicly available at: https://github.com/chaineypung/Evita.