ReCoSplat: Autoregressive Feed-Forward Gaussian Splatting Using Render-and-Compare
2026-03-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present ReCoSplat, a method to create new views of scenes from a series of images, even when the camera positions are unknown or imperfect. They use a technique called Gaussian Splatting to represent scenes, which builds the scene in parts based on the camera's viewpoint. To solve problems caused by guessing camera positions, they introduce a module called Render-and-Compare that checks the current scene against new images to correct errors. They also develop a way to store information efficiently for long sequences of frames. Their method performs very well on various tests and input types.
Novel View SynthesisGaussian SplattingCamera PosesAutoregressive ModelsRender-and-CompareKV Cache CompressionScene ReconstructionSequential ObservationsComputer VisionFeed-forward Model
Authors
Freeman Cheng, Botao Ye, Xueting Li, Junqi You, Fangneng Zhan, Ming-Hsuan Yang
Abstract
Online novel view synthesis remains challenging, requiring robust scene reconstruction from sequential, often unposed, observations. We present ReCoSplat, an autoregressive feed-forward Gaussian Splatting model supporting posed or unposed inputs, with or without camera intrinsics. While assembling local Gaussians using camera poses scales better than canonical-space prediction, it creates a dilemma during training: using ground-truth poses ensures stability but causes a distribution mismatch when predicted poses are used at inference. To address this, we introduce a Render-and-Compare (ReCo) module. ReCo renders the current reconstruction from the predicted viewpoint and compares it with the incoming observation, providing a stable conditioning signal that compensates for pose errors. To support long sequences, we propose a hybrid KV cache compression strategy combining early-layer truncation with chunk-level selective retention, reducing the KV cache size by over 90% for 100+ frames. ReCoSplat achieves state-of-the-art performance across different input settings on both in- and out-of-distribution benchmarks. Code and pretrained models will be released. Our project page is at https://freemancheng.com/ReCoSplat .