LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation
2026-06-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study how to improve long videos made step-by-step by a computer model, which often get worse as they go on because they focus only on a small recent part of the video. They propose LongLive-RAG, a new method that looks back at the whole past video history to help the model remember and stay consistent. This approach includes a special training trick to make the memory retrieval smarter and reduce repeated mistakes. Their tests show better quality in long video generation compared to previous methods.
autoregressive video diffusionsliding-window attentionlong video generationretrieval-augmented generation (RAG)latent variablesquery embeddingtemporal consistencywindow temporal delta losscontent-addressable memoryVBench-Long
Authors
Qixin Hu, Shuai Yang, Wei Huang, Song Han, Yukang Chen
Abstract
Autoregressive (AR) video diffusion enables variable-length synthesis, but long-horizon generation often suffers from accumulated errors and identity drift. For efficiency, existing methods commonly adopt sliding-window attention during generation. This creates an irreversible generation trajectory: once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. We address this limitation by formulating long video generation as a retrieval-augmented generation (RAG) problem. Rather than relying solely on the recent window, we treat previously generated latents as a dynamic, searchable history. We propose LongLive-RAG, a general retrieval framework for AR video generation. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents. This lightweight retrieval step adds only a small overhead relative to generation and lets the generator condition on non-local context instead of only the recent window. To make retrieval more discriminative, we introduce the Window Temporal Delta Loss that suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. Together, these components help reduce error accumulation caused by sliding-window attention. Experiments across multiple AR backbones and generation lengths show improved long-video quality and the best average VBench-Long rank. To our knowledge, among open-ended AR long video generation methods, LongLive-RAG is the first to formulate self-generated latent history as content-addressable retrieval memory. Code is available at https://github.com/qixinhu11/LongLive-RAG.