Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering
2026-08-18 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of answering questions about 3D scenes efficiently when using Vision Language Models (VLMs), which normally struggle with limited computing power. Instead of searching through many video frames for each question, they create a new method called MemTree3D that builds a compact 3D scene map in real-time as the camera moves. This map helps the system quickly find important video frames related to the question without redoing the whole search, leading to better performance. Their approach outperforms previous methods on a benchmark called OpenEQA.
Vision Language Models3D Scene RepresentationKey Frame SelectionEmbodied Question AnsweringCamera 6-DoF PosesVisual SearchLarge Language ModelOpenEQA Benchmark
Authors
Hsiang-Wei Huang, Fu-Chen Chen, Li-Wu Tsao, Cheng-Han Lee, Che-Chun Su, Lu Xia, Ronghui Peng, Jenq-Neng Hwang, Min Sun, Cheng-Hao Kuo
Abstract
Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D