OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
2026-07-03 • Machine Learning
Machine LearningArtificial IntelligenceComputer Vision and Pattern RecognitionSound
AI summaryⓘ
The authors identify that existing methods for reducing the number of audio-visual tokens processed by large language models often ignore important timing differences between sound and video. They introduce OmniFocus, a new method that independently estimates which parts of audio and video are important, keeping relevant information from both while balancing their importance over time. Their approach works without additional training and helps the model stay accurate while using fewer tokens, which speeds up processing. Experiments show that OmniFocus performs better than existing methods, especially when only a quarter of the tokens are retained.
Omni modal large language modelsaudio-visual inputstoken compressiontemporal localitymodality biasquery-guided compressionQwen2.5-Omniinference costtoken retentionaccuracy-efficiency trade-off
Authors
Shijie Cao, Qingyu Zhang, Boxi Yu, Yuzhong Zhang, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
Abstract
Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost. Existing audio-visual token compression methods often rely on unimodal guidance, overlooking the temporal locality of query-relevant evidence in audio-visual inputs and implicitly assuming that the two modalities share a temporally aligned information density distribution. We propose \textbf{OmniFocus}, a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio, enabling a modality-symmetric compression design that preserves modality-specific salient evidence while maintaining audio-visual alignment, thereby mitigating the modality bias issue that can arise from unimodal-guided compression. Experiments on the Qwen2.5-Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios and outperforms existing baselines on several major benchmark scores at 25\% token retention. On DailyOmni with Qwen2.5-Omni-7B at 25\% token retention, OmniFocus maintains 59.40 accuracy while delivering up to 1.38$\times$ prefill speedup relative to the full-token baseline, highlighting a favorable practical accuracy-efficiency trade-off.