Context-Aware Interleaved Batching for WhisperX
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors discuss two ways to turn spoken audio into text: one that is fast but forgets earlier parts of the conversation, and another that remembers the conversation but is slow. They created a new method called Context-Aware Interleaved Batching that uses voice activity detection to keep track of the conversation context even while processing audio faster. This method helps make the text more accurate, especially for names and tricky words, without slowing down the transcription process. They tested it on long recordings and found it worked better than previous methods.
WhisperXspeech transcriptioncontextual text conditioningvoice activity detection (VAD)Word Error Rate (WER)batchinglong-form audioproper noun transcriptioninference speed
Authors
Carlos Bain, Max Bain
Abstract
While WhisperX accelerates speech transcription via intra-audio batching, it isolates audio segments, losing the historical context needed for coherent punctuation and terminology transcription. Conversely, standard Whisper retains context sequentially but suffers from slow inference and hallucination loops. To achieve the best of both worlds, we propose Context-Aware Interleaved Batching. By using VAD-derived segment boundaries, our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous historical context across batched audio segments. As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER) and improves proper noun transcription, all while maintaining high-throughput inference speeds.