NPUsper: Eliminating Redundant Computation for Real-Time Whisper on Mobile NPUs
2026-07-01 • Sound
Sound
AI summaryⓘ
The authors developed NPUsper, a system that makes Whisper's speech-to-text faster and more efficient on mobile neural processing units (NPUs). Their method detects repeated or unnecessary words early to avoid extra work, letting the system handle shorter chunks of audio without delay. They also introduced a new way to organize the decoding steps to reduce computing overhead on mobile devices. This resulted in much faster responses and lower power use while keeping similar accuracy. The system's code is publicly available for others to try.
Whisperlive transcriptionmobile NPUautoregressive decodingdecoder cross-attentionchunk graphsKV-cachelatencytime-to-first-tokenpower consumption
Authors
Sihyeon Lee, Hojeong Lee, Sungwon Woo, Chengpo Yan, Suman Banerjee, Seyeon Kim
Abstract
We present NPUsper, a live transcription system that makes Whisper efficient on mobile NPUs by eliminating redundant computation. To avoid the heavy padding used by prior streaming systems, NPUsper detects hallucinated tokens online from temporal patterns in decoder cross-attention, allowing each inference round to process short audio inputs with minimal carryover. For efficient mobile-NPU execution, we propose controlled unrolling, which executes autoregressive decoding as K-step chunk graphs, removing unnecessary KV-cache computation and reducing graph-dispatch overhead. NPUsper achieves up to 4.84x lower per-word latency, up to 33.2x lower time-to-first-token (TTFT), and up to 88.64% lower average power consumption compared with baselines, while maintaining comparable transcription accuracy. The code is available at https://github.com/npusper/NPUsper.