Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
2026-07-14 • Artificial Intelligence
Artificial IntelligenceSound
AI summaryⓘ
The authors explore a new way to turn speech into text using a diffusion language model that refines the whole sentence at once instead of one word at a time like usual methods. They connect a large pre-trained text model to an audio encoder through a small trained part, but found normal training didn’t work well at first. By using a special loss function called connectionist temporal classification, the model learned to better understand speech sounds. Their final system can transcribe speech with low errors in just a few steps and works across multiple languages using a single trained adapter.
automatic speech recognitionautoregressive decoderdiffusion language modelWhisper encodermixture-of-experts modelconnectionist temporal classificationword error rateadaptermultilingual transcription
Authors
Harsha Vardhan Khurdula, Abhinav Kumar Singh, Yoeven D Khemlani, Vineet Agarwal
Abstract
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.