REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing
2026-07-06 • Computation and Language
Computation and LanguageArtificial IntelligenceSound
AI summaryⓘ
The authors studied how speech recognition models that add timestamps can sometimes misalign the timing during long pauses without speech, causing the time labels to drift from the actual audio. They found that simple fine-tuning fixes this but harms other parts of the system by forgetting what it learned before. To solve this, they created a two-step method called REDDIT that adjusts the timestamps without messing up the rest of the model. Their approach uses automated speech region detection and no extra human labels, resulting in much better timing accuracy while keeping overall model performance stable.
autoregressive ASRtimestamp driftnon-speech spansfine-tuningcatastrophic forgettingREDDIT methodvoice activity detection (VAD)long-gap benchmarksmodel replayWhisper-tiny
Authors
Cheng-Kang Chou, Ming-To Chuang, Ke-Han Lu, Chan-Jan Hsu, Hung-yi Lee
Abstract
Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio. We study this non-speech-induced timestamp drift with self-built gap and long-gap benchmarks across 15 evaluated timestamp-producing ASR and audio-language systems. Naive timestamp-corrected fine-tuning improves alignment but can severely degrade non-target ASR behavior, exposing a forgetting problem. We propose REDDIT(REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding this catastrophic forgetting: it first edits timestamp targets under the model's own replayed decoder context while matching the frozen base distribution on non-timestamp tokens, then applies a short edited-prefix refinement stage. In this framework, we construct correction supervision without human transcripts or human timestamp annotations by combining VAD-trimmed speech spans with inserted non-speech gaps and known concatenation offsets. On Whisper-tiny, 34.9 hours of targeted correction audio used and only 1.6% of model parameters updated, raising long-gap mIoU from 38.7% to 95.0% and reducing mixed-gap out-of-domain AAS from 2752 ms to 223 ms while preserving CV-en MER at 41.3% (versus 524.2% for ordinary SFT decoder tuning).