Efficient ASR Training with Conversations that Never Happened
2026-06-02 • Computation and Language
Computation and LanguageArtificial IntelligenceSound
AI summaryⓘ
The authors address the challenge of improving speech recognition for languages and topics that don't have much real conversation data. They create fake conversations by using AI language models to write dialogues, then turn them into speech using text-to-speech technology, preserving speaker details. Testing on Hungarian conversations showed that adding these synthetic talks helps recognition systems work better, especially with more and well-chosen generated data. Their best model, trained mostly on these generated conversations, outperformed one trained on much larger real data, suggesting this method can effectively supplement limited real recordings.
Conversational ASRLow-resource languagesData augmentationText-to-speech (TTS)Large language models (LLMs)FastConformerSpeaker metadataSynthetic dataHungarian BEA-Dialogue benchmark
Authors
Máté Gedeon, Péter Mihajlik
Abstract
Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata, maps speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations. We evaluated five LLM families under single-generator, fixed-budget mixture, and scale-up settings using the same FastConformer-Large training recipe for each one. We ran comprehensive evaluations on the Hungarian BEA-Dialogue benchmark corpus, with the method itself being applicable to any language given the resources for each component. The results show that synthetic conversations consistently improve speech recognition performance, but generator choice and data composition strongly affect the gains. Our largest training configuration, using only 67 hours of real conversations and 636 hours of simulated data, achieves better performance on the evaluation benchmark than a zero-shot model trained on 2700 hours of Hungarian speech. These findings indicate that LLM-generated conversational data synthesized with TTS is a practical complement to real conversational corpora for speech model training.