AI summaryⓘ
The authors created PolySpeech-100, a large test set to better check how well speech-language models understand spoken language across 110 different dialects and languages, including many that are less commonly studied. They combined real human recordings with computer-made speech to cover many regional dialects and low-resource languages. Testing 22 top models, they found open-source models handle heavy dialects better than traditional two-step systems because these models keep important speech details that get lost in transcription. However, open-source models struggle a lot with less-common languages compared to commercial ones. Also, a method called Chain-of-Thought prompting often made understanding worse, suggesting current models might not be well adapted for this approach.
End-to-End Speech-Large Language ModelsSpeech-LLMsDialectLow-resource languagesAutomatic Speech Recognition (ASR)Chain-of-Thought promptingParalinguistic cuesProsodyZero-shot learningBenchmark
Authors
Sicheng Yang, Shulan Ruan, Shiwei Wu, Yu Liu, Lu Fan, Zhi Li, You He
Abstract
While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription. Existing benchmarks suffer from three critical limitations: a pronounced bias towards high-resource languages, a focus on low-level recognition (ASR) rather than semantic reasoning, and a neglect of regional dialects. To bridge this gap, we introduce PolySpeech-100, a massive-scale benchmark designed to assess `native-level' speech comprehension across 110 linguistic variants. We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech, allowing us to cover 19 distinct Chinese dialects and over 80 low-resource languages. Extensive evaluation of 22 state-of-the-art models (including Gemini-3, GPT-Audio, and Qwen2.5-Omni) yields pivotal insights. First, we demonstrate that open-source E2E models outperform Cascade (ASR+LLM) systems on heavy dialects, proving that direct audio processing preserves critical paralinguistic cues and prosodic features (e.g., intonation, stress) that are often lost in standard transcription. Second, we reveal a significant performance gap: while commercial models maintain robustness, open-source models suffer catastrophic degradation on low-resource languages. Finally, counter-intuitively, we observe that under standard zero-shot settings, Chain-of-Thought prompting frequently degrades speech understanding performance for most evaluated models, revealing a potential modality alignment gap in current architectures. PolySpeech-100 establishes a rigorous standard for the next generation of inclusive, omni-capable Speech-LLMs. The data, demo, and code are publicly available at https://github.com/YoungSeng/PolySpeech-100.