SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation
2026-08-07 • Sound
Sound
AI summaryⓘ
The authors address a problem in speech generation where models that predict continuous sounds struggle to clearly understand language content. They introduce SemBridge, a method that uses helpful 'semantic tokens' during training to guide the model's understanding of speech meaning, but only uses continuous data when actually creating new speech. Testing shows that SemBridge improves how accurately the speech matches the intended words without losing speaker identity or quality. This suggests adding direct meaning guidance during training can make continuous speech generation better.
continuous-latent speech generationautoregressive language modelsemantic tokensacoustic VAEtext-to-speechsinging voice synthesiscontent fidelityzero-shotword error ratecharacter error rate
Authors
Hanke Xie, Haopeng Lin, Jiale Qian, Dake Guo, Yuepeng Jiang, Zhichao Wang, Wenxiao Cao, Jingbin Hu, Guobin Ma, Wenhao Li, Huakang Chen, Chengyou Wang, Ming Tao, Zhonghua Fu, Lei Xie, Xinsheng Wang
Abstract
Continuous-latent autoregressive speech generation has emerged as a promising alternative to discrete-token modeling by avoiding quantization loss and preserving richer acoustic information. However, continuous acoustic targets do not ex- pose linguistic structure as explicit token-level prediction tar- gets. Consequently, the autoregressive language model (LM) must acquire linguistic structure indirectly through acous- tic prediction, which can compromise the content fidelity of generated speech. We propose SemBridge, a training-only semantic-token anchoring framework for continuous-latent autoregressive speech generation. SemBridge uses discrete se- mantic tokens to directly supervise autoregressive LM states and employs a Semantic-Aligned Acoustic VAE to organize the continuous target space under the same semantic refer- ence. The semantic supervision is used only during train- ing, while inference remains entirely continuous. We evalu- ate SemBridge on zero-shot text-to-speech (TTS) and score- conditioned singing voice synthesis (SVS). Across multi- ple benchmarks, SemBridge improves content accuracy, as measured by word and character error rates (WER/CER), while maintaining competitive speaker similarity and percep- tual quality. Experimental results demonstrate that explicit semantic-token supervision for autoregressive state learning is an effective and general direction for continuous speech generation. Speech samples are available.1 The model code and checkpoints will be available at https://github.com/ASLP- lab/SemBridge