SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation
2026-03-23 • Sound
Sound
AI summaryⓘ
The authors created SelfTTS, a computer program that turns text into speech and can copy the style and emotion of different speakers without needing extra tools to understand speaker or emotion traits. They designed a way to separate who is speaking from what emotion is being shown using special math and learning methods. They also made the program learn better by practicing with its own voice-changing skills. Tests showed that SelfTTS makes more natural and emotionally expressive speech than other similar systems.
text-to-speechstyle transferspeaker embeddingemotion embeddingGradient Reversal Layercosine similarity losscontrastive learningself-augmentationvoice conversionemotional naturalness
Authors
Lucas H. Ueda, João G. T. Lima, Pedro R. Corrêa, Flávio O. Simões, Mário U. Neto, Paula D. P. Costa
Abstract
This paper presents SelfTTS, a text-to-speech (TTS) model designed for cross-speaker style transfer that eliminates the need for external pre-trained speaker or emotion encoders. The architecture achieves emotional expressivity in neutral speakers through an explicit disentanglement strategy utilizing Gradient Reversal Layers (GRL) combined with cosine similarity loss to decouple speaker and emotion information. We introduce Multi Positive Contrastive Learning (MPCL) to induce clustered representations of speaker and emotion embeddings based on their respective labels. Furthermore, SelfTTS employs a self-refinement strategy via Self-Augmentation, exploiting the model's voice conversion capabilities to enhance the naturalness of synthesized speech. Experimental results demonstrate that SelfTTS achieves superior emotional naturalness (eMOS) and robust stability in target timbre and emotion compared to state-of-the-art baselines.