Authentic Emotion in Conversational Speech Synthesis

TTS & Voice Synthesis

Beyond discrete labels — learning emotion the way humans feel it

Categorical emotion labels (happy, sad, angry…) have long been the dominant paradigm for emotional TTS, but they're a crude approximation of how affect actually works in conversation. The real challenge is capturing emotion as a continuous, context-dependent signal that shifts across a dialogue — and then rendering it naturally in synthesized speech.

AuEmoChat tackles exactly this problem. Rather than relying on a fixed emotion taxonomy, it learns continuous emotion tokens directly from large-scale emotional speech data, letting the model capture fine-grained affective nuance that discrete categories simply cannot express. On the dialogue-context side, it applies a smart merging strategy that compresses redundant conversational history while preserving the emotion-relevant information that actually drives prosody. The result is a system capable of generating expressive, contextually appropriate emotional speech across multi-turn conversations.

National University of Singapore · Jul 2026

AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis

AuEmoChat proposes conversational speech synthesis with authentic emotion understanding by learning continuous emotion tokens from large-scale emotional speech rather than discrete categories. It merges redundant dialogue context while preserving emotion-relevant information, generating expressive emotional speech.

Abstract

Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression and contextual consistency in user-agent interactions. Existing CSS methods struggle to render authentic human emotions due to limited predefined emotion label spaces (e.g., seven emotion categories), while redundant multimodal tokens in multi-turn dialogue history interfere with context understanding. To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering. First, we develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech via finite scalar quantization, enabling a more authentic emotion representation than limited basic emotion categories. We further propose AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant tokens in multimodal dialogue history while preserving emotion-relevant context. We integrate it into an autoregressive text-speech model to predict the target authentic emotion token and speech tokens. Finally, we propose Authentic Emotion Flow Matching, which renders speech by jointly conditioning on merged dialogue context, target authentic emotion, and acoustic priors. Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat outperforms state-of-the-art CSS baselines and generates more expressive and authentic emotional speech.

ttsemotiondialogueprosody

Key design choices worth noting:

• Continuous emotion representation trained from data rather than imposed from a label set — more robust to the messy, overlapping nature of real emotion
• Context compression with emotion preservation — reduces the noise of long dialogue histories without discarding the affective cues that matter for synthesis
• End-to-end conversational framing — the system is designed from the ground up for multi-turn dialogue, not retrofitted from single-utterance TTS

This is a meaningful shift in how the field thinks about grounding emotional TTS in authentic human affect, and the continuous-token approach in particular feels like a direction worth watching as conversational AI systems become more expressive.