Fine-Grained Emotional Control in Instruction-Following TTS
Today's digest spotlights a new approach to multi-emotion modeling in TTS, enabling both sequential emotion trajectories and simultaneous emotion blending via reinforcement learning with Group Relative Policy Optimization.
Multi-emotion control through sequential emotion trajectories and concurrent emotion blending. From Institute of Computing Technology, Chinese Academy of Sciences.
Today's digest is a focused one — a single, technically rich paper from the Institute of Computing Technology at the Chinese Academy of Sciences that pushes the boundaries of expressive, instruction-following speech synthesis.
TTS & Voice Synthesis
Taking emotional TTS beyond single-label control toward dynamic, composable affect.
Expressive speech synthesis has long struggled with a fundamental limitation: most systems model emotion as a single, static label applied to an entire utterance. Real speech, however, is rarely that simple — emotions shift across a sentence and can even co-occur simultaneously. This paper tackles both challenges head-on by introducing a framework for multi-emotion modeling in instruction-following TTS, supporting two distinct paradigms: sequential emotion trajectories (emotions that evolve over time within an utterance) and simultaneous emotion blending (multiple emotions mixed at the same moment).
To train the system, the authors design a hybrid reward mechanism coupled with Group Relative Policy Optimization (GRPO), allowing the model to receive nuanced feedback on both temporal and concurrent multi-emotion outputs — something standard single-emotion reward signals cannot capture. The result is a system that meaningfully advances beyond single-emotion baselines in fine-grained emotional controllability.
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
This work enables fine-grained emotional control in instruction-following TTS through emotion trajectories and emotion blending. A hybrid reward mechanism with Group Relative Policy Optimization handles both sequential and simultaneous multi-emotion feedback, advancing beyond single-emotion baselines.
Abstract
Natural-language instructions enable flexible control of synthesized speech, yet emotional TTS systems primarily model a single utterance-level affect, leaving multi-emotion control underexplored. We study two complementary multi-emotion TTS tasks: emotion trajectory, which spans several ordered affective stages, and emotion blending, in which multiple emotions coexist throughout an utterance. These tasks expose a supervision mismatch: supervised fine-tuning (SFT) does not explicitly evaluate emotion features, while single-emotion rewards provide neither structure-aware feedback for trajectory completion nor pair-aware feedback for blending. We introduce HybridEmo, a post-training framework that initializes both tasks with SFT and then aligns the speech-token policy through Group Relative Policy Optimization using a sample-aware hybrid reward. For trajectory samples, segment-aligned consistency combines average and weakest-stage evidence to preserve the correctness and completeness of prescribed stages. For blending samples, a GMM-based reward combines frame-level support from the union of target-emotion anchors in an offline emotion space with an utterance-level weaker-target margin. Both branches share an ASR reward and are routed within a unified policy. On MultiEmo-Test, HybridEmo significantly improves trajectory correctness and blending intensity, without a noticeable degradation in speaker similarity. Human evaluation prefers HybridEmo to CosyVoice 3 and EmoVoice-0.5B, with nearly balanced preferences against Qwen3-TTS.
The use of GRPO here is particularly notable: by sampling a group of candidate outputs and computing relative rewards within that group, the optimizer sidesteps the need for a separate critic network while still providing a strong training signal for the subtle differences between, say, a joy→nostalgia trajectory and a blended joy+nostalgia utterance. This is a compelling application of RL-from-feedback techniques to the structured, multi-dimensional problem space of expressive TTS.
Trending on Hugging Face
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
4DAnyone reconstructs 4D humans from monocular video by generating multiview-consistent videos and lifting them into 4D Gaussian Splatting, using reference and target context designs to overcome scaling bottlenecks.
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
VoiceMem introduces a dual-brain streaming memory architecture for speech language models that improves retrieval accuracy, emotional personalization, and real-time efficiency.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.