Dialect Speech Dialogue and Fine-Grained Prosodic Control in TTS
Today's digest covers end-to-end spoken dialogue for low-resource Chinese dialects and a coarse-to-fine approach to expressive TTS that gives word- and phoneme-level control over pitch, loudness, and duration.
Coarse-to-fine speech control pipeline. From University of Southern California.
Today's digest brings two focused but impactful contributions: one tackling the under-explored challenge of end-to-end spoken dialogue in low-resource Chinese dialects, and another pushing the boundaries of expressive TTS with granular prosodic control. Together they highlight how speech research is advancing both in linguistic breadth and in the precision with which we can shape synthesized voice.
SpeechLLMs & Spoken Dialogue
Bringing natural speech dialogue to dialects that big-data pipelines have long ignored.
Low-resource dialect speech dialogue is a hard problem — standard end-to-end speech models suffer from semantic drift when adapted to dialects with limited data, causing output quality to degrade across both naturalness and coherence. Alibaba DAMO Academy addresses this head-on with DialectS2S, a system that introduces self-aligned supervision: a training signal that continuously matches prediction targets against the model's own evolving representations rather than fixed reference embeddings. This dynamic alignment prevents the representation mismatch that typically emerges during dialect fine-tuning, enabling high-quality speech dialogue without requiring large dialect-specific corpora.
DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects
DialectS2S enables end-to-end speech dialogue for low-resource Chinese dialects using self-aligned supervision that matches targets with evolving model representations. This addresses semantic drift during dialect adaptation, improving speech naturalness and dialogue quality without large datasets.
Abstract
Current end-to-end speech dialogue models are primarily optimized for mainstream languages and remain limited in low-resource dialect scenarios due to the scarcity of dialect speech data. Moreover, during dialect adaptation, the semantic representation space of speech dialogue models continuously evolves, while conventional speech supervision remains unchanged, leading to semantic inconsistency between hidden representations and speech targets and degrading speech stability and naturalness. To address these issues, we propose DialectS2S, an end-to-end speech dialogue model for Chinese dialects. We first develop a scalable dialect speech dialogue synthesis pipeline for efficient data construction. We further introduce a two-stage post-training strategy with self-aligned speech supervision, which aligns the semantic content of speech supervision with the evolved semantic representations of the model to improve dialect speech generation quality. Experimental results show that DialectS2S consistently outperforms existing baselines across multiple Chinese dialects in speech dialogue, achieving substantial improvements in dialect consistency, response quality, and speech intelligibility. Our work provides an efficient and scalable solution for end-to-end speech dialogue modeling in low-resource dialect scenarios. To facilitate future research and practical applications, we fully open-source the DialectS2S framework, including model checkpoints, training datasets, and fine-tuning code.
TTS & Voice Synthesis
From utterance-level style to phoneme-level sculpting — expressive TTS goes granular.
Conventional TTS systems offer coarse-grained style control at best — typically at the utterance or speaker level. CtrlSpeech from the University of Southern California changes this by introducing a coarse-to-fine control hierarchy that lets users adjust pitch, loudness, and duration independently at both the word and phoneme level, all while preserving speaker identity. This level of surgical prosodic control opens the door to applications — audiobook production, voice acting, accessibility tools — where nuanced, targeted expressiveness is essential and global style transfer simply isn't enough.
CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis
CtrlSpeech enables fine-grained prosodic control in TTS by allowing word and phoneme-level adjustment of pitch, loudness, and duration while preserving speaker identity. This coarse-to-fine approach offers unprecedented flexibility for expressive speech refinement compared to conventional TTS systems.
Abstract
Recent Text-To-Speech (TTS) systems have achieved strong naturalness and zero-shot voice cloning performance, but fine-grained control of expressive speech at the word or phoneme level remains challenging. We propose CtrlSpeech, a controllable, expressive TTS framework with coarse-to-fine control. Built on the DiTAR architecture, CtrlSpeech combines global speaker conditioning with phone-aligned pitch, loudness, and duration signals, enabling localized prosodic control while preserving the target speaker's timbre. This design allows users to adjust expressive attributes at a fine temporal granularity, making speech refinement more flexible and controllable. Experimental results show that CtrlSpeech achieves competitive zero-shot TTS performance and improves controllability over expressive attributes, demonstrating its effectiveness for flexible and practical expressive speech synthesis.
Trending on Hugging Face
UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos
UniSwap enables synchronized appearance and voice replacement in talking videos through a unified streaming audio-visual diffusion transformer with specialized training and inference adaptations.
MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting
MuseTalk generates high-fidelity lip-synced face videos in real-time by encoding face images and audio features in a latent space with a Variational Autoencoder and U-Net.
LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time
LiveAnimate enables real-time, long-form pose-driven human animation via a 14B-parameter video diffusion transformer with specialized training, bounded attention caching, and sequence parallelism.
dots.tts Technical Report
dots.tts is a 2B-parameter continuous autoregressive text-to-speech model that generates speech in a semantically structured continuous latent space. Innovations include full-history conditioning and self-corrective post-training for robust, expressive, and low-latency multilingual speech.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.