Streaming Avatars, Unified TTS, and the State of Multimodal Dialogue
Today's digest covers a fix for dynamic collapse in real-time avatar synthesis, a unified speech generation and editing system stabilized by semantic supervision, and a sweeping survey of multi-turn conversational AI from text to multimodal interaction.
Figure from From University of Science and Technology of China.
Today's papers span the full conversational AI stack — from the pixels of a streaming talking head, to the acoustic tokens of a voice synthesizer, to the architectural blueprints of multi-turn dialogue systems. Each tackles a core brittleness: motion collapse in avatar distillation, error accumulation in continuous-representation TTS, and the persistent memory gap in multimodal conversation.
Talking Avatars & Lip Sync
Real-time avatar generation meets a subtle but crippling failure mode — and a new training strategy to defeat it.
Self-forcing distillation has become a popular recipe for fast, streaming talking-avatar generation, but it carries a hidden tax: the model converges toward static, nearly motionless outputs because low-motion frames incur the smallest prediction error. DynaForcing from the University of Science and Technology of China directly targets this dynamic collapse phenomenon. The solution combines three complementary mechanisms — hybrid forcing (mixing teacher and self-generated frames during training), dynamics rewards (explicit motion-magnitude signals in the loss), and reference perturbation (augmenting the conditioning signal to prevent the model from anchoring on a single static frame). Together, they recover natural temporal dynamics without sacrificing the visual quality that distillation was meant to preserve, eliminating the quality-dynamics trade-off that has plagued real-time avatar synthesis.
DynaForcing: Overcoming Dynamic Collapse in Self-Forcing Distillation for Streaming Avatar Generation
This paper solves dynamic collapse in streaming avatar generation—where self-distillation produces static, low-motion outputs. DynaForcing recovers natural temporal motion through hybrid forcing, dynamics rewards, and reference perturbation, eliminating the quality-dynamics trade-off in real-time avatar synthesis.
Abstract
Audio-driven avatar generation requires realistic lip-sync, expressive motion, and real-time streaming. Recent work achieves the latter via self-forcing with Distribution Matching Distillation (DMD), but this paradigm suffers from a critical failure that has not been systematically characterized: dynamic collapse, where the student model converges to a near-static optimum with high perceptual quality but severely suppressed temporal dynamics. We trace this to two causes: the reverse KL objective in DMD, which biases toward low-motion modes, and unanchored self-conditioning, which creates a feedback loop that amplifies collapse. This is especially harmful for avatars, where even subtle motion loss breaks lip-sync and expression. To address this, we propose DynaForcing, a training framework with three complementary strategies applied at different levels. Specifically, Hybrid Forcing anchors rollouts to ground-truth dynamics at the data level to break the feedback loop. Dynamics-Aware Reward Regularization introduces explicit motion rewards via the RL interpretation of DMD to counteract the reverse KL bias at the loss level. Reference Perturbation perturbs reference images to decouple identity from static details, forcing the model to rely on audio for motion at the conditioning level. We further introduce computation graph pruning and gradient replay, reducing the GPU footprint of self-forcing by over an order of magnitude. Experiments show that DynaForcing recovers dynamics to teacher-comparable levels (Dyn-Deg: 0.31 -> 0.73, Sync-C: 7.03 -> 7.68) while improving visual quality, resolving the quality-dynamics trade-off throughout training without early stopping.
TTS & Voice Synthesis
Semantic supervision turns out to be the missing stabilizer for continuous-representation speech models.
Autoregressive models over continuous speech representations are powerful but notoriously unstable — small prediction errors compound across decoding steps, leading to degraded or collapsed outputs. FireRedTTS3 from Zhejiang University addresses this with semantically enriched speech representations: a frozen audio encoder provides semantic supervision that anchors the autoregressive process and curbs error accumulation. The payoff is a single unified system that handles voice cloning, instruction-controlled voice design, and speech editing — all without the complex multi-stage training pipelines that comparable systems require. The instruction-controlled design is particularly noteworthy, letting users steer voice characteristics through natural-language prompts rather than reference audio alone.
FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations
FireRedTTS3 stabilizes continuous-representation speech generation and editing via semantic supervision from a frozen audio encoder, mitigating autoregressive error accumulation. It enables unified voice cloning, instruction-controlled voice design, and speech editing without complex multi-stage training.
Abstract
Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.
SpeechLLMs & Spoken Dialogue
A field-wide survey maps where multimodal conversational AI stands — and where it still falls short.
As spoken and multimodal dialogue systems proliferate, it becomes harder to see the forest for the trees. This comprehensive survey from the Qatar Computing Research Institute steps back to map the entire landscape of multi-turn conversational AI, tracing the evolution from text-only systems through speech-enabled and fully multimodal interaction. The authors catalogue datasets, model architectures, and evaluation protocols across all three modalities, but their most pointed finding is a structural gap: while multimodal perception (seeing, hearing, understanding) has advanced rapidly, systems remain weak at persistent memory and cross-turn grounding — the ability to maintain coherent context and reference resolution over extended dialogues. For anyone building or evaluating conversational systems, this survey is a useful map of the terrain and an honest accounting of where the hard problems remain.
Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges
A comprehensive survey of multi-turn conversational AI spanning text, speech, and multimodal systems. It identifies a critical gap: while multimodal perception has advanced rapidly, systems still struggle with persistent memory and cross-turn grounding—key challenges for sustained, natural dialogue.
Abstract
Conversational AI is moving beyond isolated text prompts toward sustained, multimodal interaction. In real conversations, users clarify goals, revise requests, interrupt responses, switch topics, and introduce new evidence while expecting systems to preserve context across turns. This makes multi-turn dialogue a distinct challenge requiring systems to maintain and update memory, ground responses across modalities, tools, and external knowledge, and adapt across languages and cultures. This study reviews multi-turn conversational AI across text-only dialogue, AudioLLMs and speech-native systems, multimodal and omni-modal systems, and tool-augmented agents. We organize the literature around datasets and benchmarks, modeling paradigms, training strategies, evaluation setups, and cross-cutting challenges. Our analysis shows that support for multiple modalities has advanced faster than the ability to sustain coherent interaction across a session. Despite stronger capabilities to perceive, speak, and act across modalities, current systems still struggle with persistent memory, cross-turn grounding, full-duplex interaction, robust evaluation, and cultural alignment. We conclude with a research agenda for systems that can remember, revise, ground, speak, listen, act, and adapt across turns, modalities, and cultures. (https://github.com/faiza-sfa/multiturn-conversational-ai-survey)
Trending on Hugging Face
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
4DAnyone reconstructs 4D humans from monocular video by generating multiview-consistent videos and lifting them into 4D Gaussian Splatting, using reference and target context designs to overcome scaling bottlenecks.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.