Streaming Speech Synthesis: Real-Time Dialogue and Immersive Soundscapes
Today's digest covers two advances in speech synthesis: Meta AI's low-latency, interruption-aware TTS for interactive agents, and Zhejiang University's unified vocalized audio model that weaves speech into environmental soundscapes via autoregressive flow matching.
VoiceChat-TTS architecture overview. From Meta AI.
Today's papers push the boundaries of speech synthesis in two complementary directions: making real-time dialogue with AI agents feel genuinely natural, and blending speech seamlessly into rich acoustic environments. Both tackle the streaming inference problem head-on, abandoning the comfort of offline, decoupled pipelines in favor of unified, causal models built for the real world.
TTS & Voice Synthesis
From low-latency interactive agents to vocalized soundscapes — synthesis is going end-to-end.
Real-time voice interaction demands more than fast generation; it requires graceful handling of the messy, overlapping dynamics of human conversation. Meta AI's VoiceChat-TTS directly addresses this by streaming speech from raw LLM text tokens with minimal latency and native mid-utterance interruption support — a capability that prior duplex TTS models either lacked or achieved only at the cost of audio quality. Rather than bolting interruption handling on top of a standard TTS stack, the model is designed from the ground up for the continuous, bidirectional flow of interactive dialogue.
VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
VoiceChat-TTS streams speech directly from LLM text tokens with low latency and native support for mid-utterance interruptions, enabling natural interactive dialogue without the speech-quality compromises of prior duplex models.
Abstract
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.
On a different front, Zhejiang University's VoxAudio tackles the problem of vocalized audio — speech that isn't cleanly recorded in a studio but naturally embedded within environmental soundscapes. Instead of the conventional two-stage approach (generate speech, then mix with background audio), VoxAudio unifies the process with causal autoregressive flow matching, enabling streaming inference over a single coherent audio sequence. The model is further refined with multi-reward preference optimization and temporal grounding supervision, giving it fine-grained control over when and how speech sits within the sonic scene.
VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching
VoxAudio generates vocalized audio—speech naturally embedded in environmental soundscapes—through causal autoregressive flow matching. It moves beyond decoupled TTS+mixing pipelines with streaming inference, multi-reward preference optimization, and temporal grounding supervision for fine-grained speech control.
Abstract
Vocalized audio synthesis, the task of generating audio in which intelligible speech is embedded within an environmental soundscape, underpins applications such as podcast production and video dubbing. Existing Text-to-Audio (T2A) systems either reduce quoted speech to unintelligible vocal murmur or delegate it to a separate TTS model with post-hoc mixing, which forfeits control over when speech occurs and how it interacts with the scene. We present VoxAudio, a causal autoregressive flow matching model that addresses this problem from three complementary aspects. At the architecture level, chunk-wise causal factorization with independent per-chunk noise levels lets audio be emitted through sliding-window streaming inference with KV caching at variable target durations; to enable inference at arbitrary chunk granularities, we further pretrain the model with randomized chunk boundaries. At the preference level, multi-reward Negative-aware FineTuning (NFT) jointly optimizes semantic fidelity, linguistic accuracy, aesthetic quality, and temporal grounding At the data level, to supply the missing supervision for vocal content, we build VoxCorpus, a large-scale corpus whose captions quote the verbatim transcript of embedded speech with time intervals, and VoxBench, an interval-annotated benchmark with a temporal-grounding metric. Experiments on four benchmarks spanning general audio, speech, and unified vocalized audio validate the effectiveness and efficiency of VoxAudio. Our code and demos are available at https://voxaudio.github.io.
Together, these papers signal a clear trend: the next generation of speech synthesis is causal, unified, and streaming — whether the challenge is a live conversation partner or a richly layered audio world.
Trending on Hugging Face
dots.tts Technical Report
dots.tts is a 2B-parameter continuous autoregressive text-to-speech model that generates speech in a semantically structured continuous latent space. Innovations include full-history conditioning and self-corrective post-training for robust, expressive, and low-latency multilingual speech.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.