Real-Time Speech Translation and Compact TTS for Low-Resource Languages
Today's digest covers a real-time multi-speaker speech-to-speech translation system with streaming ASR stabilization, and a staged depth-pruning distillation approach for building compact Hindi TTS models under low-data constraints.
System architecture of X-Translator. The browser streams audio and session controls to the server through WebSocket. Inside X-Translator, the ASR pipeline commits source segments for display and translation, while the speaker prompt manager tracks speaker identity and routes the selected prompt to TTS. MT and TTS are shown as separate stages: MT produces target text, and TTS uses the selected prompt to synthesize target audio for browser playback. From Shanghai Jiao Tong University.
Today's papers tackle two practical challenges at opposite ends of the speech pipeline: keeping multi-speaker translation coherent and real-time in the wild, and squeezing a capable TTS system into a compact footprint for a low-resource language. Both push toward deployment-ready speech systems rather than research-only benchmarks.
Speech-to-Speech & Voice Agents
Making multilingual, multi-speaker translation work in real conversations — not just clean single-speaker audio.
Real-time speech-to-speech translation breaks down quickly when multiple speakers are involved and ASR transcripts keep shifting mid-stream. X-Translator from Shanghai Jiao Tong University attacks this head-on with two core mechanisms: incremental commitment to stabilize streaming ASR output (preventing cascading retranslations from hypothesis churn), and dynamic voice prompts that track speaker identity and maintain voice consistency across turns. The system is fully open-source and designed for reproducible low-latency deployment, making it a practical reference implementation for multi-speaker S2ST pipelines.
X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System
X-Translator is a real-time speech-to-speech translation system that handles multi-speaker conversations by stabilizing streaming ASR through incremental commitment and maintaining speaker consistency with dynamic voice prompts. It enables open-source, reproducible deployment for practical low-latency translation.
Abstract
Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.
TTS & Voice Synthesis
Distilling capable speech synthesis into models that can run under real-world resource and data constraints.
Building a high-quality TTS system for a language like Hindi is doubly hard when labeled data is scarce — training from scratch often simply fails. This paper from an independent researcher proposes staged depth-pruning distillation, progressively removing transformer blocks from a large flow-matching TTS teacher to produce compact student models, guided by ASR-based gating to control which blocks are pruned at each stage. The result is real-time-capable models down to 190M parameters that retain synthesis quality the teacher achieved — a recipe well-suited to other low-resource language settings beyond Hindi.
Staged Depth-Pruning Distillation of a Flow-Matching Text-to-Speech Teacher: A Compact Hindi Speech Synthesizer
This work distills a large Hindi TTS teacher into compact student models via staged depth pruning, enabling synthesis under severe low-data constraints where training from scratch fails. Gradually removing transformer blocks with ASR-based gating yields real-time models down to 190M parameters.
Abstract
We present a practical recipe for building a compact Hindi text-to-speech (TTS) model by distilling a large flow-matching teacher (IndicF5, 337M-parameter DiT) under a severe data budget (~17.6 hours). Training a small model from scratch on this much data fails outright. Instead we warm-start the student from the teacher by pruning depth only: keeping the teacher's width, text dimension, attention heads, and mel/text I/O fixed so all non-block tensors copy one-to-one, and retaining an evenly-spaced subset of transformer blocks. We first measure how much depth the teacher tolerates (it remains near-functional at -27% blocks but collapses past -50%), then descend gradually (22 -> 16 -> 12 -> 8 -> 6 blocks), re-fine-tuning after each prune, with each step gated by an objective ASR word-error-rate (WER) check. The resulting students reach WER 0.00 on unseen sentences at 249M and 190M parameters, and remain robust down to 131M; at 102M we observe a clear capacity cliff that we attribute to the data budget rather than the recipe. We also document two train/inference feature- and library-parity failures (mel filterbank and rotary-embedding library versions) that silently degrade audio, and a version-independent fix. The method yields a high-quality Hindi voice that runs in real time on a 6 GB laptop GPU. An independent 50-sentence FLEURS benchmark compares the released 190M student against its teacher and MMS-TTS-hin.
Trending on Hugging Face
Moshi: a speech-text foundation model for real-time dialogue
A speech-text foundation model for real-time dialogue via end-to-end speech-to-speech generation instead of cascaded components. By jointly modeling overlapping audio streams with text-token prediction, it achieves 200ms latency while preserving emotion and handling natural conversational dynamics like interruptions.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.