Streaming S2ST and a Speech LLM for Basque
Today's digest covers two advances in speech-to-speech systems: a semantically supervised streaming translation model from Alibaba DAMO Academy, and the first open Speech Language Model for the low-resource Basque language.
Quality--latency tradeoff on RealSI, averaged equally across directions. Labels denote our fixed thresholds, SimulS2ST-Omni latency multipliers m1--m5, and Hy-SimulMT subsequent chunk durations. The star denotes our playback-adaptive policy. From Alibaba DAMO Academy.
Today's compact but punchy digest brings two papers squarely in the speech-to-speech space — one pushing the latency frontier for real-time translation, and another proving that state-of-the-art spoken interaction isn't reserved for high-resource languages.
Speech-to-Speech & Speech Language Models
From streaming translation with tunable latency to end-to-end spoken Basque — researchers are closing the gap between speech understanding and real-time spoken response.
Streaming Speech-to-Speech Translation
Real-time speech translation demands a system that can commit to output before a full utterance has been heard. Most approaches either wait too long (hurting latency) or commit too early (hurting quality). Alibaba DAMO Academy's S3-S2ST reframes this as a learned decision problem: given a fixed input grid, a commit policy is trained end-to-end to determine exactly when enough partial input has arrived to translate accurately. Crucially, the quality-latency tradeoff is controlled at inference time via a single adjustable threshold — no retraining required — and the system delivers competitive results across multiple language pairs.
S3-S2ST: Semantically Supervised Streaming Speech-to-Speech Translation
S3-S2ST learns to decide when partial input is sufficient for accurate streaming speech-to-speech translation through semantically supervised training, converting utterance-level speech pairs into incremental examples with a learned commit policy. The approach uniquely controls the quality-latency tradeoff via an adjustable threshold on a fixed input grid, delivering competitive results across multiple language pairs.
Abstract
Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation. We present S3-S2ST, a model that learns to translate and speak from semantically aligned streaming supervision. By converting utterance-level speech pairs into incremental training examples, we teach a learned policy when to commit translations. Its adjustable threshold controls the quality--latency tradeoff on a fixed 640\,ms input grid. S3-S2ST delivers competitive English--Chinese translation on FLEURS and leads the evaluated streaming systems in translation quality on the internal 9-pair test set. On RealSI, adapting the threshold to buffered speech further improves quality over a low fixed threshold with little change in first-audio latency. We plan to release the model and code.
A Speech Language Model for Basque
Building spoken dialogue systems for minority languages is notoriously hard: scarce data, limited pre-trained backbones, and no clear recipe for multimodal integration. The University of the Basque Country (UPV/EHU) tackles this head-on by presenting the first open Speech Language Model for Basque. The system pairs a Basque LLM backbone with synthesized training data and a modular architecture to enable low-latency, end-to-end spoken interaction — demonstrating that the playbook used for resource-rich languages can, with careful data augmentation, transfer to low-resource settings.
Developing a Speech Language Model for Basque
This work presents the first open Speech Language Model for Basque, enabling low-latency end-to-end spoken interaction in a minority language. By combining a Basque LLM backbone with synthesized training data, the system demonstrates that advanced multimodal spoken interaction can be developed in low-resource languages using modular architectures and data augmentation.
Abstract
Although the recent development of LLMs has paved the way for multi-modal interaction, minority languages face difficulties keeping pace. This work presents the first open Speech Language Model for Basque, designed for low-latency, end-to-end spoken interaction. Based on the LLaMA-Omni architecture, the system integrates a Latxa model as its backbone, transferring its Basque capabilities to the spoken modality by fine-tuning several modules. To address data scarcity, the English VoiceAssistant-400K corpus was translated into Basque, and the audio was synthesized using a Basque TTS. Evaluation results confirm the system's viability, achieving 3.86/5 in content quality, a 9.94% WER for spoken intelligibility, and a 3.07 UTMOS for audio quality. This work demonstrates that advanced spoken interaction systems can be developed using modular architectures and synthetic data even in low-resource languages.
Trending on Hugging Face
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
VoiceMem introduces a dual-brain streaming memory architecture for speech language models that improves retrieval accuracy, emotional personalization, and real-time efficiency.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.