Real-Time Avatars, Multilingual Speech LLMs, and Singlish TTS
Today's digest covers real-time multi-modal avatar generation, an open-source multilingual projector bridging Whisper and LLMs across 28 European languages, and fine-tuning zero-shot TTS to authentically capture Singapore English.
We propose OmniMate, an open-ended real-time streaming audio-visual interactive avatar generation framework, which jointly generates speech, motions, and sound effects while preserving visual and speaker identity across multi-turn conversations. From Xi'an Jiaotong University.
Today's papers span three fronts in conversational AI: interactive avatars that generate speech, motion, and sound in lockstep; a scalable open-source bridge between speech encoders and LLMs for multilingual understanding; and a focused study on preserving regional accent identity in zero-shot TTS. Together they highlight a shared theme — the growing pressure to make speech and audio systems work authentically, in real time, and across the full diversity of the world's languages and accents.
Talking Avatars & Audio-Visual Generation
Keeping identity and cross-modal consistency alive across open-ended multi-turn conversations.
Generating interactive avatars in real time is hard enough for short, fixed-length sequences — but production deployments demand open-ended streaming where conversation length is unknown and identity drift is a real danger. OmniMate from Xi'an Jiaotong University tackles exactly this regime, jointly synthesizing speech, body motion, and ambient sound effects in a single streaming framework while actively preventing cross-modal inconsistency from compounding over extended dialogues.
OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
OmniMate enables real-time interactive avatar generation that jointly synthesizes speech, motion, and sound effects while preserving identity. It addresses open-ended streaming with unknown length and maintains cross-modal consistency, preventing identity drift during extended multi-turn conversations.
Abstract
Recent advances in diffusion-based generative models have enabled real-time audio-driven avatar generation and unified audio-visual synthesis, providing a promising foundation for interactive avatar systems. However, extending unified audio-visual synthesis to real-time interactive streaming remains challenging, as the generation horizon is unknown in advance and the generated identity may drift over long-term generation. To address these challenges, we propose OmniMate, a unified framework for open-ended real-time interactive audio-visual avatar generation. OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions. To achieve adaptive response progression, we introduce a Generation Progress Controller (GPC) that explicitly models the generation progress of each streaming chunk, allowing the model to complete responses according to the desired progress and achieve seamless transitions between execution and listening states. To preserve long-term cross-modal identity consistency, we propose a Multi-Reference Conditioning Module (MRCM), which leverages multiple reference images and a reference speech segment to provide persistent visual and speaker identity cues throughout long-duration streaming interactions. Extensive experiments on an interaction-oriented adaptation of VerseBench demonstrate that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency. The results further show that OmniMate supports realistic, coherent, and responsive interactive avatar experiences over extended multi-turn conversations.
Key challenges OmniMate addresses include maintaining identity coherence (voice, appearance, and gesture style staying consistent with the target person) and cross-modal synchrony (lip motion matching phonemes, gestures aligning with prosodic rhythm, sound effects fitting the scene) — all without knowing how many turns remain. This is a meaningful step toward avatars that can anchor multi-turn, real-world deployments rather than canned demos.
SpeechLLMs & Multilingual Speech Understanding
Connecting Whisper to LLMs across Europe's linguistic landscape.
The Whisper-to-LLM pipeline is a natural architecture for speech understanding, but most work has been English-centric or closed-source. MEUSLI (Multilingual European Understanding of Speech with LLM Integration) from Fondazione Bruno Kessler is the first open-source multilingual projector that bridges Whisper encoders with LLMs, covering 28 European languages for ASR out of the box.
MEUSLI: a Multilingual Projector for LLM-based ASR and Beyond
MEUSLI is the first open-source multilingual projector connecting Whisper encoders with LLMs, enabling ASR across 28 European languages and generalizing to speech translation and topic identification with minimal task-specific data—providing a scalable foundation for inclusive multilingual speech understanding.
Abstract
Lightweight projectors are an established way to connect pre-trained speech encoders with large language models (LLMs), mapping acoustic features into token-level embeddings for tasks like ASR and spoken question answering. Existing systems, however, typically only support a few languages and are often limited to English. We introduce MEUSLI, the first open-science multilingual projector family that links a Whisper encoder with open-source multilingual LLMs, enabling fully open-source end-to-end ASR in 28 European languages. MEUSLI extends prior monolingual pipelines, delivering strong results across high- and low-resource languages. Using proper continual leaning techniques, MEUSLI can be easily extended to other languages not seen in training. We further demonstrate that the MEUSLI projector can be leveraged beyond ASR, enabling multilingual speech translation and topic identification with only a few hours of task specific supervision per language. Overall, MEUSLI provides a solid foundation for multilingual speech understanding tasks, supporting scalable and inclu- sive open-source SpeechLLM
Beyond raw transcription, MEUSLI generalizes to speech translation and topic identification with minimal task-specific data — a critical property for low-resource European languages that rarely benefit from dedicated task models. Releasing this projector openly is a meaningful infrastructure contribution: researchers targeting under-resourced European locales now have a principled, scalable starting point rather than building bespoke pipelines from scratch.
TTS & Voice Synthesis
Authentic regional accents in zero-shot synthesis — Singlish as a case study.
Generic TTS systems trained on standard varieties of English reliably flatten regional accents, producing output that sounds fluent but inauthentic to local speakers. The paper "Singlish, Can or Not?" from the National University of Singapore takes Singapore English — a creole with distinctive phonology, prosody, and code-switching — as a demanding testbed for accent-preserving zero-shot TTS.
Singlish, Can or Not? Fine-Tuning and Evaluating Zero-Shot TTS for Singapore English
This paper fine-tunes zero-shot TTS models to preserve Singapore English (Singlish) accent. By adapting state-of-the-art systems on Singlish speech data, it closes the gap between generic and authentic regional speech synthesis while generalizing to unseen speakers.
Abstract
Zero-shot text-to-speech (ZS-TTS) achieves near-human quality for standard English, but it copies regional accents poorly. Prompted with a short Singlish utterance, state-of-the-art systems reproduce a speaker's timbre while flattening the accent toward generic English. We investigate whether targeted fine-tuning off-the-shelf ZS-TTS can close the gap for Singapore English (Singlish). We fine-tune two cutting-edge ZS-TTS models, Chatterbox and CosyVoice 3, on 50 Singlish speakers from the IMDA National Speech Corpus. Three speech distributions are evaluated: real recordings against off-the-shelf and fine-tuned generation driven by the same Singlish audio prompts. The evaluation covers four dimensions: naturalness, intelligibility, speaker similarity, and accent similarity. We separate adaptation (in-domain speakers seen during fine-tuning) from consistency (held-out speakers) to test whether accent transfer generalises beyond the training data. Fine-tuning raises accent similarity on in-domain and out-of-domain speakers for both Chatterbox and CosyVoice 3. It moves the generated distribution measurably toward real Singlish, with the gain persisting on held-out speakers. To our knowledge, this is the first systematic study of Singlish-accented TTS.
By fine-tuning state-of-the-art zero-shot TTS systems on Singlish speech data, the authors demonstrate a meaningful reduction in the gap between generic and authentic regional synthesis, while retaining generalization to unseen speakers. The evaluation framework they develop is also noteworthy: rigorously measuring accent fidelity for a non-standard variety requires metrics and listener panels that go beyond standard intelligibility benchmarks — methodology that should transfer to other regional varieties seeking the same treatment.
Trending on Hugging Face
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.