One-Shot Avatars, Low-Latency TTS, and Chain-of-Thought for Spoken LLMs
Today's digest covers animatable 3D Gaussian head generation from a single image, a sparse-embedding approach to real-time mobile TTS, and efficient chain-of-modality reasoning that brings CoT gains to spoken language models.
Qualitative visualizations on the VFHQ dataset. From Nanyang Technological University.
Today's three papers tackle distinct but complementary challenges across the conversational AI stack: building photorealistic animatable avatars from a single photo, squeezing natural prosody out of non-autoregressive TTS at mobile latency, and unlocking chain-of-thought reasoning inside spoken language models without ballooning token budgets.
Talking Avatars & 3D Head Generation
From a single selfie to a fully animatable 4D Gaussian head — new attention mechanisms push quality at the hard edges.
Reconstructing a high-fidelity, animatable head from one image is notoriously difficult in occluded regions and at extreme viewpoints — precisely where downstream applications (video conferencing, virtual try-on, gaming) need it most. FA-LAM (Focus-Aware Large Avatar Model) from Nanyang Technological University attacks this with a dual-phase training pipeline that explicitly disentangles reconstruction from animation. Attention regularization keeps the model focused on the right facial regions, while token fusion ties appearance and motion representations together tightly, yielding cleaner geometry and texture in the hard-to-see areas.
FA-LAM: Focus-Aware Large Avatar Model for One-Shot 4D Animatable Gaussian Head
FA-LAM creates animatable Gaussian heads from single images using a dual-phase training pipeline that disentangles reconstruction and animation. Through attention regularization and token fusion, it achieves superior quality in occluded regions and extreme viewing angles.
Abstract
We propose FA-LAM, a Focus-Aware Large Avatar Model for one-shot animatable Gaussian head creation, while simultaneously enabling static 3D and dynamic 4D full-head recovery. The core of our method lies in a thorough analysis of the attention mechanisms and the entangled reconstruction and animation training pipeline adopted by prior state-of-the-art approaches. Our analysis identifies two main factors that compromise the quality of 3D full-head generation: (1) incorrect and noisy attention activations, and (2) conflicts between the tasks of reconstruction and animation. To address the first issue, we introduce a symmetric and semantic attention regularization strategy that leverages the inherent semantics and structural symmetry of human heads. To disentangle the objectives of reconstruction and animation, we develop a novel dual-phase training pipeline that separates the model's capabilities for large-view hallucination and animation into distinct modules. Moreover, we enhance our model to support multi-view and streaming 4D reconstruction in an efficient and memory-friendly manner through a core autoregressive modification with tailored visibility-aware token fusion. Collectively, these innovations enable FA-LAM to reconstruct animatable Gaussian full heads with superior quality, particularly in fine facial regions and large viewing angles.
TTS & Voice Synthesis
Sparse temporal embeddings let a non-autoregressive synthesizer have its latency cake and eat natural prosody too.
Non-autoregressive TTS models are fast but notoriously rigid — they trade expressive prosody for speed, making them a poor fit for mobile deployment where both are required. StellarTTS from Shanghai Jiao Tong University introduces sparse temporal embeddings within a NAR framework to give the model fine-grained, phoneme-level control over duration and prosody without the sequential bottleneck of autoregressive decoding. The result is real-time inference on mobile hardware while preserving the naturalness that rigid NAR systems typically sacrifice.
StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis
StellarTTS addresses the latency-robustness-prosody trade-off using sparse temporal embeddings in a non-autoregressive framework, enabling fine-grained control over phoneme duration and prosody while achieving real-time mobile inference—maintaining natural prosody where rigid NAR systems typically sacrifice quality.
Abstract
The trade-off between robustness, latency, and prosody critically challenges text-to-speech (TTS) systems. Autoregressive models, despite fidelity, are slow and error-prone; non-autoregressive (NAR) alternatives, while fast, often sacrifice prosodic naturalness via rigid alignments. This paper introduces StellarTTS, a novel mobile-optimized NAR TTS framework based on a sparse temporal embedding strategy, enabling granular control of phoneme duration, pronunciation, and prosody. Furthermore, we propose a semantic-aware codec that facilitates efficient single-stage decoding. Conditioned on the sparse temporal embedding, our 83M-parameter lightweight masked generative transformer achieves a real-time factor (RTF) of 0.08. Experiments demonstrate that StellarTTS attains lower latency and stronger robustness compared to state-of-the-art TTS systems, while maintaining competitive performance in audio quality, prosodic naturalness, and speaker similarity.
SpeechLLMs & Spoken Dialogue
Progressive token compression brings chain-of-thought reasoning to spoken LLMs at 40% lower text-token cost.
Spoken language models have lagged text-only LLMs on reasoning tasks partly because grafting chain-of-thought onto a speech-centric architecture is non-trivial — verbose reasoning chains are expensive and don't naturally interface with the acoustic modality. Alibaba DAMO Academy's ECoM Reasoning (Efficient Chain-of-Modality) solves this via progressive compression of text representations that jointly guide both speech decoding and reasoning steps. The compressed representations act as a shared scaffold across modalities, achieving a 21% accuracy gain over non-reasoning baselines while using 40% fewer text tokens than standard CoT approaches.
Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models
This paper introduces ECoM Reasoning, which enables spoken language models to perform chain-of-thought reasoning via compressed text representations that jointly guide speech and reasoning. It achieves 21% accuracy gains over non-reasoning baselines while using 40% fewer text tokens than standard approaches.
Abstract
Spoken language models (SLMs) enable natural human-computer interaction, but their reasoning ability still lags behind that of text-based large language models, especially on spoken mathematical question answering tasks. One important reason is that SLMs reason over purely verbalized mathematical expressions, which are harder to interpret than symbolic text. However, directly transferring text-based reasoning to SLMs is nontrivial due to architectural constraints and the additional computational requirements. To address this challenge, we propose Efficient Chain-of-Modality Reasoning (ECoM Reasoning), the first framework to introduce compressed reasoning into SLMs. By compressing the textual component so that it jointly serves as speech guidance and reasoning representation, ECoM Reasoning improves reasoning accuracy while using a smaller token budget than the standard Chain-of-Modality (CoM) architecture, which generates intermediate text before speech. To train this capability, we further propose Progressive Compression, a curriculum-based strategy that gradually trains the model from full-form reasoning to compressed reasoning. Experiments on spoken mathematical question answering benchmarks show that ECoM Reasoning improves accuracy by 21% over standard CoM without explicit reasoning, and by 3% over CoM with full reasoning traces while using only 40% of the text tokens, demonstrating that it enhances SLM reasoning while remaining inference-efficient.
Trending on Hugging Face
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.