Unified Audio LLMs and Expressive TTS with Non-Verbal Vocalizations
Today's digest covers two advances in speech synthesis: FireRedAudio's decoupled continuous representations for unified audio understanding and generation in a 9B LLM, and a preference optimization framework for synthesizing non-verbal vocalizations like laughter and sighs in expressive TTS.
Four representative long-form audio-understanding capabilities: structured organization, long-form summarization, bidirectional retrieval between time and content, and global analysis over evidence distributed across a recording. Our quantitative temporal-grounding evaluation focuses on structured organization. From Tsinghua University.
Today's pair of papers both push the frontier of speech and audio synthesis — one at the architecture level, asking how a single large language model can handle both understanding and generation without representation compromise, and the other at the expressiveness level, tackling the often-overlooked paralinguistic layer of human speech.
TTS & Voice Synthesis
From unified models to expressive vocalizations — rethinking what speech synthesis should produce
A persistent tension in audio language models is that the representations most useful for understanding audio (semantic, compressed) are different from those needed to generate high-fidelity audio (fine-grained acoustic detail). FireRedAudio from Tsinghua University attacks this head-on with a decoupled dual-pathway design inside a 9B-parameter LLM. Separate input encoders handle the understanding and generation pathways independently, enabling long-form audio analysis alongside high-quality speech synthesis with fine-grained editing — all without the representation mismatch that has constrained prior unified audio models.
FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation
FireRedAudio unifies audio understanding and generation through decoupled continuous representations in a 9B LLM. Separate input pathways enable long-form audio analysis and high-quality speech synthesis with fine-grained editing, solving the representation mismatch that limits prior unified audio models.
Abstract
A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
On the expressiveness side, most TTS systems still treat non-verbal vocalizations — laughter, sighs, gasps — as an afterthought, if they handle them at all. Researchers at Nanyang Technological University tackle this gap with a preference optimization framework specifically designed for non-verbal vocalization (NV) synthesis. The key methodological contribution is an NV-aware character error rate (CER) metric that treats vocalization tags as distinct symbols rather than noise, enabling direct preference optimization of NV quality without any modification to the underlying algorithm. This clean metric design makes it straightforward to plug into existing preference optimization pipelines.
Preference Optimization for Non-Verbal Vocalization Synthesis
This work studies preference optimization for synthesizing non-verbal vocalizations like laughter and sighs in expressive TTS. A key innovation is an NV-aware character error rate metric that treats vocalization tags as distinct symbols, enabling optimization without modifying the underlying algorithm.
Abstract
Non-verbal vocalizations (NVs), such as laughter, coughs, and sighs, are essential for expressive TTS, but the effectiveness of preference optimization for NV generation remains poorly understood. We systematically study preference optimization for NV-capable TTS, focusing on preference signals, preference-pair construction, and DPO-based optimization objectives. We formulate an NV-aware character error rate (NV-CER) by treating NV tags as distinct output symbols and computing a weighted pinyin-based CER over both verbal and non-verbal content, enabling controllable optimization of NV realization without modifying the underlying optimization algorithm. Experiments on Emilia-NV and the augmented NV-Bench covering 18 NV types reveal how different design choices affect NV realization and lexical fidelity, and establish an effective setup using standard DPO. Objective, LLM-based, and human evaluations provide converging evidence for our findings, offering practical insights into NV-aware post-training for expressive TTS.
Trending on Hugging Face
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
4DAnyone reconstructs 4D humans from monocular video by generating multiview-consistent videos and lifting them into 4D Gaussian Splatting, using reference and target context designs to overcome scaling bottlenecks.
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
VoiceMem introduces a dual-brain streaming memory architecture for speech language models that improves retrieval accuracy, emotional personalization, and real-time efficiency.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.