Emotional Intelligence for Spoken LLMs and Faster Zero-Shot TTS
Today's digest covers two papers pushing the frontier of spoken AI: a theory-grounded framework for benchmarking and aligning emotional intelligence in spoken language models, and an efficient autoregressive TTS system that cuts voice-cloning latency by 23%.
Illustrative examples of EmoSBench evaluation dimensions and EmoS model thought process. From Harbin Institute of Technology.
Today's two papers tackle complementary challenges in spoken AI — one asks how emotionally intelligent our spoken language models really are, and the other asks how fast and faithful a zero-shot voice synthesis system can get. Both push the state of the art in their respective lanes.
SpeechLLMs & Spoken Dialogue
Benchmarking and aligning the affective side of voice-based AI.
Measuring emotional intelligence in spoken language models has lacked a principled foundation — most existing evals borrow loosely from NLP sentiment benchmarks without grounding in cognitive theory. Researchers at Harbin Institute of Technology close that gap with EmoS, which builds on a four-branch cognitive model of emotional intelligence to construct both a comprehensive benchmark and a dedicated alignment training recipe. The framework pairs the benchmark with dual reward mechanisms — one targeting emotional understanding, the other dialogue coherence — and claims to reach human-level performance on emotional dialogue understanding tasks.
EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
EmoS introduces the first theory-grounded framework for evaluating emotional intelligence in spoken language models using a four-branch cognitive model. It combines a comprehensive benchmark with specialized alignment training and dual reward mechanisms to achieve human-level emotional dialogue understanding.
Abstract
Despite significant advances in instruction-following and auditory comprehension, the evaluation of Emotional Intelligence (EI) in Spoken Language Models (SLMs) remains confined to rudimentary paralinguistic perception, lacking a systematic, theory-driven cognitive framework. We introduce EmoSBench, the first comprehensive EI evaluation benchmark for SLMs constructed upon the four-branch theoretical model, covering Perceiving, Understanding, Using, and Managing Emotion across ten sub-tasks. Preliminary assessments on EmoSBench reveal a substantial gap: even leading proprietary models like GPT-4o-Audio achieve only 52.6%, significantly trailing human baselines. To bridge this gap, we develop EmoS, a specialized evaluator model optimized via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). To facilitate its effective training, we curate EmoDialogue, a bilingual dataset providing necessary fine-grained supervision through response pairs with rigorously defined EI gradations. Concurrently, we introduce a reward mechanism integrating a Steep Exponential Accuracy Reward (SEAR) and a Rationale Fidelity Reward (RFR) to enforce precise ordinal scoring and valid reasoning. Experiments demonstrate that EmoS reaches 83.8% accuracy, approaching human-level performance. Furthermore, evaluations on authentic, unconstrained spoken interactions validate its robust real-world generalization, establishing a foundational framework for advancing emotionally intelligent dialogue systems.
TTS & Voice Synthesis
Squeezing latency out of autoregressive continuous-latent synthesis.
Zero-shot voice cloning at inference speed remains a tough engineering problem, especially for autoregressive systems that must decode many steps. Fudan University's CuteTTS addresses this with a combination of continuous latents, patch-level autoregression (processing multiple frames per step), and a guidance-step distillation technique that folds classifier-free guidance into a single decoding step rather than running it at every token. The result is a 23% reduction in latency without sacrificing synthesis fidelity — a meaningful win for real-time applications.
CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
CuteTTS combines continuous latents with patch-level autoregression and guidance-step distillation for efficient zero-shot voice cloning. By absorbing classifier-free guidance into a single step, it reduces latency by 23% while maintaining high-fidelity synthesis quality.
Abstract
Zero-shot text-to-speech (TTS) now supports interactive assistants, personalized media, and accessibility tools. All TTS systems require faithful linguistic rendering, consistent speaker identity, and low-latency response. Yet compact streaming systems must preserve sufficient acoustic detail in a predictable low-rate latent sequence, while iterative diffusion sampling and classifier-free guidance multiply inference cost at every autoregressive step. To strike a balance between high-fidelity synthesis and low-latency inference, we present CuteTTS, a compact continuous-autoregressive TTS system. It combines semantically aligned causal VAE latents with patch-level autoregression, explicit speaker conditioning, and a bidirectional flow-matching head. We further introduce guidance-step distillation, which absorbs classifier-free guidance and multiple solver steps into a single interval-conditioned student. Evaluations on LibriSpeech and Seed-TTS-Eval demonstrate competitive intelligibility and speaker similarity in zero-shot voice cloning, while distillation lowers first-audio latency by 23.3% and real-time factor by 40.8% relative to the base model with comparable objective and subjective quality. These results provide a practical path toward continuous-autoregressive TTS that reconciles high-fidelity generation with the latency demands of real-time interaction.
Trending on Hugging Face
MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting
MuseTalk generates high-fidelity lip-synced face videos in real-time by encoding face images and audio features in a latent space with a Variational Autoencoder and U-Net.
LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time
LiveAnimate enables real-time, long-form pose-driven human animation via a 14B-parameter video diffusion transformer with specialized training, bounded attention caching, and sequence parallelism.
dots.tts Technical Report
dots.tts is a 2B-parameter continuous autoregressive text-to-speech model that generates speech in a semantically structured continuous latent space. Innovations include full-history conditioning and self-corrective post-training for robust, expressive, and low-latency multilingual speech.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.