Benchmarking Voice AI and Building Real-Time Conversational Agents
Today's digest covers two fronts in voice and conversational AI: Hume AI's multidimensional real-world benchmark exposing critical gaps in voice systems, and Alibaba's unified video model enabling sub-second multimodal conversational agents.
Overview of the benchmark. evaluates voice AI along four domains: (1) Text-to-Speech (TTS), (2) Speech-to-Speech (STS), (3) Speech Understanding (SU), (4) ASR Robustness. Together the four domains characterize a voice system as a capability profile rather than a single aggregate score. From Hume AI.
Today's digest is a compact but pointed pair of papers tackling the two sides of the voice AI coin: how to build real-time conversational agents, and how to evaluate whether the systems we've built actually work in the wild. Both papers land squarely in the Speech-to-Speech & Voice Agents space, and together they paint a clear picture of where the field stands and where it needs to go.
Speech-to-Speech & Voice Agents
From architectural blueprints to honest evaluation — two complementary perspectives on conversational AI.
Real-Time Multimodal Agents
Building a truly interactive conversational agent requires more than language understanding — it demands synchronized perception and generation across modalities with minimal latency. Alibaba Group's work introduces a compelling new framing: video is not a monolithic stream, but rather a persistent world state overlaid with a continuous event stream of temporal changes. This decomposition underpins a unified pretraining objective that teaches the model to predict how a world responds to multimodal user input, producing synchronized audio-visual output in real time. The result is a system capable of sub-second latency — a genuine prerequisite for natural, turn-taking conversational agents. This paper was featured on today's Hugging Face Daily tab.
Video = World + Event Stream
This paper decomposes video as "world + event stream"—persistent context plus temporal changes. Unified pretraining enables the model to predict how a world responds to multimodal user input with synchronized audio-visual output in real-time, achieving sub-second latency for natural conversational agents.
Abstract
We present Wan-Streamer v0.3, which reframes our native-streaming interaction model under a single organizing view: a video is a world plus an event stream. The world is the persistent context in which a video unfolds, including the environment, scene, subjects, ambient acoustic conditions, voice characteristics, and other relatively stable conditions. The event stream is everything that changes over time within that world, including scene or environmental changes, subject behavior, speech, and other sounds. This yields a general-purpose pretraining task over large amounts of real video: given a world and incoming input, predict how the world moves, changes, and responds in real time. The resulting competence can be specialized to a broad family of real-time downstream tasks. We instantiate it on real-time full-duplex audio-visual interaction, where the event stream is the agent's speech together with free-form behavior. Functionally, the model's multimodal understanding process is vision-language-action-like: it maps multimodal user input to language-form speech and behavior actions. Wan-Streamer v0.3 preserves the v0.2 operating point: 640x368 video at 25 FPS, a 160 ms streaming unit, approximately 200 ms model-side response latency, and approximately 550 ms total interaction latency under a 350 ms bidirectional network budget.
Benchmarking Voice AI in the Real World
Even as architectures grow more capable, evaluation remains a stubborn weak point. Hume AI's RW-Voice-EQ Bench directly addresses this gap with a multidimensional benchmark spanning text-to-speech, speech-to-speech, speech understanding, and ASR. Crucially, the benchmark is designed to probe real-world robustness — and its findings are sobering: many contemporary voice AI systems fail to meaningfully leverage acoustic information, defaulting instead to transcript-driven behavior. This means expressive, paralinguistic, and acoustic cues are largely being ignored, and performance gaps that standard benchmarks hide become plainly visible here.
RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems
A multidimensional benchmark evaluating voice AI across text-to-speech, speech-to-speech, speech understanding, and ASR. It reveals that many systems fail to leverage acoustic information and remain transcript-driven, exposing real-world robustness gaps invisible to standard benchmarks.
Abstract
Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.
Takeaway: The gap between lab benchmarks and real-world voice AI performance is larger than most results suggest. Systems that look strong on standard metrics may be leaning heavily on text transcripts rather than the acoustic signal itself — a fragility that RW-Voice-EQ Bench is designed to expose.
Trending on Hugging Face
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.
Moonshine: Speech Recognition for Live Transcription and Voice Commands
Moonshine, an encoder-decoder transformer architecture for speech recognition, uses Rotary Position Embedding, reducing compute requirements without decreasing accuracy.