Voice Agents Level Up: Unified Speech-Reasoning, Enrollment-Free Speaker Lock, and a New ASR Benchmark
Today's digest covers three advances in voice agent infrastructure: a 27B open model unifying speech, reasoning, and tool use; enrollment-free target-speaker extraction for live calls; and a new benchmark reframing ASR as grounded generative reformulation.
Overview of GSR-Bench: Data Construction, Quality Assurance, and Hierarchical Task Organization. From China Telecom Artificial Intelligence Technology Co. Ltd.
Today's papers push the frontier of voice agents on three fronts simultaneously — making them smarter at reasoning and tool use, more robust to noisy multi-speaker environments, and evaluated against a richer definition of what "good" speech recognition actually means. A lean but cohesive set of three papers, all squarely in the SpeechLLM and voice-agent space.
SpeechLLMs & Voice Agents
From unified backbones that reason and act, to zero-enrollment speaker isolation, to rethinking how we score ASR — the building blocks of production voice agents are getting a serious overhaul.
Open, Reasoning-Native Voice Models
ServiceNow Research's work on Apriel Voice tackles one of the most persistent annoyances in voice-assistant stacks: the need for a separate dialogue model stitched on top of a speech encoder. Instead, Apriel Voice folds speech understanding, multi-step reasoning, and tool-call generation into a single 27B backbone. The two-stage training recipe is worth noting — stage one plants the speech interface, and stage two uses efficient online distillation to jointly tune the model for conversational reasoning and tool interaction, avoiding the usual catastrophic forgetting trap. This paper comes from today's Hugging Face Daily tab and has been picking up upvotes quickly.
Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool Use
Apriel Voice unifies speech understanding, reasoning, and tool use within a single backbone, enabling voice assistants to process speech directly and generate both responses and tool calls without separate dialogue models. A two-stage training procedure first establishes the speech interface, then jointly adapts the model for conversational reasoning and tool interaction using efficient online distillation.
Abstract
A capable voice assistant must do more than recognize words; it must understand requests, reason over context, sustain conversations, and act through tools. We present Apriel Voice 27B Preview, an open speech-language model that unifies these capabilities within a shared reasoning backbone while retaining text and image input modalities. Built on the post-trained Gemma 4 26B-A4B checkpoint, Apriel Voice directly processes speech and generates both responses and tool calls without requiring a separate dialogue model. We introduce a two-stage training procedure to adapt the model to speech. The first stage establishes the speech interface, while the second jointly adapts the projector and backbone through supervised fine-tuning, text replay, and source-dependent cross-entropy and forward Kullback--Leibler divergence supervision. Automatically constructed speech-conditioned dialogues provide conversational training data, while local teacher serving, overlapping teacher and student computation, and fused full-vocabulary losses enable efficient online distillation. In thinking mode, Apriel Voice achieves a score of 98.90 on Big Bench Audio, 81.13 on Speech-BFCL, and a task success rate of 65.51% on $τ$-Voice, with a mean latency of 1.91 seconds. It also achieves 51.27% English EVA accuracy and 74.13% experience pass@1. Finally, we deploy Apriel Voice 27B Preview on a custom serving platform to support interactive voice interactions.
Enrollment-Free Target-Speaker Extraction
One of the quiet pain points for voice agents deployed in the wild is background noise and overlapping speakers — most target-speaker extraction systems require a pre-registered enrollment utterance, which is a non-starter for cold-start calls. VoxLock sidesteps this entirely by bootstrapping speaker identity from a snippet of audio captured during the current call itself, with no prior enrollment step needed. The system fine-tunes a causal speaker-extraction model to deliver near-production-quality separation at minimal latency, and it runs comfortably on consumer hardware — a practical constraint that often gets ignored in academic work.
VoxLock: Target-Speaker Extraction for Voice Agents Without an Enrollment Step
VoxLock enables voice agents to isolate the target speaker from background noise using only a voice sample from the current call, eliminating the need for prior enrollment. By fine-tuning a causal speaker-extraction model, it achieves near-production-quality performance with minimal latency and runs efficiently on consumer hardware.
Abstract
A voice agent's speech recognizer transcribes everyone near the microphone, including people other than the caller. I fine-tuned a public causal target-speaker extraction model (REAL-TSE, 27M parameters) on one GPU to work from a voice sample cut from the call itself, and released it as VoxLock. I score it by what a recognizer writes on real recordings. On Krisp's 265-recording benchmark it reaches 4.95% WER against 23.21% unprocessed and lets 9 background words into the transcript over 85 minutes of background-only speech. Its WER is not significantly different from Krisp's best filters and is significantly lower than the other commercial and open systems I tested. On 400 AMI meeting windows built and frozen before scoring, it cuts WER from 42.95% to 10.63%. The 8-bit model runs a live call on 77% of one CPU thread with 24 ms of algorithmic latency. Model, code and per-file scores are open.
Rethinking ASR Evaluation: Grounded Generative Speech Recognition
The final paper challenges something most of the field has taken for granted: that ASR quality equals transcript fidelity. China Telecom AI's work argues that modern speech language models are doing something richer — they reformulate speech into semantically appropriate text, not just literal transcripts. The paper formalizes this as Grounded Generative Speech Recognition (GSR) and introduces GSR-Bench, the first benchmark explicitly designed to accept multiple semantically valid outputs. Evaluation combines grounding assessment (is the output faithful to the audio?) with LLM-based semantic judgment (is the reformulation coherent and useful?), making it far better suited to scoring models like Whisper or speech-capable LLMs than word-error-rate ever was.
From Transcription to Reformulation: Defining and Benchmarking Grounded Generative Speech Recognition
This work reformulates speech recognition from transcription to grounded generative text, introducing GSR-Bench—the first benchmark accepting multiple semantically valid outputs. Unlike conventional ASR evaluation based on exact transcript matching, the paper defines reformulation capabilities and proposes a multi-dimensional evaluation framework combining grounding assessment with LLM-based semantic judgment for modern speech language models.
Abstract
Speech language models have expanded automatic speech recognition (ASR) from transcription to general speech-to-text generation. Yet existing benchmarks still assume a single ground-truth transcript per utterance, an assumption that struggles when models produce readable, formatted, or structured outputs. In this work, we introduce Grounded Generative Speech Recognition (GSR), reformulating speech recognition as generating semantically grounded text instead of a single transcript. We define a grounded reformulation space that characterizes all valid textual realizations supported by the input speech, and establish a four-level hierarchy of reformulation capabilities, including literal transcription, canonical reformulation, semantic reformulation and speech-grounded structuring. Based on this formulation, we present GSR-Bench, the first comprehensive benchmark designed to evaluate generative speech recognition systems beyond exact transcript matching. GSR-Bench covers diverse scenarios including refined meeting transcription, cross-dialect translation, context-aware speech understanding, and robustness under silent or noisy acoustic conditions. To address the limitations of word-level evaluation metrics, we develop a multi-dimensional evaluation framework combining speech grounding assessment and LLM-based semantic judgment, whose reliability is validated through human preference studies. Extensive experiments across representative architectures, ranging from conventional ASR systems to end-to-end Speech-LLMs, reveal fundamental trade-offs between transcription fidelity and generative reformulation ability. Our analysis further identifies critical failure modes, including acoustic hallucination, unsupported generation, and instruction drift, providing insights into the design and evaluation of future grounded generative speech systems.
Trending on Hugging Face
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
VoiceMem introduces a dual-brain streaming memory architecture for speech language models that improves retrieval accuracy, emotional personalization, and real-time efficiency.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.