Full-Duplex Dialogue via RL and Robust Contextual ASR
Today's digest covers two papers from National Taiwan University: one decouples conversational timing from reasoning in full-duplex spoken models using reinforcement learning, and another introduces COALA, a framework for robust contextual biasing in speech-augmented ASR.
Example token-level conversational pattern. Compared with the SFT baseline, DuplexPO maintains and releases / impulses around user speech instead of staying near PAD. From National Taiwan University.
Today's two papers from National Taiwan University tackle complementary challenges at the frontier of spoken AI: making real-time dialogue feel natural through principled turn-taking, and making ASR reliably precise when domain-specific rare words are at stake.
SpeechLLMs & Spoken Dialogue
Teaching models when to speak — and when to stay quiet.
Full-duplex spoken dialogue systems must handle turn-taking, backchanneling, and barge-in in real time, but most architectures conflate these timing decisions with higher-level semantic reasoning. This entanglement often forces an uncomfortable tradeoff: better conversational dynamics at the cost of response quality, or vice versa. This paper tackles that tension head-on by factoring the two apart, training a separate RL policy dedicated solely to conversational timing while leaving the semantic reasoning module untouched.
Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning
This paper decouples conversational timing from semantic reasoning using a separate RL policy, enabling natural turn-taking and backchanneling without sacrificing reasoning ability. A factorized reward design provides fine-grained credit assignment for turn initiation, yielding, and barge-in handling in dialogue.
Abstract
Recent full-duplex spoken dialogue models have demonstrated compelling progress toward human-like interaction, enabling agents to respond with low latency, produce backchannels, and handle user barge-ins. Yet these improvements in conversational dynamics often come with weaker reasoning and instruction-following abilities, revealing a potential tension between interactive dynamics and intelligence capability. In this paper, we argue that such an intelligence--dynamics trade-off is not fundamental: conversational dynamics can instead be learned as a separate real-time decision policy from human dialogue data. To this end, we propose DuplexPO, a reinforcement learning (RL) framework that decouples when to speak from what to say. It preserves the semantic response capability of an instruction-tuned assistant, while optimizing its temporal interaction behavior over selected high-impact windows from long human conversations. To quantitatively optimize these dynamics, we formulate the Factorized Conversational Dynamics Reward (FCDR) to enable fine-grained temporal credit assignment for turn initiation, backchanneling, yielding, and regularized participation. The policy is then optimized with a GRPO-style objective. Experiments show that DuplexPO substantially improves full-duplex behaviors, including timely backchannels, smooth turn-taking, and barge-in handling, while maintaining strong reasoning and instruction-following performance. Moreover, improvements in dynamics-oriented metrics are reflected in better user experience, suggesting that optimizing conversational timing as a standalone objective can promote more natural full-duplex interaction.
The key insight is a factorized reward design that assigns fine-grained credit to distinct dialogue events — turn initiation, turn yielding, and barge-in handling — independently. This granularity allows the RL policy to learn nuanced timing behavior without noisy, entangled gradient signals muddying the semantic backbone. The result is a full-duplex model that can engage naturally in overlapping speech scenarios without sacrificing reasoning ability, a meaningful step toward voice agents that actually feel conversational.
ASR & Voice Agent Architectures
Recognizing the rare words that matter most.
Contextual biasing for ASR — boosting recognition of domain-specific entities like product names, medical terms, or proper nouns — becomes significantly harder when candidate lists are large and multiple rare words co-occur in the same utterance. Existing speech-augmented language models tend to collapse under these conditions, losing the ability to differentiate relevant entities from distractors. COALA addresses this with a framework built around contrastive regularization and explicit biasing score estimation.
COALA: Robust Contextualized Speech-augmented Language Modeling for ASR via Contrastive Regularizer and Biasing Score Estimation
COALA enhances speech-augmented language models for ASR by learning to score and select domain-specific entities from large candidate lists. The framework introduces decoupled loss functions that prevent training collapse when multiple rare words co-occur, enabling robust contextual biasing in complex scenarios.
Abstract
Contextual biasing seeks to integrate external knowledge into automatic speech recognition (ASR) systems to accurately recognize domain-specific entities. In this paper, we propose COALA (Contextualized ASR Leveraging Biasing Scoring), a robust framework designed to enhance speech-augmented language models (SLMs) in complex multi-entity scenarios. Considering the inherent context-window limitations of SLMs, identifying relevant target entities from a large-scale biasing list is crucial for effective recognition. To this end, COALA maps SLM latent representations into a specialized discriminative space to quantify the matching intensity between audio segments and candidate entities. Furthermore, we address the training collapse in prior study when handling multi-target utterances-where multiple rare words co-occur. Experimental results on the LibriSpeech benchmark demonstrate that COALA consistently achieves superior contextual biasing performance across various biasing list scales.
The framework introduces decoupled loss functions that prevent training collapse in multi-entity scenarios, paired with a learned scoring mechanism that ranks and selects the most relevant entries from large candidate pools. By separating the biasing signal from the core language modeling objective, COALA achieves robust contextual ASR even in complex, real-world conditions where standard approaches degrade — a practical advance for voice agents deployed in specialized domains.
Trending on Hugging Face
Vidu S1: A Real-Time Interactive Video Generation Model
Vidu S1 is a real-time interactive video generation model that supports voice-controlled digital character animation with infinite-length output and high frame rate on consumer hardware.
Moshi: a speech-text foundation model for real-time dialogue
A speech-text foundation model for real-time dialogue via end-to-end speech-to-speech generation instead of cascaded components. By jointly modeling overlapping audio streams with text-token prediction, it achieves 200ms latency while preserving emotion and handling natural conversational dynamics like interruptions.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.