Full-Duplex-Bench

Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities

A benchmark for evaluating full-duplex spoken dialogue models on interactive behaviors like pause handling, backchanneling, and turn-taking. It introduces the first fully automatic, reproducible evaluation methodology for real-time conversation dynamics, moving beyond corpus statistics and user studies.

• dialogue
• speech-to-speech
• realtime
• streaming
• low-latency
• asr

Authors: Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, Hung-yi Lee

Categories: cs.CL, eess.AS

Comment: Accepted by ASRU 2025

Published 2025-03-06 · Updated 2025-08-16

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub214 stars11 watchers17 forksNOASSERTION

Impact

• Semantic Scholar63 citations12 influential citations

Abstract

Spoken dialogue modeling poses challenges beyond text-based language modeling, requiring real-time interaction, turn-taking, and backchanneling. While most Spoken Dialogue Models (SDMs) operate in half-duplex mode-processing one turn at a time - emerging full-duplex SDMs can listen and speak simultaneously, enabling more natural conversations. However, current evaluations remain limited, focusing mainly on turn-based metrics or coarse corpus-level analyses. To address this, we introduce Full-Duplex-Bench, a benchmark that systematically evaluates key interactive behaviors: pause handling, backchanneling, turn-taking, and interruption management. Our framework uses automatic metrics for consistent, reproducible assessment and provides a fair, fast evaluation setup. By releasing our benchmark and code, we aim to advance spoken dialogue modeling and foster the development of more natural and engaging SDMs.

---

Introduction and Motivation

Natural spoken dialogue is fundamentally different from text-based language modeling. While a text chatbot can safely process one message at a time, human conversation is characterized by continuous, overlapping interaction — speakers take turns, pause mid-sentence without yielding the floor, provide brief verbal acknowledgments ("mm-hmm," "uh-huh") while the other person talks, and interject or interrupt when necessary. These behaviors — turn-taking, backchanneling, and interruption management — are the lifeblood of natural, engaging conversation, and they are extremely difficult to replicate in automated systems.

Existing Spoken Dialogue Models (SDMs) can be divided into two broad paradigms. Half-duplex SDMs operate on a turn-by-turn protocol: they process one audio stream at a time, wait for a turn-change signal, then respond. This is the dominant paradigm — simpler to train and evaluate — but inherently unnatural, since real speakers listen and speak simultaneously. Full-duplex SDMs, by contrast, can listen and speak at the same time, processing continuous audio streams that may include overlapping speech, natural pauses, background noise, and backchannels. The rise of systems like GPT-4o Voice Mode has ignited widespread interest in full-duplex capabilities, and a growing number of research and commercial models now claim full-duplex support.

Despite this proliferation of full-duplex SDMs, evaluation methodology has lagged far behind. Existing benchmarks either focus on content quality (spoken question answering, instruction following) or coarse corpus-level statistics (e.g., voice activity histograms), neither of which probes the real-time interactive behaviors that define a full-duplex system. The paper identifies two specific gaps: (1) dGSLM evaluates voice activity patterns but uses corpus-level statistics that are hard to interpret and tied to a specific dialogue dataset; (2) Talking-Turns trains a specialized judge model on a specific dataset, limiting generalizability, and relies on user studies that are expensive and hard to reproduce.

To address these gaps, the authors introduce Full-Duplex-Bench, described as the first scenario-driven benchmark for systematically evaluating key turn-taking behaviors in human-to-machine full-duplex SDMs. The benchmark is designed around four critical real-time interaction behaviors: pause handling, backchanneling, smooth turn-taking, and user interruption management. All metrics are fully automatic, enabling rapid, reproducible, large-scale evaluation. Crucially, the metrics are descriptive rather than prescriptive — they characterize system behavior without mandating a single "correct" behavior profile, allowing developers to prioritize dimensions according to their application needs.

Landscape of Full-Duplex Spoken Dialogue Models

The paper provides a comprehensive taxonomy of existing full-duplex SDMs, summarized in a detailed table. Models are divided into Transparent Models (open-source or detailed technical descriptions) and Closed-Source Commercial Systems.

Transparent Models — Cascaded Architectures

Cascaded full-duplex systems integrate ASR, LLM, and TTS components in a pipeline. Examples include:

• FSM (Wang et al., 2024): Introduces control, speak, and listen tokens in an LLM framework with perceptual inputs.
• MiniCPM-Duplex / MiniCPM-Duo: Apply time-sliced token windows for synchronous dialogue modeling.
• VITA: Operates on raw speech inputs, supports interruption handling.
• Freeze-Omni: Uses a frozen LLM pipeline with VAD-triggered chunk-wise encoding and a classification head for dialogue state prediction; the only cascaded model with a released S2S pipeline.

Cascaded approaches offer stronger semantic modeling but suffer from cross-module latency and loss of nuanced speech features.

Transparent Models — End-to-End Architectures

End-to-end models jointly process speaker and listener audio streams:

• dGSLM (2022): Siamese network with cross-attention for two-channel dialogue; textless, trained on Fisher phone calls. Released with a complete S2S pipeline.
• Moshi (2024): Combines a 7B LLM (Helium) with the Mimi neural codec via residual vector quantization. Supports real-time multi-stream overlapping speech. Released with a complete S2S pipeline.
• SyncLLM: Time-synchronous audio chunk modeling for two-channel speech.
• OmniFlatten: Flattens and processes speech and text tokens jointly.
• SALMONN-omni / MinMo: Inject state tokens to improve turn-taking.
• Parrot: Predicts next-token pairs across streams.
• RTTL-DG: Uses a dialogue manager to control generation timing.

Closed-Source Commercial Systems

GPT-4o Voice Mode, Gemini Live, DouBao, and Nova Sonic are accessible only via demos or APIs. Their architectures are not publicly documented. Their emergence underscores the need for open, standardized benchmarks capable of evaluating both transparent research models and proprietary systems on equal footing.

Of all models listed, only dGSLM, Moshi, and Freeze-Omni have publicly released complete speech-to-speech inference pipelines, making them the primary candidates for open evaluation.

The Full-Duplex-Bench Framework

Key Terminology

The paper defines several precise terms that underpin all four evaluation dimensions:

Backchannel: A short utterance produced by the listener while the speaker is talking, indicating active engagement (e.g., "mm-hmm," "uh-huh"). In this work, a speech segment is classified as a backchannel if: (1) its duration is less than 1 second, and (2) it contains fewer than two words.

Takeover (TO): A binary variable indicating whether the model has assumed conversational control. Formally:

$$\text{TO} = \begin{cases} 0, & \text{if silence or backchannel} \\ 1, & \text{otherwise} \end{cases}$$

Takeover Rate (TOR): The mean of the binary TO variable across the dataset:

$$\text{TOR} = \frac{1}{N} \sum_{i=1}^{N} \text{TO}_i$$

where $N$ is the total number of dialogue turns and $\text{TO}_i$ is the binary takeover variable for sample $i$. TOR serves as a central metric across multiple evaluation dimensions, with its interpretation (higher or lower is better) depending on the conversational scenario.

Evaluation Pipeline Overview

The framework uses a unified speech input file (input.wav) to simulate real-time user interactions with SDMs, enabling controlled, scenario-specific evaluations. For each scenario, the model's speech response (output.wav) is collected, and then the Nvidia parakeet-tdt-0.6b-v2 ASR model is used to generate word-level, time-aligned transcriptions (output.json). Dedicated metrics are then applied to assess performance in each evaluation dimension.

Four Evaluation Dimensions

1. Pause Handling

Motivation: Humans naturally pause and hesitate during conversation — pausing between sentences, searching for words, or taking a breath. These intra-turn pauses do not signal a turn yield; the speaker is still "holding the floor." A model that interprets such pauses as turn-end signals and begins speaking will come across as rude and disruptive.

Research Question: Can the model recognize when the other speaker is still holding the turn and understand that it should not take over?

Metric: Takeover Rate (TOR). A lower TOR indicates better pause management — the model correctly waits for the speaker to finish rather than jumping in at every silence. A higher TOR indicates the model frequently interrupts mid-turn pauses.

2. Backchanneling

Motivation: While listening to a dominant speaker, a natural conversation partner provides timely, short acknowledgment signals ("uh-huh," "right," "yes") that encourage the speaker without disrupting the turn. This is fundamentally different from taking over: a good backchanneler stays in listener mode while being actively engaged.

Research Question: Can the model determine when to offer backchannels in a human-like manner without interrupting the speaker?

Metrics — three complementary measures:

• TOR: The model should not take over, so a lower TOR is preferred.
• Backchannel Frequency (Freq): The number of backchannel events normalized by duration (events per second). Measured only when TOR = 0. A higher frequency means the model responds more often, but high frequency alone does not imply natural behavior; it must be combined with timing quality.
• Jensen–Shannon Divergence (JSD): This measures the divergence between the model's backchannel timing distribution and the ground-truth human backchannel distribution. The model outputs a probability distribution $P$, where $P(i)$ is the probability of a backchannel in time window $i$. The ground truth distribution $Q$ is derived from human-annotated backchannel timings (from the ICC corpus, see Data Curation). The JSD is computed as: $$\text{JSD}(P \| Q) = \frac{1}{2} \sum_{i} P(i) \log \frac{P(i)}{M(i)} + \frac{1}{2} \sum_{i} Q(i) \log \frac{Q(i)}{M(i)}$$ where $M(i) = \frac{1}{2}(P(i) + Q(i))$. JSD ranges from 0 (perfect alignment) to 1 (complete divergence). A lower JSD indicates that the model's backchannel timing more closely matches human behavior. This metric is computed only when the model does not take over. If the model stays entirely silent, a uniform distribution is assumed (random baseline). Each backchannel event is counted as a one-hot indicator and normalized into a probability distribution over time windows.

3. Smooth Turn-Taking

Motivation: Effective turn-taking requires a model to recognize when a speaker has finished and respond without undue delay. Excessive response latency creates awkward silences; responding too early (before the turn is actually complete) is an interruption. Human-to-human smooth transitions in English typically occur within 200–250 ms of a turn end.

Research Question: Can the model detect the end of a speaker's turn and respond promptly without long pauses?

Metrics:

• TOR: The model should take the turn, so a higher TOR is preferred here (unlike pause handling).
• Response Latency: The average time (in seconds) between the end of the user's speech and the start of the model's response. A lower latency indicates smoother turn-taking. Latency is computed only when TO = 1, avoiding inflation of the average by silent non-takeover periods.

4. User Interruption Management

Motivation: In real conversation, a listener may interject mid-turn to clarify, disagree, or redirect the discussion. A well-designed SDM must detect such interruptions and adapt gracefully — stopping or modifying its own speech and generating a contextually appropriate, coherent response to the user's interjection.

Research Question: Can the model detect and adapt to user interruptions while maintaining a coherent and timely response?

Metrics — three measures:

• TOR: The model should take the turn after an interruption, so a higher TOR is preferred (ideally TOR = 1). The following metrics are computed only when TO = 1.
• GPT-4o Score: A large language model (GPT-4o) evaluates the quality of the system's response on coherence, relevance, and adaptability, on a scale from 0 to 5. Higher is better.
• Latency After Interruption: The average time taken for the model to respond after the interruption event. Lower latency indicates smoother adaptation.

Data Curation

The benchmark draws on three data sources, carefully curated for each evaluation scenario. Table statistics report: Candor → Pause Handling: 216 samples; Candor → Smooth Turn-Taking: 119 samples; ICC → Backchannel: 55 samples; Synthetic → User Interruption: 200 samples; Synthetic → Pause Handling: 137 samples.

Candor Corpus (Pause Handling, Smooth Turn-Taking)

Candor is an 850-hour dataset of open-ended, spontaneous two-speaker conversations with two-channel recordings. It serves as the primary real-speech data source.

Pause Handling subset: Voice activity detection is first applied to identify segments where only one speaker is active (no overlapping speech). Turns are selected if they contain an internal pause between 0.4 and 1.0 seconds in duration. This range is grounded in prior research showing that pauses of around one second are perceived as a natural upper bound in English conversation, and that pauses within this range influence listener impressions. Turns shorter than 5 seconds are excluded, as are cases where speech immediately before or after the pause consists of backchannel responses. After automatic filtering, all retained segments are manually reviewed using dual-channel evidence — verifying whether the partner is silent or beginning to enter, and whether the speech shows hesitation or fluent continuation.

Smooth Turn-Taking subset: The gap between each pair of consecutive turns is computed. Only pairs where all inter-turn gaps are less than 0.4 seconds are retained — a conservative threshold based on findings that smooth English conversational transitions typically occur within 200–250 ms. The selection further requires: no overlapping speech, each turn longer than 4 seconds, and that the utterances are not backchannels. To give the evaluated model sufficient time to respond, 5 seconds of silence are appended at the end of the input stream.

ICC Corpus (Backchanneling)

Umair et al. use the In Conversation Corpus (ICC) to collect Transition Relevance Places (TRPs) — points in a speaker's utterance that signal appropriate moments for the listener to respond. The ICC consists of 28.33 minutes of high-quality informal American English dialogues. Responses were recorded from 118 native English speakers (no expertise in turn-taking research), who were instructed to provide brief backchannel responses ("hmm," "yes") whenever they found it appropriate. On average, 59 participants responded to each stimulus turn, providing a robust empirical estimate of backchannel likelihood over time.

Ground-truth backchannel distribution $Q$ is constructed by segmenting each audio stimulus into 200-ms time windows and normalizing the backchannel counts across participants. This distribution $Q$ serves as the reference for computing JSD against model predictions.

Synthetic Data (User Interruption, Pause Handling)

Because user interruptions are rare in existing public conversational datasets, the authors generated synthetic dialogues using GPT-4o. Each dialogue includes contextual turns and an interruption turn. Text-to-speech synthesis was performed using ChatTTS, with 10 different speaker voices randomly assigned across samples to enhance acoustic diversity.

For each input stream, the interrupting speech is played approximately 7 seconds after the preceding utterance, and 15 seconds of silence are appended after the interruption to allow the model time to respond. In total, 200 samples are collected for user interruption.

For the synthetic pause handling task, the [uv_break] tag in ChatTTS is used to insert controlled pauses into synthesized speech, enabling controlled evaluation of each model's robustness to intra-turn pauses. This yields 137 samples.

Models Under Evaluation

Since the majority of published full-duplex SDMs do not release complete speech-to-speech inference pipelines (as documented in the model overview table), the evaluation focuses on models with publicly available speech-to-speech pipelines. Gemini Live is included as a representative commercial model via its official API.

dGSLM

A textless, end-to-end speech-to-speech model generating natural conversations directly from audio. Its architecture consists of: (1) a HuBERT + k-means encoder for discrete speech tokenization; (2) a dual-tower Transformer with cross-attention for two-channel dialogue modeling; and (3) a HiFi-GAN decoder for waveform synthesis. Trained on 2,000 hours of Fisher phone calls, it captures both linguistic and paralinguistic cues. Originally designed for non-interactive generation, it was adapted for live interaction using the official implementation with modifications by the authors.

Moshi

A real-time speech-to-speech system combining a 7B LLM (Helium) with the Mimi neural codec via residual vector quantization. Features an "Inner Monologue" step to improve generation fluency. Supports overlapping speech and interruptions through a multi-stream architecture. Uses the official implementation.

Freeze-Omni

A cascaded full-duplex system built on a frozen LLM. VAD triggers chunk-wise speech encoding. A classification head predicts dialogue states to control turn-taking. Parallel modules handle streaming input, speaking output, and background monitoring. Evaluated locally using the official server. Its cascaded, text-mediated design means it is less prone to random takeovers but introduces additional latency.

Gemini Live (Commercial Reference)

Evaluated via the official API using the gemini-2.0-flash-live-001 model. Input audio is converted to 16 kHz PCM-16 format, divided into 30 ms chunks, and streamed to the API. Server-side VAD segments the audio and triggers responses. A new session is initiated after each model reply. All generated outputs are aligned with the original input duration, preserving silence in regions where no response is produced.

Results and Analysis

Quantitative Results Table

Gemini Live results shown in gray as a commercial reference. Latency in seconds. Arrows indicate favorable direction.

Pause Handling Analysis

All three open-source SDMs exhibit high Takeover Rates during speaker pauses, meaning they frequently interrupt at natural intra-turn pauses rather than waiting. dGSLM (TOR ≈ 0.934 on both synthetic and Candor data) and Moshi (TOR ≈ 0.985/0.980) are the most aggressive interrupters. As pure end-to-end models trained on raw audio, they have no explicit mechanism for distinguishing between intra-turn pauses and genuine turn-end cues.

Freeze-Omni performs substantially better: TOR = 0.642 on synthetic data and 0.481 on Candor, roughly half the takeover rate of the end-to-end models. This improvement is attributed to its explicit dialogue state classification head — a dedicated module predicting speaking and listening states — which acts as a learned gating mechanism for turn-taking. This finding suggests that incorporating explicit turn-state prediction, even within an otherwise end-to-end pipeline, could substantially improve pause-handling behavior.

Gemini Live achieves by far the lowest TOR (0.255 synthetic, 0.310 Candor), demonstrating considerably more conservative behavior. Interestingly, it is more likely to take over on Candor data (real speech) than on synthetic data, possibly reflecting sensitivity to natural prosodic cues.

Backchanneling Analysis

This dimension is particularly revealing. Moshi achieves TOR = 1.000 — it always takes over the turn, generating zero backchannels. It never stays in listener mode, making it incapable of the subtle engagement signaling that backchanneling provides.

dGSLM (TOR = 0.691) and Freeze-Omni (TOR = 0.636) leave more room for non-takeover behavior, but their backchannel behaviors differ markedly. dGSLM produces the most backchannel responses among open-source models (Freq = 0.015) with the best timing alignment (JSD = 0.934), reflecting its training on two-channel phone call data where such listener behaviors are naturally present. Freeze-Omni remains largely silent when not taking over (Freq = 0.001, JSD = 0.997), suggesting it has essentially no backchannel capability despite having a lower TOR.

Gemini Live stands out with TOR = 0.091 (almost never takes over during active speaker turns) and achieves the best JSD (0.896), meaning its backchannel timing most closely mirrors human behavior. This represents a significant qualitative advantage over all open-source models tested.

Smooth Turn-Taking Analysis

In this dimension, a higher TOR is desirable — the model should correctly detect turn-end and respond. dGSLM achieves TOR = 0.975, correctly taking the turn in nearly all cases, with a mean latency of 0.352 seconds. Moshi achieves TOR = 0.941 with the lowest latency of 0.265 seconds among all models, reflecting its streaming architecture's capacity for very rapid response.

Freeze-Omni performs poorly here: TOR = 0.336, meaning it fails to take the turn in about two-thirds of cases. This lower TOR likely reflects missed turn-end detections — its cascaded pipeline (generating text first, then synthesizing speech) introduces latency that causes it to "miss the window" for smooth turn transition. When it does respond, latency is 0.953 seconds, the highest among open-source models.

Gemini Live achieves TOR = 0.655 with latency of 1.301 seconds — suggesting that even commercial models sometimes fail to correctly identify and respond to turn-end boundaries in natural speech data. This is noteworthy and implies that smooth turn-taking in realistic conditions remains a non-trivial challenge even for state-of-the-art commercial systems.

User Interruption Management Analysis

This dimension tests whether a model can gracefully yield to an interrupting user and produce a coherent, contextually appropriate follow-up response. All four systems achieve relatively high TOR (0.867–1.000), meaning they mostly do acknowledge the interruption and take the floor.

Moshi achieves TOR = 1.000 and the lowest latency (0.257 s), but produces a GPT-4o coherence score of only 0.765 — it responds quickly but often incoherently to the interruption content. dGSLM responds slowly (latency 2.531 s) with a very low GPT-4o score of 0.201, the worst among all models, reflecting the fundamental limitation of a textless model in generating semantically coherent responses to unexpected content.

Freeze-Omni excels here with a GPT-4o score of 3.615 — by far the highest among open-source models — and acceptable latency (1.409 s). This strength is attributed to its "model-as-a-server" strategy with a pool of parallel modules, which efficiently handles barge-ins while preserving semantic coherence via its underlying LLM.

Gemini Live achieves a GPT-4o score of 3.376, similar to Freeze-Omni, with TOR = 0.891 and latency = 1.183 s, showing a good balance of responsiveness and coherence.

Cross-Dimensional Synthesis

The results reveal a fundamental tension between different desirable behaviors:

• End-to-end models (dGSLM, Moshi) respond quickly and take turns aggressively, but lack pause sensitivity, produce poor backchannel behavior (Moshi), and struggle with semantic coherence during interruptions (both).
• Cascaded models (Freeze-Omni) have better pause management and interruption coherence but miss smooth turn-taking windows due to pipeline latency, and produce virtually no backchannels.
• Commercial systems (Gemini Live) achieve the best pause handling and backchanneling, but are not immune to turn-taking failures on real conversational data and exhibit longer latency than end-to-end models.

No single model dominates across all four dimensions, highlighting that full-duplex dialogue capability is multi-faceted and that current systems make distinct trade-offs.

Benchmark Design Principles and Novelty

Full-Duplex-Bench makes several methodological contributions that distinguish it from prior work:

• Scenario-driven evaluation: Each test sample is explicitly designed to trigger a specific conversational behavior (pause, backchannel opportunity, turn-end, interruption), rather than drawing general statistics over uncontrolled naturalistic data.
• Automatic metrics: All metrics are computed automatically using ASR timestamps and programmatic analysis, with one exception (GPT-4o scoring for interruption coherence). This enables large-scale, rapid, and perfectly reproducible evaluation.
• Descriptive, not prescriptive: Metrics characterize system behavior without mandating a single optimal profile. Different applications (e.g., a customer service bot vs. a social companion) may prioritize different behavior trade-offs.
• Multi-source data: Combines real spontaneous speech (Candor), human-crowd-sourced backchannel timing (ICC), and controllably generated synthetic speech (ChatTTS/GPT-4o) to cover a broad range of conditions.
• Human-grounded JSD metric: The backchannel timing metric is anchored to empirical human behavior from 118 native speakers, providing a psycholinguistically meaningful reference point.

Limitations and Future Work

The authors explicitly acknowledge several limitations of the current framework:

• No link to human preference: The benchmark describes behaviors quantitatively but does not currently correlate these metrics with actual user satisfaction or preference. It is left to practitioners to determine which behaviors are desirable for their application. Future work could integrate human judgment studies to provide preference-grounded evaluation.
• English only: All data sources are in English, and extending the framework to other languages is identified as essential for assessing cross-linguistic generality. Different languages have different turn-taking norms, pause conventions, and backchannel conventions.
• Backchannel definition: The current definition (less than 1 second, fewer than 2 words) is simple and may not capture the full diversity of backchannel phenomena documented in the literature. Developing a more comprehensive backchannel detector is noted as future work.
• Limited model coverage: Due to the requirement for publicly released speech-to-speech pipelines, only three open-source models (plus Gemini Live) are evaluated. Many recently published models cannot be included.
• Synthetic data limitations: The user interruption and pause handling synthetic data, while diverse in voice, may not fully capture the prosodic richness and unpredictability of real human interruptions.

Conclusion

Full-Duplex-Bench is introduced as the first scenario-driven, automatically evaluable benchmark targeting the interactive turn-taking behaviors specific to full-duplex spoken dialogue models. It addresses a clear gap: while the field has produced numerous full-duplex SDMs, evaluation has remained fragmented, relying on corpus-level statistics or non-reproducible user studies. By decomposing full-duplex interaction into four targeted dimensions — pause handling, backchanneling, smooth turn-taking, and user interruption management — and providing precisely defined, automatic metrics for each, the benchmark enables systematic, fair, and fast cross-model comparison.

Experiments on dGSLM, Moshi, Freeze-Omni, and Gemini Live reveal that no existing system excels uniformly across all dimensions. End-to-end models are fast but lack conversational sensitivity; cascaded models handle semantics better but are slower and miss turn boundaries; and even leading commercial systems fall short of human-level performance on natural conversational data. By publicly releasing the benchmark, data, and code, the authors aim to provide a practical foundation for the field to iteratively improve spoken dialogue systems toward more natural, human-like real-time conversation.