Context-Aware TTS Evaluation

Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation

A perceptual study revealing that text-to-speech quality depends critically on the target application domain, with naturalness and appropriateness diverging substantially across contexts—challenging the assumption that universal naturalness scores adequately reflect real-world TTS performance.

• tts
• prosody
• emotion

Authors: Dominika Woszczyk, Andreas Triantafyllopoulos, Jura Miniota, Éva Székely, Bjoern Schuller

Lab: Iconic

Categories: eess.AS, cs.LG

Comment: Accepted at Interspeech 26'

Published 2026-06-30 · Updated 2026-06-30

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub0 stars0 watchers0 forksApache-2.0

Impact

• Semantic Scholar0 citations

Abstract

Text-to-speech (TTS) evaluation is an open challenge. While the primary target was "naturalness," recent fidelity gains shifted focus toward "appropriateness" and whether speech is correct for its context. In this work, we examine how perception changes when the expected downstream use varies. We measure the appropriateness and human-likeness of five SOTA TTS systems across five domains: AI assistant, reader, actor, animated character, and spontaneous speaker. Results show appropriateness varies across domains independently of naturalness. While systems shine at reading, expressive domains remain challenging, and optimizing for one can degrade others. Furthermore, naturalness scores tend to penalize stylized speech while rewarding spontaneity. Finally, our study also highlights blind spots in one-size-fits-all evaluation metrics across more expressive domains. We demonstrate that TTS performance is not "solved" but depends on the target domain, requiring context-aware evaluation.

---

Introduction & Motivation

Text-to-speech (TTS) synthesis is fundamentally a one-to-many problem: a single sentence can be spoken in countless ways while remaining perfectly intelligible. Prosody, pacing, and delivery style shift naturally with the situation, the speaker's intent, and the intended audience. A voice perfectly calibrated for an audiobook may feel jarring on an emergency hotline, entirely wrong for a conversational AI assistant, or comically out of place as an animated cartoon character. This contextual dependency means that what counts as "good" synthesis is inseparable from delivery style.

For a long time, naturalness — most often operationalised via Mean Opinion Scores (MOS) — was the dominant evaluation target in speech synthesis research. Yet naturalness is now widely recognised as an ill-defined and multi-dimensional concept. MOS ratings are neither stable nor absolute: they are sensitive to evaluation setup, listener expectations, and even the framing of the intended use, which can flip system rankings entirely. As TTS models have improved dramatically in fidelity and approach ceiling-level MOS values, differences that matter in real applications are no longer reliably reflected in a single naturalness score.

In response, the research community has developed a range of automatic metrics as "objective" alternatives, spanning ASR-based intelligibility, spectral fidelity measures, distributional prosodic similarity scores (e.g., TTSDS), emotional embedding distances, and more recently persona-based benchmarks that assess instruction following and character consistency. While these methods extend evaluation beyond simple MOS, they each provide only partial views: many focus on global similarity or reconstruction accuracy rather than contextual suitability, LLM-as-judge approaches raise reliability concerns, and current benchmarks tend to prioritise linguistic "stress tests" (difficult pronunciations, rare words) over functional performance across spontaneous or expressive domains.

Crucially, task-agnostic scores ignore situational needs. A model with high MOS or low WER may still be perceived as entirely inappropriate for its intended real-world use. Several works now argue evaluation should consider appropriateness — whether speech fits its intended communicative purpose — rather than human-likeness alone. This moves the focus from "does it sound like a human?" to "does it sound right for this task?" However, the systematic impact of the target domain on appropriateness judgements has not been previously studied. Existing work compares read versus spontaneous speech, or examines how accented/disordered speech affects naturalness, but leaves an important gap: how is the same utterance judged when framed for fundamentally different target tasks requiring different operationalisations of expressivity?

Contributions

This paper makes the following core contributions:

• Cross-Domain Perceptual Analysis: A systematic measurement of how perceived naturalness and appropriateness change across five distinct TTS application domains for five state-of-the-art TTS systems, evaluated by 150 human listeners.
• Human-Likeness Paradox Exploration: An empirical demonstration that human-likeness (naturalness) and domain suitability (appropriateness) are distinct dimensions that can be positively correlated, decoupled, or even negatively correlated depending on the domain.
• Domain-Specific Evaluation & Metrics Profiling: Evidence that TTS systems perform differently across domains, and that common automatic metrics (MOS predictors, embedding-based metrics) are not universally indicative of appropriateness — and can actively mislead for expressive or spontaneous domains.

Study Design & Methodology

Perceptual Study Setup

The authors designed a listening test using a Gradio interface hosted on the Prolific crowdsourcing platform. They recruited 150 native English speakers with a ≥95% approval rating, split into 6 sessions of 25 participants each. A Latin Square design was used to distribute 180 samples (30 sentences × (5 TTS systems + 1 ground truth)) across the six sessions, ensuring that each participant evaluated all 30 sentences and all systems without repetition, enabling independent judgement while anchoring comparisons across TTS profiles. Two attention checks were included to filter bots and inattentive participants.

Participants rated each stimulus on two dimensions using a 5-point Likert scale:

• Human-likeness (a more precisely defined operationalisation of "naturalness")
• Convincingness (used instead of "appropriateness" to improve conceptual clarity and reduce framing bias) — rated separately for each of five persona contexts

Each audio stimulus was presented five times, once per persona context (AI assistant, reader, actor, animated character, spontaneous speaker), so that listeners explicitly considered how well the same sample fits each domain.

Stimuli Curation

The authors manually curated 30 speech–text pairs spanning four speech task families:

• Narration: Quotation-style narration excerpts from LibriQuote (narration-only subset), designed to probe descriptive statements in a read-aloud style.
• Spontaneous Conversational: Sentences from MSP-Podcast to stress-test informal phrasing, disfluencies, and natural conversational prosody in realistic acoustic conditions.
• Affect Conversational: Acted dialogue from MELD and AnimeVox, probing emotional expressivity and serving as ground truth for actor and animated character domains.
• Inform: Six informative, conversational sentences generated using Gemini 3 Pro (manually checked), covering statement and interrogation forms of varying length. A proxy ground truth for the AI assistant condition was generated using ElevenLabs v3 (Katie X voice).

To mitigate participants confusing content appropriateness with delivery, an LLM pass was run to label sentences for plausible emotions and suitability per persona. Sentences were then manually selected so that they are less idiomatically tied to any single source domain and span six emotions per stimuli set: anger, fear, sadness, disgust, neutral, and joy.

TTS Systems Evaluated

Systems were selected from the Emergent TTS benchmark based on low WER (≤0.13) and ≥20% overall win rate, while also covering different stylistic archetypes along the axes of expressivity and spontaneity. Female voices with similar timbral profiles were chosen to minimise voice-preference bias:

• Kokoro (af_heart): A lightweight 82M-parameter StyleTTS 2 model. Provides high-quality, prosodically consistent speech but with low spontaneity.
• Gemini TTS (Flash 2.5, Despina): Google's flagship multimodal model, first on the Emergent TTS leaderboard. Delivers highly expressive and stylised speech.
• Kyutai-TTS (1.6B, p037): Built on the Moshi audio-to-audio framework, trained on raw conversational data. Captures high spontaneity and natural disfluencies.
• GPT-4o-mini-TTS (Coral): A high-quality commercial TTS with a balanced profile — medium-to-high spontaneity and moderate expressivity.
• ElevenLabs (multilingual_v2, Bella): A state-of-the-art commercial model designed for professional delivery with medium spontaneity and moderate expressivity.

Acoustic Features Analysed

To understand which acoustic characteristics are linked with appropriateness in each domain, the authors extracted features spanning three categories:

• Rhythm: Articulation rate standard deviation, speech rate, nPVI (normalised Pairwise Variability Index)
• Expressivity: F0 range (semitones), F0 percentiles, RMSE standard deviation, arousal, valence
• Voice Quality: Jitter, shimmer, H1–H2, alpha ratio, CPPS

Features were computed using praat-parselmouth, eGeMAPSv02 via openSMILE, and WavLM-large-MSP-Podcast-Emotion-Dim for valence and arousal.

Automatic Metrics Evaluated

To assess the effectiveness of automatic evaluation across domains, a broad suite of metrics was analysed at the sentence level:

• Quality Estimation: UTMOSv2, DNSMOS, Squim, PESQ, MCD, STOI
• Prosodic Distance: F0 correlation (SwiftF0), AutoPCP, WavLM distance
• Style: AudioBox CE/CU/PQ
• Intelligibility: WER (measured with nvidia/parakeet-tdt-0.6b-v2)
• Diversity: DS-WED

Results

Appropriateness Across Domains

The main result is that appropriateness varies substantially across domains, independently of naturalness (human-likeness). While most systems achieve relatively high appropriateness for reading and AI assistant roles, spontaneous conversation, actor, and animated character domains remain considerably more challenging.

Domain-specific profiles observed for each system:

• Kokoro performs well for reading and assistant tasks but scores poorly on conversational and expressive tasks. Its high assistant score despite relatively lower naturalness suggests listeners may actually expect AI assistants to sound somewhat robotic or clean.
• Kyutai-TTS excels for spontaneous conversation — it is perceived as highly appropriate in that domain — but scores poorly for AI assistant and animated character personas, sounding "too raw" for structured roles.
• ElevenLabs, Gemini, and GPT-4o-mini-TTS show similar mid-range profiles, scoring higher on reading and acting than on raw conversational or animated character domains. Interestingly, Gemini scored lower on AI assistant despite being the highest-ranked on the Emergent TTS leaderboard.

Inter-rater agreement was measured using Krippendorff's $\alpha$: 0.20 for TTS samples and 0.44 for ground truth samples, indicating a relatively low level of agreement — particularly for synthetic speech. This underscores that appropriateness is a highly subjective judgement, heavily shaped by individual listener expectations rather than a shared objective standard.

Human-Likeness and Appropriateness: The Correlation Paradox

The Spearman correlation between sentence-level human-likeness scores and appropriateness scores was computed for each domain:

The results reveal a striking range of relationships:

• Actor, Spontaneous, and Reader domains show positive correlations between human-likeness and appropriateness (Spearman $\rho$ = 0.47, 0.40, 0.38 respectively), suggesting that for these domains, sounding more natural aligns with sounding suitable.
• Animated Character shows a near-zero correlation ($\rho$ = 0.08), indicating that human-likeness is essentially irrelevant to appropriateness in this domain — stylised performance, not realism, is what listeners expect.
• AI Assistant shows a negative correlation ($\rho$ = −0.44), meaning that for this domain, more human-like speech is actually judged as less appropriate. Kokoro — the most "robotic" sounding system tested — scores highest on assistant appropriateness despite lower naturalness. Conversely, Kyutai-TTS, with its high naturalness, scores poorly as an assistant voice, confirming that high naturalness does not guarantee and can even hurt appropriateness for certain applications.

Impact of Speech Task on Human-Likeness Scores

Even ground truth (human) recordings varied in rated human-likeness across task families. Conversational samples from MELD and MSP-Podcast were preferred over LibriQuote and AnimeVox samples. Manual inspection suggests the lower LibriQuote scores may stem from the reader's Irish accent or more mature voice, which participants perceived as less natural — consistent with prior literature on non-standard speech. Lower scores for AnimeVox (but not MELD) similarly suggest that highly stylised, acted delivery is penalised under a human-likeness frame, regardless of the task.

For TTS systems, naturalness scores reflect a model's ability to handle specific styles rather than the task's inherent difficulty. Kyutai-TTS, for example, performed more naturally on conversational tasks than on narration — the reverse of what might be expected given that narration is traditionally considered an easier synthesis target. Crucially, a parallel analysis of mean appropriateness scores across speech tasks found no considerable differences, suggesting that naturalness and appropriateness capture different variance in the data.

Appropriateness and Acoustic Features

Spearman correlations between sentence-level appropriateness and acoustic features reveal strongly domain-specific acoustic profiles:

• Animated Character: Correlates most strongly with articulation rate variability ($\rho = 0.43$) and nPVI ($\rho = 0.41$). Appropriate animated character speech requires significant fluctuations in pacing and rhythm.
• Reader: Shows negative correlations with the same rhythmic variability features ($\rho \approx -0.30$), confirming that listeners prefer a steady, controlled rhythmic delivery for long-form reading — the opposite profile from animated character.
• Assistant: Preference for stability and neutrality: negative correlation with F0 range ($\rho = -0.35$) and valence ($\rho = -0.32$), indicating that overly expressive or emotionally charged speech is deemed less appropriate for an AI assistant. Spectral tilt (alpha ratio) shows a slight positive correlation ($\rho = 0.20$) for assistant.
• Actor & Spontaneous: Both show positive correlations with RMSE standard deviation ($\rho \approx 0.35$) and voice quality features. Uniquely, the Spontaneous domain is the only one to show notable positive correlations with jitter ($\rho = 0.34$) and creakiness (H1–H2, $\rho = 0.29$) — "human imperfections" are actually expected in casual interaction. Alpha ratio is penalised in both Spontaneous ($\rho = -0.28$) and Actor ($\rho = -0.21$) contexts.

A key insight from this analysis is that appropriateness is not solely determined by prosodic or rhythmic features but also by voice quality. While a TTS model can be prompted or trained to adapt rhythmic properties based on linguistic context, voice quality characteristics (jitter, shimmer, spectral tilt) are harder to adapt at inference time. This suggests that both the architectural design of the TTS system and the choice of voice must account for the target delivery style.

Appropriateness and Automatic Metrics

The correlation analysis between sentence-level appropriateness and automatic metrics exposes important blind spots in widely used evaluation tools:

• Quality Estimators (UTMOS, DNSMOS): Show a considerable negative correlation with appropriateness for Actor ($\rho \leq -0.41$) and Spontaneous ($\rho \leq -0.47$) domains. These metrics penalise the naturalistic disfluencies and high-dynamic range that are precisely what listeners expect in expressive or conversational speech. Conversely, for the Assistant domain, these same metrics correlate positively ($\rho \approx 0.35$), confirming they remain valid proxies only for clean, neutral, high-fidelity audio.
• Embedding-based Metrics (AudioBox, AutoPCP): Effective for Assistant and Reader styles ($\rho \geq 0.36$), but fail to capture human-likeness nuances, showing stronger inverse correlations ($\rho \approx -0.50$) with human-likeness scores.
• WavLM Distance: Strongly negatively correlated with Assistant appropriateness ($\rho = -0.72$) but positively correlated with human-likeness, indicating that WavLM embeddings encode a bias toward neutral speech patterns.
• DS-WED: A more robust metric for diversity, performing more consistently across domains.
• PESQ and F0 Correlation: Remain largely uninformative for predicting perceived appropriateness across domains.

Overall, these results demonstrate that no single automatic metric is universally appropriate for evaluating TTS across diverse application domains. Metrics optimised for clean, read, high-fidelity audio — the dominant paradigm in TTS training and evaluation — actively mislead when applied to expressive, conversational, or spontaneous contexts.

Radar Profile Analysis

The radar chart visualisation makes visible the domain-specific profile of each TTS system. Kokoro shows a narrow profile concentrated on reader and assistant dimensions. Kyutai-TTS presents a broader, more spontaneous profile but dips sharply for assistant and animated character. ElevenLabs, GPT-4o-mini-TTS, and Gemini cluster as generalist mid-range systems, performing comparably across reading and acting but struggling at the extremes of spontaneous casual speech and highly expressive animated character voices. No system achieves uniformly high scores across all five dimensions, confirming the domain-specific nature of TTS performance.

Discussion

The study demonstrates several critical insights for TTS evaluation and system design:

1. Evaluation is inherently domain-dependent. Listener judgements for the same utterance shift based on the framed scenario. Perceived quality depends as much on situational expectations as on the acoustic signal itself. A single global score cannot capture this variance.
2. Human-likeness and appropriateness are distinct dimensions. Human-likeness fails to reliably predict cross-domain suitability and tends to favour spontaneous speech over stylised speech. Even highly human-like voices can feel inappropriate for specific tasks.
3. TTS systems exhibit domain-specific affinities. Systems likely reflect their training data and optimisation targets; a system trained on clean read speech excels in reader and assistant domains but fails in expressive domains, and vice versa. TTS systems and voices are not one-size-fits-all solutions.
4. Bias in naturalness scoring. Naturalness scores penalise stylised or heavily processed speech (e.g., animated character, acted performance) while rewarding spontaneous or disfluent delivery, creating a systematic bias that does not reflect domain suitability.
5. Voice quality matters, not just prosody. Features like jitter, shimmer, and creakiness are positively correlated with appropriateness in the spontaneous domain — meaning "imperfect" voice quality is desirable in casual interaction. These properties are difficult to adapt at inference time and must be considered at the voice and system design stage.
6. The "assistant bias" in listener expectations. The negative correlation between human-likeness and assistant appropriateness, and Kokoro's strong assistant scores despite lower naturalness, suggests that listeners have formed stable expectations of AI assistant voices being cleaner and less spontaneous than natural human speech.

Limitations

• Isolated sentences: Real-world speech involves dialogue, turn-taking, and emotional progression. The evaluation of single sentences does not capture multi-turn dynamics, which is a natural next step for future work.
• Speaker characteristics not explored: The study did not investigate how perceived gender, age, accent, or socioeconomic background of a voice influences appropriateness perception across domains. These social identity dimensions are important and remain an open avenue.
• Low inter-rater agreement for TTS: Krippendorff's $\alpha$ of 0.20 for TTS stimuli (vs. 0.44 for ground truth) indicates that appropriateness judgements for synthetic speech are highly variable across listeners, making it difficult to draw universally generalisable conclusions.
• AI assistant ground truth: The proxy ground truth for the AI assistant domain was generated using ElevenLabs v3 rather than a true human recording, which may introduce some circular evaluation for ElevenLabs as a system.

Conclusion

In light of the dramatic recent improvements in TTS fidelity, this work re-examines the central question: what does it mean for speech synthesis to be "good"? The listening experiments presented here demonstrate that while all five tested SOTA systems achieve competitive human-likeness scores, appropriateness varies substantially depending on the target domain — and the two dimensions are not only distinct but can be negatively correlated (e.g., for AI assistant applications).

Systems perform best in reading scenarios, while conversational and expressive domains remain considerably more challenging. Optimising a voice or system for one domain can actively reduce perceived appropriateness in others. Common automatic metrics — particularly MOS predictors — are valid for clean, neutral domains but actively mislead when applied to expressive, spontaneous, or acted speech. The finding that naturalness metrics penalise stylised speech while rewarding spontaneity is a systematic bias that the community should correct for.

The central message is that TTS performance is not "solved" — and that progress is not a simple monotonic advance but a profile across domains. Rather than asking whether an utterance "sounds like a human," evaluation should ask whether an utterance "sounds right" for its intended use. Future work should extend domain-aware evaluation to multi-turn dialogue, investigate the role of voice identity characteristics (age, accent, gender presentation) in shaping appropriateness perception, and develop interpretable, domain-sensitive metrics that allow iterative and targeted improvement of TTS systems for their intended applications.